Key takeaways

  • RFC 4786 (BCP 126) defines anycast as making a service address available in multiple, discrete, autonomous locations. Datagrams sent to that address are routed to one of those locations by normal BGP path selection.
  • RFC 4786 recommends tying route advertisements to service health, so that if the service stops answering, the node withdraws its route. It also advises a minimum delay before re-advertising, to avoid oscillation.
  • Anycast spreads an attack. RFC 4786 notes that it can constrain distributed denial-of-service attacks or flash crowds to local regions around the nodes that receive them, and that the work of filtering is spread across all nodes.
  • UDP is the natural transport. RFC 7094 warns that packets of one TCP connection may be delivered to different instances if routing changes, which can reset the connection. DNS is mostly UDP and each query is self-contained, which is why it suits anycast so well.
  • RFC 7094 discusses route withdrawal and flap damping. Frequent advertise and withdraw cycles can trigger route-flap damping at other networks and suppress valid paths, so health checks need hysteresis.
  • Use a unique origin AS per node when you can (RFC 7094 cites RFC 6382). It gives monitoring tools a way to tell which node answered and can simplify RPKI route origin authorizations.
  • Anycast is not a replacement for capacity. Each site still needs headroom for the traffic that a failed neighbor would shed onto it, and RFC 4786 says nodes should be scaled to handle loads far in excess of the average.

One address, many places

Ordinary unicast addressing gives each host one address that exists in one place. Anycast reverses that. The same address, for example the address of a DNS resolver or an authoritative name server, is configured at several sites and announced into the routing system from each of them. Every router on the internet picks the best path it knows to that address, and for a client in Frankfurt that is usually a site in Europe, while a client in Miami reaches a site in the Americas. RFC 4786, which is also Best Current Practice 126, puts it plainly: anycast is the practice of making a particular service address available in multiple, discrete, autonomous locations, with datagrams routed to one of them.

Nothing in the packet marks it as anycast. RFC 7094, from the Internet Architecture Board, observes that anycast addresses are syntactically indistinguishable from unicast addresses. The difference exists only in how the address is announced. That is both the strength of the technique, since no client changes are needed, and the source of its caveats, since the client and the application do not know which instance they are talking to.

How the announcement works

In practice, each site runs a BGP session with its upstream carriers and peers, and announces the service prefix, often a /24 for IPv4 because longer prefixes are commonly filtered, and a /48 for IPv6. A daemon such as BIRD or FRRouting on a server, or the site's edge routers, handles the session. The announcing network usually runs its own autonomous system number (ASN) and has its own address space. Routing then does the rest: each remote network compares the paths to the prefix it hears and prefers the one with the shortest AS path, the highest local preference and so on, according to its own policy.

Two consequences follow. First, proximity is a matter of routing policy, not geography. A client's network might prefer a distant site because of a commercial relationship, so the nearest site on the map is not always the one used. Second, operators have some levers: prepending the AS path to make a site less attractive, attaching communities that ask upstreams to adjust preference or to limit propagation, and announcing different prefixes through different carriers. Our BGP article covers filters, RPKI and communities in more detail.

RFC 4786 also distinguishes global nodes, which are announced widely and reachable from everywhere, from local nodes, whose routes are deliberately restricted to a local catchment. It says local nodes advertise covering routes with propagation restricted, using communities such as NO_EXPORT or NOPEER, or with peering-style import policies. A mix of global and local nodes lets a service serve a region from a nearby node and still fail over to a global node if the local one fails.

Why DNS is the classic use

A DNS query is a single small UDP packet and a single response. Each query stands alone. Nothing in the server needs to remember the client between queries, and the client retries if no answer comes. RFC 7094 describes the conditions under which anycast is safe: requests are self-contained, the transport is stateless such as UDP, replies come back from a unicast address, the server keeps no hard state, and retries are idempotent. DNS meets all of them. That is why, as RFC 7094 reports, at least ten of the thirteen root server identities were using anycast by the late 2000s.

The benefits for DNS are practical. Latency drops because answers come from a nearby site, and that matters because DNS sits in front of every web page load. Capacity grows by adding sites. Failure of one site shifts its clients to the next best site with no change in the DNS records. And attacks are spread across the whole platform instead of landing on one server, which is the subject of a later section.

Two caveats apply to DNS. Responses that exceed the UDP size limit fall back to TCP, and DNSSEC-signed answers are larger, so some queries will use TCP. Short TCP exchanges usually finish before a route change can interfere, and the practice works at scale, but the risk exists. And zone data must be identical at every site. Anycast spreads the answers, so a stale zone at one site gives wrong answers to the clients near it. Use zone transfers (AXFR and IXFR) with notifications, or push the same zone to every node from a signed source, and monitor serial numbers at each node.

Health-checked withdrawals

An anycast site only helps if it stops attracting traffic when it cannot serve it. Routing knows nothing about whether the DNS software is running. If the server process crashes but the router keeps announcing the prefix, the network keeps delivering queries into a black hole. RFC 4786 addresses this directly: it recommends coupling the route advertisement to the health of the service, so that non-availability of the service triggers a route withdrawal. It also advises a minimum delay before the route is re-advertised, so a flapping service does not cause constant advertise and withdraw cycles.

A common design runs a health checker on each node. It sends a real DNS query for a known name to the local resolver or name server every second or so, checks that the answer is correct, and tells the BGP daemon to withdraw the prefix after several consecutive failures. The announcement is restored only after a longer run of successes. The design choices are the check interval, the number of failures needed, and the hold-down. A tight interval reacts quickly but risks pulling a healthy node during a brief hiccup. A loose one leaves clients waiting. The numbers belong in the design, tested for the specific platform.

Several traps are worth naming. A check that only tests the loopback address misses firewall and interface faults, so test through the same path clients use where possible. A check that tests only that the process is listening misses an empty or stale zone. And a guard is needed against the check itself failing everywhere at once, for example a bad monitoring change: a rule such as never withdraw if this would remove the last N sites avoids a self-inflicted outage. RFC 4786 makes a similar point about single defects: it advises minimizing the opportunity for a single defect to compromise many or all nodes, by making nodes autonomous and mixing implementations.

RFC 7094 adds a nuance about withdrawal. An older document, RFC 3258, advised against withdrawing routes and relying on DNS-level failover instead. RFC 7094 argues the opposite for most deployments: the benefits of anycast mostly depend on withdrawal, because otherwise traffic keeps flowing to a failed instance. It also warns of the cost. Frequent withdrawals can trigger route-flap damping at other networks, which suppresses valid paths, and withdrawing a site during a sustained attack shifts the attack to another site. Withdrawal during a flood is a decision to take with care, not an automatic reflex.

DDoS absorption

Anycast helps against floods because it divides them. A botnet spread around the world sends traffic to the nearest instance from each bot. Each site sees only the share that routes to it, which is less than the whole. RFC 4786 states the benefit in two ways: anycast can constrain distributed denial-of-service attacks or flash crowds to local regions around the nodes, and the work of filtering attack traffic is spread across all nodes. A non-distributed attack, from one source, is localized to the single node that attracts it, leaving the others untouched.

This does not make a site immune. A site's capacity is the sum of its links and its servers, and an attack aimed at one region can still exceed it. Defense in depth is still needed: ingress filtering to stop spoofed sources, response-rate limiting on DNS servers to blunt reflection and amplification, packet filtering on the servers (XDP can drop obvious junk before the kernel allocates buffers), BGP FlowSpec and remote-triggered blackholing for upstream filtering, and a scrubbing service for the biggest floods. Our DDoS article describes each layer. What anycast adds is a way to choose where the attack lands. An operator can make a site less attractive by prepending the AS path or withdrawing a route, which pushes attack traffic toward better-provisioned sites, and can shape attack traffic toward the sites best able to filter it.

RFC 4786 gives one more design rule that matters here. Because failover from local nodes tends to land on global nodes, and because an overloaded node can fail and shed its load onto its neighbors, nodes should be scaled to handle loads far in excess of the average. Plan the capacity of each site so that the loss of the busiest neighbor does not push the others over the edge. Cascading failure is the main way anycast platforms fail.

Edge load balancing beyond DNS

The same technique spreads web and API traffic. Each site announces the service address by anycast, and inside each site a load balancer or a set of servers shares the traffic. Within a site, equal-cost multipath routing (ECMP) can hash flows across many servers that all announce the address into the local routers, which removes the need for a central load balancer. Because the hash keeps packets of one flow together, a TCP connection stays on one server as long as the set of servers does not change. Consistent hashing reduces the number of flows that move when one server is added or removed.

The harder problem is between sites. If routing changes during a long TCP connection, for example a download, a video stream or a WebSocket, the next packet may arrive at a different site that has no state for it, and the connection resets. RFC 7094 flags this risk for TCP and stateful protocols. In practice, operators mitigate it with stable routing (avoid unnecessary withdrawals and path changes), short-lived connections, application retry, and designs that use anycast only to find the best site and then move the session to a unicast address of that site. Some platforms also forward packets between sites, which is possible but adds complexity. For services where a reset is costly, regional unicast with DNS-based traffic steering may serve better, with anycast used for the front door only.

Per-packet load balancing across unequal paths is the other known hazard. RFC 7094 notes that BCP 126 considers per-packet balancing over equal-cost paths pathological for anycast, since packets of one flow reach different instances. Routers should hash on flow fields, not spray packets.

Monitoring and security of the platform

You cannot manage what you cannot see. Each node should answer a query for a special name that returns its own identity (a mechanism such as the NSID option or a hostname.bind query is common), and probes from many external vantage points should record which node answers and how fast. Per-node unique origin ASNs, mentioned in RFC 7094 with a pointer to RFC 6382, give outside observers a clean discriminator, and they simplify RPKI route origin authorizations. Alert on a node that stops announcing, a node that announces without answering, an unexpected origin, and differences in zone serials between nodes.

Security of the announcements matters as much as security of the servers. A hijacked prefix sends your queries to someone else's server. Publish accurate ROAs, filter your sessions as RFC 7454 recommends, watch public route collectors for unexpected origins of your prefixes, and sign your zones with DNSSEC. RFC 7094 itself advises end-to-end security such as DNSSEC or IPsec, because anycast discovery is exposed to man-in-the-middle attacks.

A short design checklist

  • Own your address space and ASN where you can, so you control announcements and can publish ROAs.
  • Place sites where your users are and where upstream connectivity is diverse. Use at least three, with different upstreams.
  • Keep every node autonomous: its own health check, its own local data, no dependency on a central controller to keep answering.
  • Tie the BGP announcement to a real service check with hysteresis and a never-withdraw-the-last-sites guard.
  • Size each site so that it survives the loss of its largest neighbor, and test it with a game day.
  • Instrument node identity, zone serials and external probes from day one.
  • Document the attack playbook: when to prepend, when to withdraw, when to call the scrubbing provider.

Our team designs and operates anycast and edge platforms with BGP, BIRD and health-checked announcements, and ties them into DDoS protection and monitoring. A small organization rarely needs its own anycast network; many are well served by a managed DNS provider that already runs one. When you do need one, the points above are what decide whether it holds up on its worst day. Related reading: DNS Security: DNSSEC, Response Policy Zones and Encrypted DNS.

Sources

This article is general technical information, not a design for your network. We read RFC 4786 and RFC 7094 through a summarizing fetch tool and not in full, so confirm quotations and section numbers against the RFC text. The practical details (a /24 or /48 announcement size, ECMP hashing, consistent hashing, health-check intervals and guards, NSID and hostname.bind identification, zone transfer practice) are general engineering knowledge, not quoted from the sources listed. Timers and thresholds are examples to test, not recommendations for your platform. Last reviewed .

Keep reading

Need help with this?

Cloud cost review and management, without compromising on quality or security.

Cloud Management