Architecting a Zero-Latency Anycast Network for LLM Token Routing

Most LLM-routing infrastructure inherits its topology from generic web CDN design: pick a region, pin a client to it via DNS, and hope the region stays healthy. That model breaks down for inference traffic, where a single provider-side degradation event can spike latency for an entire region's worth of clients with no fast path to reroute. We replaced DNS-based regional pinning with a true Anycast network, where every PoP advertises the same IP block and BGP determines the shortest path to the client.

BGP Configuration and Route Propagation

Each of our 19 PoPs runs BIRD as the routing daemon, advertising a /24 from our allocated block to upstream transit providers and peers. The critical design decision was withdrawal behavior: when a PoP's local health checks fail, BIRD withdraws the route within 2-3 seconds, and upstream routers reconverge to the next-best path. We tuned BGP hold timers down from the default 180 seconds to 60 seconds specifically to shrink this reconvergence window, accepting slightly higher keepalive overhead in exchange for faster failure detection.

protocol bgp upstream_transit {
    local as 64512;
    neighbor 203.0.113.1 as 65000;
    hold time 60;
    keepalive time 20;
    export filter {
        if net = 198.51.100.0/24 then accept;
        reject;
    };
    import none;
}

protocol bgp upstream_transit {
    local as 64512;
    neighbor 203.0.113.1 as 65000;
    hold time 60;
    keepalive time 20;
    export filter {
        if net = 198.51.100.0/24 then accept;
        reject;
    };
    import none;
}

protocol bgp upstream_transit {
    local as 64512;
    neighbor 203.0.113.1 as 65000;
    hold time 60;
    keepalive time 20;
    export filter {
        if net = 198.51.100.0/24 then accept;
        reject;
    };
    import none;
}

Health Checks That Actually Reflect Inference Health

Generic TCP health checks are insufficient for inference routing because a PoP can accept TCP connections perfectly well while the upstream model provider it depends on is degraded. We run an application-layer health check every 5 seconds that issues a synthetic short-context inference request to each configured provider and measures time-to-first-token. If TTFT exceeds a 400ms threshold for 3 consecutive checks, the PoP marks that provider unhealthy locally and the BGP route withdrawal logic above is triggered for traffic that would otherwise route there.

Failure Modes We Had to Design Around

The two failure modes that caused us the most trouble during rollout were route flapping and asymmetric routing. Flapping occurred when a PoP's health oscillated near the threshold, triggering rapid advertise/withdraw cycles; we fixed this with a dampening algorithm requiring a stable healthy state for 30 seconds before re-advertisement. Asymmetric routing, where a request's outbound and return paths traversed different PoPs, complicated our connection-level metrics; we resolved this by making session affinity sticky at the application layer rather than relying on routing symmetry.