The Problem: Single-Region Inference Was a Single Point of Failure

Tarn Systems’ command-bar product issues over 40M autocomplete and AI-action requests per day, each with a strict sub-200ms latency budget before the UI feels unresponsive. The original architecture routed every inference call through a single us-east-1 endpoint backed by a single model provider. When that provider experienced a degraded-service event, Meridian’s client SDK had no failover path, and P99 latency spiked past 4 seconds during incidents that occurred roughly twice per quarter.

The core architectural flaw was coupling request routing to a single provider’s health, with no abstraction layer to detect and react to degradation in real time. Retries were implemented client-side with fixed backoff, which amplified load on an already-degraded upstream rather than redirecting it.

The Implementation: Meridian Edge Router with Health-Aware Failover

Meridian migrated inference traffic to the Meridian Edge Router, deployed across 19 PoPs on an Anycast IP. Each PoP maintains a rolling 5-second health window per upstream provider, tracking error rate, TTFB, and token-stream stall events. Routing decisions are made at the edge using a weighted least-connections algorithm that excludes any provider whose health score falls below a configurable threshold (set to 0.92 in Tarn Systems’ production config).

We replaced four separate provider SDKs and a homegrown retry wrapper with a single Meridian client. The failover logic that used to live in our application code now lives at the edge, where it has the telemetry to actually make good decisions.

Meridian also terminates TLS at the edge node nearest the requesting client, which removed an average of 38ms of round-trip negotiation time previously incurred reaching us-east-1 from European and APAC clients.

Results: Measured Infrastructure Impact

Post-migration, Meridian’s median failover time (detection-to-redirect) dropped from 6.1 seconds to under 1.2 seconds, an 80% reduction. P99 latency across all regions settled at under 12ms for routing overhead specifically (excluding model inference time), down from 340ms of routing and retry overhead in the legacy architecture. Provider-side incidents that previously caused customer-visible outages are now absorbed transparently, with zero failed requests logged during the two provider-side incidents that occurred in the 90 days following migration.