Notes on infrastructure.

We rebuilt our edge routing layer on a 19-PoP Anycast network to remove region-pinning from LLM request paths. This post covers the BGP configuration, health-check propagation, and the failure modes we had to design around.

Our original Server-Sent Events implementation buffered tokens at three separate layers, adding unnecessary latency. Here's how we rebuilt the pipeline to deliver tokens to clients within 50ms of model generation.

Caching LLM responses across tenants is straightforward until you need correct, fast invalidation. We detail the Redis key design and TTL strategy that lets us cache aggressively without serving stale or cross-tenant-leaked results.

We benchmarked GPTQ and AWQ 4-bit quantization against full-precision baselines across our production workload mix. The results show where quantization is safe to ship and where it quietly degrades output quality.

Treating prompt content as untrusted input requires the same rigor we apply to network traffic. This post covers our mutual TLS rollout between gateway and model servers, plus the request-level isolation we added to contain prompt-injection blast radius.

Meridian Router v2 ships with adaptive provider scoring, automatic model-equivalence mapping, and configurable failover policies. Here is what changed under the hood and why.