Architecting a Zero-Latency Anycast Network for LLM Token Routing

Response caching for LLM inference has an obvious appeal: if two requests share an identical prompt and model configuration, serving a cached completion avoids redundant inference cost entirely. The complexity is entirely in invalidation correctness, particularly in a multi-tenant system where a cache key collision across tenants is not a performance bug but a data-isolation incident.

Key Design: Tenant Isolation by Construction

Every cache key is constructed as a composite hash that includes tenant ID, model identifier, a normalized prompt hash, and a configuration fingerprint covering temperature, top_p, and any system-prompt overrides. We deliberately avoid any key structure where tenant isolation depends on application-level logic remembering to check tenant ownership after a cache hit; isolation is built into the key itself, so a cross-tenant collision is structurally impossible rather than merely guarded against.

def build_cache_key(tenant_id, model_id, prompt, config):
    config_fp = hashlib.sha256(
        f"{config.temperature}:{config.top_p}:{config.system_prompt}".encode()
    ).hexdigest()[:16]
    prompt_hash = hashlib.sha256(normalize(prompt).encode()).hexdigest()
    return f"infer:{tenant_id}:{model_id}:{prompt_hash}:{config_fp}"

def build_cache_key(tenant_id, model_id, prompt, config):
    config_fp = hashlib.sha256(
        f"{config.temperature}:{config.top_p}:{config.system_prompt}".encode()
    ).hexdigest()[:16]
    prompt_hash = hashlib.sha256(normalize(prompt).encode()).hexdigest()
    return f"infer:{tenant_id}:{model_id}:{prompt_hash}:{config_fp}"

def build_cache_key(tenant_id, model_id, prompt, config):
    config_fp = hashlib.sha256(
        f"{config.temperature}:{config.top_p}:{config.system_prompt}".encode()
    ).hexdigest()[:16]
    prompt_hash = hashlib.sha256(normalize(prompt).encode()).hexdigest()
    return f"infer:{tenant_id}:{model_id}:{prompt_hash}:{config_fp}"

TTL Strategy and the Staleness/Cost Tradeoff

We use a tiered TTL rather than a single global value. Deterministic, low-temperature completions (temperature under 0.2) are cached for up to 24 hours, since repeated identical prompts are highly likely to be acceptable as cached results for that window. Higher-temperature completions, where a tenant likely expects variation across calls even for an identical prompt, are cached for only 90 seconds, primarily to absorb accidental duplicate requests from client-side retry logic rather than to serve as a general-purpose cache.

Active Invalidation on Model Version Changes

TTL alone is insufficient when we deploy a new model version, since stale completions from a prior model version could otherwise remain servable for hours. We maintain a Redis SET per model identifier tracking all active cache keys for that model, and on version rollover we issue a pipelined UNLINK across the full set rather than relying on natural TTL expiry, ensuring zero cached responses survive a model swap.

Operational Results

Cache hit rate across tenants currently averages 23%, concentrated heavily in low-temperature, high-repetition workloads like classification and structured extraction. For those workload types specifically, hit rate exceeds 60%, translating directly into avoided inference cost without any measured incident of cross-tenant data exposure since the composite-key design shipped.