Architecting a Zero-Latency Anycast Network for LLM Token Routing

Inference gateways occupy a privileged position: they see every prompt, every system instruction, and often have credentials to downstream tools invoked via function calling. A compromised or improperly isolated gateway is not just a latency or availability risk, it is a data-exfiltration and tool-misuse risk. We rebuilt our gateway-to-model-server trust model around zero-trust principles rather than relying on network perimeter security alone.

Mutual TLS Between Gateway and Model Servers

Previously, model servers behind our gateway trusted any connection originating from inside the VPC, a common but increasingly insufficient assumption once a fleet spans multiple cloud accounts and third-party inference providers. We issued short-lived client certificates to each gateway instance, rotated every 4 hours via an internal CA, and configured model servers to reject any connection that does not present a valid certificate signed by that CA.

server {
    listen 8443 ssl;
    ssl_certificate     /etc/meridian/server.crt;
    ssl_certificate_key /etc/meridian/server.key;
    ssl_client_certificate /etc/meridian/internal-ca.pem;
    ssl_verify_client on;
    ssl_verify_depth 2;
}

server {
    listen 8443 ssl;
    ssl_certificate     /etc/meridian/server.crt;
    ssl_certificate_key /etc/meridian/server.key;
    ssl_client_certificate /etc/meridian/internal-ca.pem;
    ssl_verify_client on;
    ssl_verify_depth 2;
}

server {
    listen 8443 ssl;
    ssl_certificate     /etc/meridian/server.crt;
    ssl_certificate_key /etc/meridian/server.key;
    ssl_client_certificate /etc/meridian/internal-ca.pem;
    ssl_verify_client on;
    ssl_verify_depth 2;
}

Containing Prompt-Injection Blast Radius

Mutual TLS secures the transport layer, but it does nothing to contain a successful prompt-injection attack that convinces a model to misuse a legitimate function-calling tool it has access to. Our second change was scoping tool credentials per request rather than per tenant: each function call is executed with a narrowly-scoped, single-use credential generated for that specific request, valid only for the tool and resource explicitly referenced in the original user-facing request.

Request-Level Sandboxing for Tool Execution

Tool calls themselves execute in an isolated subprocess with an explicit allowlist of reachable network destinations, rather than inheriting the broader network access of the gateway process. This means even a successful prompt injection that convinces a model to attempt an unauthorized action, such as exfiltrating data to an attacker-controlled endpoint, fails at the network layer because the sandboxed execution context has no route to that destination.

Operational Impact

Since deploying request-scoped credentials and sandboxed tool execution, our red-team exercises simulating prompt-injection attacks against function-calling workflows have shown zero successful out-of-scope tool invocations, compared to a 12% success rate against the prior tenant-scoped credential model in equivalent test scenarios.