The Problem: Sequential RAG Stages Created Compounding Latency

Kestrel Search’s search product executes a retrieval-then-generation pipeline: a vector search against a 2.3B-document index, followed by reranking, followed by LLM generation over the retrieved context. In the original architecture, each stage was fully synchronous and blocking, meaning the client received no output until retrieval, reranking, and the entire generation pass had completed.

The architectural problem was that latency was additive across stages with no overlap, and the client-perceived wait time included the full generation pass rather than just time-to-first-token. At P90, total response time exceeded 3.8 seconds, which Kestrel Search’s own usability research had identified as the threshold where users began perceiving the product as ‘slow.’

The Implementation: Meridian Streaming Pipeline with Speculative Prefetch

Kestrel Search restructured their pipeline using Meridian’s streaming primitives. Vector search and reranking now execute against a pre-warmed connection pool, and generation begins streaming tokens to the client via Server-Sent Events as soon as the first reranked context chunk is available, rather than waiting for the full context set.

We stopped treating retrieval and generation as two sequential black boxes. Meridian’s speculative prefetch lets us start the generation request before reranking has fully converged, and we cancel and re-issue if the context changes materially.

Meridian’s connection multiplexer also maintains persistent, pre-authenticated connections to Kestrel Search’s downstream model provider, eliminating the TLS handshake and cold-start overhead that previously added 180-220ms to every cold request.

Results: Measured Infrastructure Impact

End-to-end P90 latency fell from 3.8 seconds to 1.86 seconds, a 51% reduction, with time-to-first-token specifically dropping to under 280ms. Stream completion rate, the percentage of streams that deliver a final token without client-side timeout or connection drop, rose to 98.7%, up from 94.1% under the prior synchronous architecture.