Vol. 7 - Why Your RAG Costs Too Much and Answers Too Slowly
Series: Building Production RAG
The previous article kept the index correct and current as the corpus grew. This one is about what happens on every single request once real traffic arrives: five thousand users, thousands of concurrent requests, answers taking eight to thirty seconds, and a bill nobody budgeted for.
None of these are corpus problems. They are serving problems, and they have their own failure modes. An answer that takes thirty seconds is a failed answer no matter how correct it is, because the user has already left. A system that costs ten thousand dollars a day is a failed system no matter how good it is, because it will be switched off.
Three things decide both: the model layer, the context window, and how you behave under load.
The LLM gateway: the production layer most teams skip
In a demo, the app calls the model provider directly. In production, you put a gateway between your application and the model providers: a single layer (LiteLLM and similar tools do this) through which every model call flows.
The gateway buys you three things.
Routing. Not every query needs your most expensive model. A gateway lets you tier them: a small, cheap model handles simple lookups, and the frontier model is reserved for the hard, multi-step questions that actually need it. This one capability is where most of the cost savings live.
Our corpus makes the split obvious. “What’s the default request timeout in v4.2?” is a lookup: retrieve one config document, read one value out of it. A small model does that perfectly, and paying frontier prices for it is waste. “Why did the retry behaviour change between v3 and v4, and what do we need to update?” is a different animal: several documents, a comparison across versions, and an inference the documents never state outright. That one needs the frontier model. The two questions look similar and cost about twenty times more from one than the other.
So how do you actually decide? In rough order of what to try first:
Rules, where they work. Cheap and deterministic. Query length, an explicit version mention, whether it parses as a single lookup. Rules are free and they are debuggable, which matters more than it sounds when something misroutes at 3am.
A small classifier. One call to your cheapest model: “is this a simple factual lookup or a multi-step reasoning question?” Costs a fraction of a cent and catches what rules miss. Notice the recursion, and accept it: you are spending a small model call to avoid a large one.
Signals from retrieval. You already know things by the time you generate. How many chunks came back above the relevance threshold? Do they agree with each other? One strong chunk suggests a lookup; six chunks across four documents suggests synthesis.
Escalate on failure. Route optimistically to the small model, then check the answer. If it abstains, contradicts the context, or scores badly on grounding, retry on the frontier model. You pay twice on the minority of queries that need it, and once on the majority that do not.
Whatever you choose, route on evidence, not on vibes, and put the router itself through the evaluation loop. A router that sends ten percent of hard queries to the small model is not saving money, it is quietly degrading quality where nobody is looking.
Fallback. Providers have outages and rate limits. Fallback is what turns a provider’s bad day into a slightly different answer rather than an error page, and it needs to be designed rather than assumed:
Order the chain by capability, not by price. Falling back from a frontier model to something markedly weaker is not resilience, it is a silent quality drop. The fallback should be able to do the job.
Distinguish the failure. A timeout or a 500 is worth retrying. A rate limit means back off, or shift to the secondary immediately. A malformed request will fail identically no matter how many times you send it, so do not.
Break the circuit. When a provider fails consistently, stop sending it traffic for a cooling period rather than hammering it and paying for the privilege.
Make it visible. A fallback that fires silently is a quality regression nobody is measuring. Log it, alert on the rate, and tag the response so evaluation can see which model actually answered.
Cost tracking. Every call flows through one place, so you can attribute cost per feature, per team, even per user, set budgets, and alert when something runs away. You cannot control a cost you cannot see, and direct provider calls scattered across a codebase are invisible.
The cost math
Routing is not a marginal optimisation. Here is an illustrative day at a million queries.
Without a gateway, every query hits the frontier model. At a blended cost of about $0.01 per query, that is $10,000 a day. Now add caching and routing:
Same traffic, same quality where it matters (the hard queries still get the strong model), at roughly one-sixth the cost. Over a year that is the difference between about $3.6M and about $640K. The gateway is the layer that turns this from a rewrite into a config change.
The exact numbers are illustrative; your blend depends on your traffic mix and cache hit rate. The shape holds widely: most queries are simpler than your worst case, and they shouldn’t have to pay for it.
Prompt caching: the cost lever inside the call
Routing and the semantic cache both work by avoiding calls. Prompt caching works inside a call that neither of them caught, and most teams never turn it on.
Every request you send begins with the same long, unchanging prefix: the system prompt, the instructions, the few-shot examples, the output format. Providers will cache that prefix and charge a fraction of the input price for it on subsequent calls, so long as it is byte-identical and sits at the front of the prompt.
The practical consequence is that prompt layout is a cost decision. Put everything static at the front (system prompt, instructions, examples), and everything variable at the back (retrieved chunks, conversation history, the user’s question). Reorder them and the cache misses. Interpolate a timestamp or a request ID into the system prompt and you have silently broken it on every single call, which is a spectacular way to lose a discount you thought you had.
This is also the real answer to re-sending a fifty-page manual with every request: do not. Retrieve the handful of relevant pages instead, and let prompt caching cover the fixed scaffolding around them.
Cost guardrails: the bill you did not intend
Routing lowers your average cost. Guardrails stop your worst cost, and they are a different problem. The bills that shock people are almost never a gradual creep; they are one bug, running fast.
The classic is a retry storm. A feature retries failed requests, and on failure it retries again, resending the full context every time. Ten retries on a large prompt is ten times the cost of a call that never succeeded. Multiply by a traffic spike and a bad afternoon becomes an order-of-magnitude overrun. The gateway is where you cap this, because every call already passes through it:
Cap retries and back off exponentially. Three attempts, not ten, and never retry a request that failed because it was malformed. It will fail identically the second time, at full price.
Break the circuit. When a downstream model starts failing consistently, stop sending it traffic for a cooling period instead of hammering it. Retrying a dead provider is pure spend.
Cap the request itself. Set a maximum input size and a maximum output length. Without an output cap, one pathological query can generate until the context runs out.
Budget per feature, and alert on the derivative. Alert on the rate of spend, not the monthly total, since a monthly threshold tells you about the disaster after it has finished happening.
Estimating the bill before you build
Leadership will ask what this costs before you have written any of it, and “it depends” is not an answer. It is estimable, and the estimate is worth doing because it often changes the design.
Cost has two halves. Ingestion is one-time-ish and usually small: embedding the corpus, plus re-embedding whatever changes. Fifty thousand documents is a rounding error against your serving bill; the trap is re-embedding the whole corpus every time you tweak chunking, which is exactly what the caching and versioning above exist to prevent. Serving is the number that matters, and it is driven by four things: queries per day, tokens in per query (system prompt plus retrieved chunks plus history, which is why the context budget is a cost lever), tokens out per query, and the blended price you pay after routing.
That gives a workable model:
daily cost = queries
x (1 - cache_hit_rate)
x (tokens_in x price_in + tokens_out x price_out)
Reranking and any auxiliary model calls (query rewriting, decomposition, the judge in your eval loop) sit on top, and they are not free at scale. Vector DB hosting is real but usually secondary to inference.
Run the numbers before committing. The exercise tends to reveal that the cache hit rate and the routing split, the two things you control, matter more than the model you picked, which is the point of the section above.
Context window management: the hidden budget
This is the one that surprises people, because it is invisible at small scale. With ten users, you can stuff everything into the model’s context window (the full conversation history, a dozen retrieved chunks, every tool result) and it works. The window is large and you are not paying much attention to how full it gets.
At scale that habit breaks in three ways at once. Cost: you pay per token, so bloated context is pure waste on every single call. Latency: more tokens means slower responses, multiplied across thousands of concurrent requests. Quality: models attend less reliably to information buried in a very large context, the well-documented “lost in the middle” effect. The fix is to treat the context window as a budget and spend it deliberately.
Everything competing for that budget:
System prompt and instructions. A fixed cost. Keep it tight.
Conversation history. Grows without bound if you let it. Cap it: keep the last few turns verbatim and replace older history with a running summary.
Retrieved chunks. More is not better. Rerank, then keep only the top few that clear a relevance bar, rather than padding the window with marginal matches (which also drags down context precision, from the evaluation article).
Tool results. The silent killer. A single SQL result or fetched document can be enormous. Truncate or summarise tool output before it enters the context; never inject it raw.
Compress rather than drop. When the answer genuinely needs information spread across twenty chunks and only five will fit, you do not have to choose between them. Send each chunk through a compression step first: an LLM or a small extractive model strips it to only the sentences relevant to this query. Twenty compressed chunks then fit in the token budget of five. You trade a little latency and a cheap extra model call for coverage you could not otherwise afford. This is the escape hatch when breadth is genuinely required, and it is a better answer than silently truncating.
The discipline is to allocate a token budget to each category and enforce it, so no single component can crowd out the others. What breaks at five thousand users that works at ten is, very often, an unbudgeted context window: tolerable when it is occasionally large, ruinous when it is always large and always multiplied by your concurrency.
Latency and load
A RAG answer that takes thirty seconds is a failed answer. Users abandon, and no amount of correctness recovers them. Two separate problems hide here: the pipeline is slow, and the pipeline is serial.
Find where the time goes before optimising. RAG latency is additive, because the stages run in sequence: embed the query, search, rerank, build context, generate. Generation usually dominates, and reranking is the tax people forget, since a cross-encoder over a hundred candidates is not free. Instrument each stage separately and set a budget per stage. A p95 target of five seconds means nothing until it is decomposed into the stages that consume it.
Stream the answer. This is the single highest-leverage change, because it attacks perceived latency rather than actual latency. Tokens appearing in under a second, with the full answer completing in eight, feels dramatically faster than a nine second wait staring at a spinner, even though the second number is better. Stream by default. Show retrieval progress while the model thinks.
Parallelise what is independent. Query rewriting and the keyword arm of a hybrid search do not depend on each other. Shard queries do not depend on each other. Sub-questions from a decomposition do not depend on each other. Serial code turns each of these into an addition when it could be a maximum.
Cut work for the common case. Reranking a hundred candidates when twenty would do, or running a decomposition on a simple lookup, is latency spent for nothing. The router that saves you money in the gateway saves you time here too.
Concurrency: what actually breaks at five thousand users
The ceiling you hit first is almost never your own infrastructure. It is the model provider’s rate limit, measured in tokens per minute, and when you cross it the provider does not slow down, it returns errors. A traffic spike becomes a wall of failures.
So put a queue in front of the model layer and apply backpressure: accept the request, admit it to the model when there is capacity, and if the queue is deep, degrade deliberately rather than failing. The gateway is the natural place for this, since every call already flows through it, and it is where fallback to a second provider lives when the primary is throttled.
Degrade gracefully instead of failing. Under load or partial failure, a slightly worse answer beats an error page. Skip the reranker and serve the top vector-search results. Answer from three shards when a fourth is unavailable, and say the results may be incomplete. Drop to a smaller model. Each of these is a deliberate trade you configure in advance, not an outage you explain afterwards.
Watch the cold start. After a deploy, or after the cache flush that a re-embedding cutover requires, your hit rate is zero. Latency and cost both jump to their worst case at exactly the moment the system is least stable. Warm the cache with your most common queries before sending real traffic to a new deployment.
When the question depends on the last one
Conversation breaks retrieval in a way that is easy to miss, and it is a serving concern because it happens per request, on the hot path.
A user asks “what’s the default timeout in v4.2?”, gets an answer, then follows with “and in v3.9?”. That second query embeds to almost nothing useful. “And in v3.9?” has no content to retrieve on. The system searches for a fragment and returns noise, or worse, plausible noise.
The fix is to rewrite the query before retrieving it. Take the conversation history and the new turn, and use a cheap model to produce a standalone question: “what is the default request timeout in v3.9?” That rewritten query is what goes to retrieval. The user never sees it. This one step, sometimes called query contextualisation, is what makes a RAG system feel conversational rather than amnesiac, and since it is a small model call it is cheap enough to run on every turn.
Rewriting also handles the pronoun case (“why did they change it?”) and the implicit-topic case (“what about pooling?”), both of which are unretrievable as written and trivially retrievable once resolved.
That leaves the history itself, which grows every turn and competes for the context budget above. Compress it without losing what matters: keep the last two or three turns verbatim, since those carry the immediate thread, and replace older turns with a rolling summary. Preserve entities and decisions explicitly, because those are what later turns refer back to (the version under discussion, the setting being changed, a constraint the user stated ten turns ago). A summary that drops “we are talking about v3.9” has thrown away the only thing that made the next query resolvable. Summarise the prose; keep the facts.
Why the same question gives two different answers
Two users ask an identical question and get different answers. This alarms people, and the first instinct is to look for a bug in retrieval. Usually there isn’t one.
LLMs sample. At each step the model produces a probability distribution over the next token and draws from it, rather than always taking the most likely one. That is by design: it is what makes the output fluent instead of stilted. But it means an identical prompt can produce different text on two runs.
Two knobs control this, and they are often confused:
Temperature reshapes the distribution. Low temperature sharpens it toward the most likely tokens; high temperature flattens it, giving unlikely tokens a real chance. At zero it becomes near-deterministic: always take the top token.
Top-p (nucleus sampling) truncates the distribution. It keeps only the smallest set of tokens whose probabilities sum to p, then samples from that set. It removes the long tail of bad options without changing the relative odds of the good ones.
For a RAG system answering factual questions from documents, you want low temperature. There is exactly one correct default timeout for v4.2, and creative rephrasing of it has no value. Set temperature near zero and leave top-p alone. Save the higher settings for tasks where variety is the point, which a documentation assistant is not.
Two honest caveats. Low temperature is not the same as grounded. A near-deterministic model will hallucinate the same wrong answer every time, consistently and confidently. Determinism buys you reproducibility, not truth; grounding comes from retrieval and the checks in your evaluation loop. And temperature zero is not a guarantee, since batching and floating-point non-determinism on the provider’s side can still produce small variations. Do not build anything that depends on byte-identical outputs.
The upstream causes are worth ruling out too. If two identical questions retrieve different chunks, the problem is not sampling: it is a non-deterministic index (an ANN search that changed because the index was updated between the two calls), or the two users have different permissions and are genuinely seeing different documents, which is the access control working correctly.
Evaluation has to scale too
The loop from the evaluation article does not disappear at scale; it gets more expensive, so it gets more selective. You cannot run an LLM judge on a million responses a day, so you sample, scoring a representative slice rather than everything. The golden set needs governance as it grows, and the reference-free KPIs (faithfulness, relevancy, precision) become live monitoring signals, watched on a dashboard next to latency and cost, rather than only gates in CI. Context recall still runs offline against the golden set, since it needs to know what should have been retrieved.
Scale does not change what you measure. It changes how much of it you can afford to measure, and forces you to be deliberate about the sample.
Putting it together
A production RAG system is the prototype plus the operational layer that lets it survive contact with real traffic: a gateway routing every model call by cost and falling back on failure, guardrails that cap the damage a bug can do, a budgeted context window, a streamed response, a queue that degrades instead of collapsing, and an evaluation loop sampling production to keep all of it honest.
The prototype proves the system can be right. This layer is what keeps it right, fast, and affordable when a thousand people are asking at once.
Lets move to next article to understand RAG security concerns.