Vol. 6 - Why Your RAG Breaks at Scale: Corpus, Freshness & Versioning

Series: Building Production RAG

So far in this series we’ve built the pipeline (chunking, embeddings, retrieval, generation) and the evaluation loop that keeps it honest. All of it answered from the corpus we started with: around 50,000 documents spanning technical manuals, product guides, release notes, support tickets, and internal wikis, across more than a hundred product lines with three to five active versions each.

Now the corpus grows to millions of documents, and it stops holding still. Documents change daily. Versions multiply. A better embedding model ships. And every one of those events can leave your index quietly, invisibly wrong.

This article is about the data plane: how you keep the index correct and current as it grows. The next one is about the serving plane, how you answer fast and affordably once thousands of people are asking at once.

The uncomfortable truth is that at scale, nothing breaks loudly. The system does not go down. It just starts returning last quarter’s answer, or the wrong version’s answer, and nobody notices for a month.

Ingestion at scale

At 50,000 documents you can re-embed and re-index the whole corpus overnight and never think about it. At several million you cannot. Ingestion, storage, and retrieval all have to change.

Ingestion becomes a pipeline, not a job

A one-off script that loads everything is fine once. In production, documents arrive and change constantly: new product versions, edited docs, deprecated features. You need incremental ingestion that processes only what changed, runs continuously, and is idempotent, so a retry doesn’t create duplicates. In practice that means tracking a content hash per document, re-embedding only when the hash changes, and deduplicating before anything reaches the index. Re-embedding an unchanged document is wasted money at scale, which is the first reason caching matters.

The first load is its own project

Incremental ingestion handles the steady state. It does not handle the day you start. Backfilling millions of documents is a bounded but non-trivial job: you are rate-limited by the embedding provider, and doing it naively means a single sequential loop that takes a week.

Treat it as a batch pipeline, not a script. Parallelise across workers, batch documents into the largest requests the embedding API accepts (per-request overhead dominates at small batch sizes), and checkpoint your progress, so that a failure at eighty percent resumes at eighty percent rather than zero. Expect to be throttled and back off gracefully rather than crashing. The same machinery serves you later, because a full re-embed (below) is the same job again.

Sharding the index

A single vector index has limits, in memory and in query latency, once vectors run into the millions. The fix is sharding: split the index into pieces along a natural boundary (product, tenant, or document type) and route each query to the shard or shards it needs. This keeps every index small enough to stay fast, and it lets you scale by adding shards rather than buying a bigger machine. Add replication on top so a single node failure doesn’t take retrieval down.

The choice of ANN index also starts to bite. (We covered approximate nearest-neighbour search earlier in the series.) Graph indexes like HNSW are fast but memory-hungry, and at millions of vectors that memory cost is real; you may trade a little recall for a more memory-efficient index. That is a measurable trade-off, which means it runs through the eval loop from the previous article: A/B the index change on context recall before you commit to it.

Sharding only pays off if you query the shards concurrently. Fan the query out to every relevant shard at once, then merge the results; do it sequentially and you have simply added latency. The same applies to any independent work in the request: query rewriting, a keyword search running alongside the vector search, or a multi-hop question fanned out into sub-queries. All of it should run in parallel and be gathered at the end.

This is where an orchestration framework such as LangGraph earns its place. It models the request as a graph, so independent nodes fan out concurrently and fan back in at a join, and it gives you retries and partial-failure handling on each branch (if one shard times out, you can return results from the rest rather than failing the whole query). To be clear about what it does and doesn’t buy you: the parallelism comes from your architecture, not from the framework. LangGraph makes that structure explicit and easier to operate. You can hand-roll the same fan-out with async code, and plenty of teams do.

Who is allowed to see what

Optional. This is the section people look for under “multi-tenancy,” but tenancy is only the strictest case of a broader question, and the first thing to establish is whether you need any of it.

Ask this before anything else: is there any document in this corpus that some user should not see? If everyone can read everything, stop. One index, no filter, nothing to build. A lot of internal corpora genuinely end here, and adding access control you do not need is a real cost for no benefit.

If the answer is yes, it is usually a small number of document classes rather than the whole corpus. Support tickets containing customer data. Documentation for a version that has not shipped yet, which sits in your docs system months before launch and will be indexed like everything else. A restricted wiki space. Everything else stays open.

Two ways to enforce it, and they answer different questions.

Labels on chunks. One index. Every chunk carries the audience allowed to read it, and every query carries the user’s groups. Retrieval returns only chunks where the two intersect. This is the right answer for one organisation with graded visibility, and the reason is worth spelling out. Suppose manuals are open to all, support tickets are restricted to the support org, and unreleased version docs are limited to the product team building them. Alice, in support, reads manuals and tickets but not the unreleased docs. Bob, in product, reads manuals and the unreleased docs but not tickets. They share the manuals and differ on everything else. Neither one’s access is a subset of the other’s.

That is what breaks shards. A chunk lives in exactly one index, so which index do the manuals go in, Alice’s or Bob’s? Both need them. You would end up with an index per combination of permissions, and three document classes already give you eight of them. With labels there is no combinatorics at all: the manual chunk carries [open] and exists once. Alice’s token says [open, support], Bob’s says [open, unreleased], both intersect open, and both retrieve it. Add a fourth restricted class and you add one label, not eight indexes.

A shard per tenant. A separate index per customer. Alice’s query is routed to Customer A’s index and never touches Customer B’s, so there is no filter to leak, because there is nothing in the index to exclude.

This works only when access is genuinely disjoint: Customer A’s documents and Customer B’s documents, with nothing shared between them. Then a chunk really does belong in exactly one box and the combination problem never arises. It is heavier, though. N customers means N indexes to build, monitor, and re-embed, and the moment you do have a shared document you are back to awkward choices: copy it into every shard, or keep a global shard and query it alongside. It earns that weight when the boundary is contractual or regulatory, and not otherwise.

Most internal systems want labels. Shards are for true multi-tenancy.

Two things have to go right, and they are unrelated. The label has to be true, and someone has to check it. Think of a locked door: the lock has to match the real key list, and somebody has to actually try the handle. A perfect lock on a propped-open door protects nothing.

Inherit, never invent. Whatever system holds the document already knows who can read it. Ask it, and copy that permission onto every chunk. If you invent your own labels instead, you have built a second permission model, and the day someone changes the permission upstream your label becomes a lie. On the query side, the user’s groups come from the validated SSO token, server-side, never from the client. Both sides must use the same group names, or the filter silently matches nothing.

The filter is an argument, so it can be forgotten. A search call without it does not error. Every chunk still carries its perfectly inherited label; nobody looked. The store had no instruction to compare anything, so it returns the nearest neighbours from the whole corpus, restricted chunks and all, and it looks exactly like a successful query.

index.search(vec, filter={"acl": user.groups})    # the label is checked
index.search(vec)                                 # the label is ignored

That second line is what a new feature, a debug script, or a new engineer writes six months from now. Postgres row-level security cannot fail this way, because the engine applies the policy whether the query asked or not. A vector store filter fails open. So do not leave it optional: wrap the retriever so it injects the filter from the session, and make unfiltered search unreachable from application code. One place to get right, rather than every call site, forever.

Two things leak even when retrieval is correct. Suppose the labels are right and the filter is wrapped, so every search returns exactly what the user is allowed to see. You can still leak, in two places that have nothing to do with retrieval doing its job.

a) The cache skips retrieval entirely. Alice, in support, asks about the P1 escalation process. The system retrieves support-only chunks, answers, and caches the result against the question. Bob, who is not in support, asks the same question, hits the cache, and receives Alice’s answer. No search ran, so no filter ran. Retrieval made no mistake; it was bypassed. So key the cache by question and permission scope, never by question alone. Bob then gets a cache miss, goes through retrieval, and correctly finds nothing.

b) Permissions go stale. Alice leaves the support team and is removed from the group upstream. If your index still carries her old access, or her existing token still claims the group until it expires, the filter runs perfectly against out-of-date information. Same story when a document’s permission tightens upstream and your chunk label still says open. ACL changes need the same near-real-time invalidation as document changes: re-stamp the affected chunks, and keep token lifetimes short or resolve groups fresh at query time. Deletions are the same problem in a harsher form. A document removed for legal reasons whose vectors still sit in your index will keep surfacing in answers, because deleting it at the source does not delete it from your index.

The pattern is worth naming: access control is not a single gate at retrieval. It leaks in the layer that skips retrieval, and in the data that feeds it.

Finally, the mundane version of tenancy: quotas. One tenant running an enormous backfill should not degrade everyone else’s latency. Per-tenant limits at the ingestion and query layers keep one customer’s bad afternoon from becoming everybody’s.

The three caches

Three caches are worth building, and they solve different problems.

• Embedding cache, so unchanged documents are never re-embedded.

• Semantic cache, so a question that is near-identical to one asked minutes ago returns the stored answer instead of running the full pipeline again. Note near-identical, not identical: the cache matches on embedding similarity, so “what’s the default timeout in v4.2” and “v4.2 default request timeout?” hit the same entry, which a plain string cache would miss entirely.

This is where a documentation corpus is unusually well suited to caching. Questions cluster hard: a release goes out and everyone asks about the same handful of changes, and the same onboarding questions recur every week. Cache hit rates in the tens of percent are realistic, and every hit skips retrieval and generation entirely, which is the cheapest latency and cost win available.

There is a third cache, and it is the one most teams miss. Prompt caching (sometimes prefix caching) applies inside a call that the semantic cache did not catch. Every request you send begins with the same long, unchanging prefix: the system prompt, the instructions, the few-shot examples, the output format. Providers will cache that prefix and charge a fraction of the input price for it on subsequent calls, so long as it is byte-identical and sits at the front of the prompt.

The practical consequence is that prompt layout is a cost decision. Put everything static at the front (system prompt, instructions, examples) and everything variable at the back (retrieved chunks, conversation history, the user’s question). Reorder them, or interpolate a timestamp into the system prompt, and you have silently broken the cache on every call. This is also the real answer to re-sending a fifty-page manual with every request: do not. Retrieve the relevant few pages instead, and let prompt caching cover the fixed scaffolding around them.

One caution that becomes a security question at scale: cache entries must be scoped. If different users can see different documents, a shared cache can serve one user an answer built from documents another user was allowed to see. Key the cache by permission scope, not by question alone. Keeping the cache fresh is a large enough problem to take on its own, which is next.

Freshness: the two-layer staleness problem

Here is a failure that looks trivial and is not. A policy document is updated this morning. Your assistant keeps serving yesterday’s answer.

The instinct is to blame the index, but there are two places staleness hides, and fixing one still leaves you wrong:

1. The vector index. The document changed, but its embedding was never refreshed, so retrieval either returns the old chunk or fails to match the new content at all.

2. The semantic cache. Even with a perfectly fresh index, a cached answer generated from the old document sits there and gets served without retrieval ever running. The pipeline you fixed never executes.

So invalidation has to happen at both layers, in that order. Re-embed and re-index first, then evict the affected cache entries. Reverse the order and you race: the cache repopulates from the stale index and you are back where you started.

Re-indexing is the easy half, and the incremental pipeline above already does it: the content hash changes, the document is re-chunked, re-embedded, and the old vectors for that document are replaced. Deletions need explicit handling too, or a superseded document keeps getting retrieved forever. Delete its vectors, or tombstone it and filter tombstoned documents at query time.

Cache invalidation is the harder half, because of a mismatch: cache entries are keyed by question, but documents are what change. Given a modified document, you cannot tell which cached answers are now wrong without knowing which answers were built from it. So store that link. Every cache entry records the document IDs that went into it, giving you a reverse mapping from document to dependent answers. When a document changes, look up its dependents and evict exactly those. This is the piece most teams skip, and it is why a fresh index still serves stale answers.

Two backstops on top. Put a TTL on cache entries so anything the event pipeline misses expires on its own. And decide your freshness SLA deliberately: how stale is acceptable? For our versioned documentation, minutes are fine, so a batched pipeline running every few minutes is cheaper and simpler than streaming. For a policy or pricing document where a wrong answer is a real problem, you want event-driven invalidation firing the moment the document is saved. Freshness is a cost trade-off, not a free property, so pick the target and then engineer to it rather than quietly hoping.

Re-embedding the whole corpus with no downtime

Freshness handles one document changing. Sooner or later you face the other version of the problem: every document changing at once. A better embedding model ships, or you change chunking strategy, and the entire corpus has to be re-embedded.

You cannot do this in place. Vectors from two different embedding models are not comparable, so a half-migrated index is not a partly-improved index, it is a broken one: similarity scores across the two spaces are meaningless and retrieval returns nonsense. That rules out re-embedding document by document. And at millions of vectors the backfill takes hours or days, so taking the index offline and rebuilding is not available either.

The answer is blue-green indexing:

1. Build alongside. Stand up a second, empty index and backfill it with the new model while the old index keeps serving every query. Nothing changes for users.

2. Dual-write. While the backfill runs, send every incoming document change to both indexes. Otherwise the new index is already stale by the time it finishes.

3. Validate before you trust it. A new embedding model is exactly the kind of change the evaluation loop exists to gate. Run the golden set against the new index and compare context recall and precision with the old one. A newer model is not automatically better on your corpus.

4. Flip the read path. One atomic config change, not a gradual migration. Keep the old index warm for a while so you can roll straight back if the live KPIs move the wrong way.

Two things catch people at the cutover. The query embedding must switch at the same instant as the index, or you are embedding questions with the new model and searching vectors written by the old one, which is the same broken-space problem wearing a different hat. And the semantic cache must be flushed, because every entry in it was built from retrievals against the old index.

The cost is real and worth planning for: you pay to embed the entire corpus again, and you run two full indexes side by side for the duration. That is the price of not being down.

When several versions of the truth coexist

Our corpus documents four live versions of the same product. This looks like a staleness problem and it is not one, and confusing the two leads you to exactly the wrong fix.

Staleness means the old content is wrong and should be evicted. Multiple versions means every version is simultaneously correct. A user on v3.9 needs the v3.9 answer. Evict nothing.

The failure is different, and quieter. Someone asks “what’s the default request timeout?” without naming a version. Retrieval returns four near-identical chunks, one per version, because they are ninety percent the same text and sit almost on top of each other in embedding space. The model’s context now contains 30 seconds, 30 seconds, 60 seconds, and 60 seconds. It picks one, states it with confidence, and is faithful to a retrieved document while being wrong for this user. Note what your evaluation does here: faithfulness passes, because the answer is grounded in a real retrieved chunk. Only context precision drops, and only if you are watching. This is the failure that ships.

No embedding model will fix this, because the texts really are nearly identical. It has to be solved before the vector search runs.

Resolve the version before you retrieve. Every chunk carries its version as metadata at ingestion. At query time, you establish the version first, in this order: an explicit mention in the query (”in v4.2”), then the user’s known context (the version they are actually running, from their account or session), then a default. Only then do you search, pre-filtered to that version. Filter before the ANN search, not after: post-filtering lets the near-duplicates consume your top-k and then throws them away, leaving you with fewer results than you asked for.

Do not guess when it is ambiguous. If no version can be resolved, you have three honest options, and picking one is a product decision, not a technical one: answer for the latest version and say so explicitly, ask the user which version they are on, or answer for all versions and present the differences. What you must not do is silently pick one, which is precisely what an unfiltered vector search does for you by default.

Comparison questions are a different query type. “What changed in retry behaviour between v3 and v4?” deliberately needs several versions at once. Handle it by fanning out one filtered retrieval per version and merging, rather than hoping a single unfiltered search returns a balanced sample of both. And whichever path you take, label every chunk with its version in the context you hand the model, so it can attribute rather than blend. A model that can see “v4.2 documentation says X, v3.9 documentation says Y” will tell you they differ. A model handed four unlabelled chunks will average them into a confident, wrong sentence.

Retire versions deliberately. When a version reaches end of life, do not delete it, since someone may still ask about it, but drop it out of the default retrieval scope so it stops competing for top-k. Tombstone and filter, rather than delete.

Decision for this corpus. With a hundred-plus product lines and three to five versions each, product is the sharding key and version is a metadata filter inside the shard. Sharding on version alone would be a mistake: it would put v4.2 of a hundred unrelated products in the same shard while splitting a single product’s history across five, which is the opposite of the locality you want. Almost every query names or implies one product, so it routes to one shard, and the version filter then runs against a small index rather than a large one. Cross-version comparisons stay inside a single shard. Cross-product questions, which are rare, fan out and merge.

Monitoring and drift

You can no longer eyeball it. Track index freshness (how far behind live the index is), retrieval latency at p50, p95, and p99 (averages hide the tail that users actually feel), ingestion failure rates, and recall on the golden set over time, so silent degradation shows up as a chart rather than a complaint.

Detecting drift deserves its own attention, because nothing fails loudly. The system does not break; it gets worse. Three things move underneath you. Your data drifts as the corpus changes: new versions, new features, new terminology. Your traffic drifts as users ask about things they did not ask about last quarter. And your model drifts when a provider silently updates the model behind an endpoint. The symptom is identical in all three cases: KPIs sag slowly and nobody notices for a month.

So watch for the change, not just the level. Track the reference-free KPI scores as a time series and alert on a sustained drop, not a single bad day. Watch the distribution of incoming queries shift, since a spike in questions your corpus cannot answer is a content gap, not a retrieval bug. Watch retrieval confidence fall, since falling top-k similarity scores mean questions are drifting away from what you indexed. And pin your model versions through the gateway, so a provider-side update is a change you make, not a change that happens to you.

What this bought you

None of this is the interesting part of RAG. It is the part that decides whether the interesting part stays true. A prototype proves the system can be right once. This machinery, incremental ingestion, sharding, layered caches, invalidation that reaches both the index and the cache, a migration path that does not require downtime, version-aware retrieval, and monitoring that catches slow decay, is what keeps it right as the corpus grows underneath it.

The index is now correct and current. That is half the problem. The other half is answering from it fast enough that people stay, and cheaply enough that finance stays quiet, when five thousand of them are asking at once.

---

Next in this series: Why Your RAG Costs Too Much and Answers Too Slowly, on the gateway, the cost model, the context budget, and what actually breaks under concurrent load.