Vol. 10 - What Nobody Tells You About Production RAG
Series: Why Your RAG Is Failing
This series has been, until now, about technique. Chunking, retrieval, generation, evaluation, scale, cost, security. Nine articles of how to build the thing well.
This last one is about everything the techniques do not touch, and it is the article I most wanted to write, because it is the part nobody warns you about. You can execute every previous article perfectly and still ship a RAG system that fails. Not because the engineering was wrong, but because the hard problems in production RAG are not engineering problems at all. They are problems of content, of judgement, of trust, and of knowing what the tool cannot do.
The uncomfortable summary: the model is the easy part. By now it is a commodity. What is hard is everything around it, and almost none of it is discussed, because it is not fun to build and it does not demo well.
Here is what actually decides whether your RAG system lives or dies.
Your corpus is the product, and it is probably a mess
Every article in this series quietly assumed the answer was somewhere in the corpus. Retrieval’s job was to find it, generation’s job was to phrase it. That assumption is the biggest lie in RAG.
In a real organisation, the answer often is not in the corpus at all. It is in someone’s head. It is in a Slack thread from eight months ago. It is in three documents that contradict each other, none of them marked as current. It is in a wiki page last updated in 2021 that everyone knows to ignore, except your RAG system, which has no way to know.
No amount of retrieval engineering fixes a corpus that does not contain the answer. This is the single most important sentence in this article. You can tune chunk sizes forever, add a reranker, switch embedding models, and none of it matters if the information was never written down, or was written down wrong. Garbage in is still garbage out, and RAG launders that garbage into a fluent, confident, cited paragraph that is worse than no answer, because it looks trustworthy.
So the work that actually moves the needle is not model work. It is content work, and it looks like this:
Find the gaps by instrumenting for them. The questions that consistently retrieve nothing good are not a retrieval bug. They are a documentation backlog. Log every query where the top retrieval score is low, or where the user rephrased and left, and you have a ranked list of the documents your organisation needs to write. That list is worth more than any model upgrade.
Treat content debt like technical debt. Outdated docs, contradictory docs, and duplicated-across-five-systems docs are liabilities that accrue interest. Every stale document is a wrong answer waiting to be retrieved. Someone has to own deprecating them, and that someone is usually nobody, which is the problem.
Someone must own the corpus. Not the model, not the pipeline: the content. In most failed RAG projects, no single person was responsible for whether the underlying documentation was correct, current, and complete. The system was treated as a search problem when it was a knowledge-management problem. The teams that succeed appoint an owner whose job is corpus quality, and they treat writing and pruning documentation as first-class work rather than a chore.
The engineers want this to be a modelling problem, because modelling is what they enjoy. It is a content problem wearing a modelling costume.
Where RAG is structurally the wrong tool
RAG is retrieval plus generation, and both halves have hard limits that no amount of tuning removes. Knowing these boundaries is what separates people who have shipped from people who have read the blog posts, because the failure here is not that RAG does it badly. It is that RAG cannot do it at all, and every hour spent trying is wasted.
Exhaustive retrieval and aggregation. “Sum every expense in this 2,000-page report.” “How many tickets mentioned this bug last quarter?” RAG retrieves the top-k most similar chunks. It is structurally incapable of retrieving all matching chunks, and it cannot count or do arithmetic reliably over what it does retrieve. The right answer is to not use RAG: extract the data into a structured store and run a query. This is a boundary, not a tuning problem, and it is the one people waste the most time fighting.
Questions that need relationships, not similarity. “Which components does this service depend on, transitively?” Vector search finds text that is similar, not text that is connected. Multi-hop relationship questions want a graph, and the honest answer is often GraphRAG or a real graph database, not a better embedding model.
A single, stable, in-scope document. If the answer always lives in one 40-page contract that fits in the context window, you may not need retrieval at all. Long-context models can read the whole thing. RAG adds machinery, failure modes, and a retrieval step that can only hurt, in exchange for solving a problem you did not have.
A fixed, deterministic workflow. If the task is “when the form has field X, do Y,” that is code, not a language model. Wrapping a deterministic rule in a probabilistic model buys you unpredictability and cost, and removes the one thing you wanted: a guarantee
The pattern across all four: RAG is for open-ended questions over a large, changing, unstructured corpus. The moment your problem is exhaustive, relational, small, or deterministic, RAG is the wrong shape, and reaching for it anyway is the most expensive mistake in this whole series.
Build versus buy: the question the series skipped
Nine articles told you how to build it. Not one asked whether you should. That was deliberate, because the honest answer belongs here, at the end, once you have seen everything building it entails.
Most teams should buy. Glean, Vertex AI Search, Azure AI Search, and Bedrock Knowledge Bases exist, they are good, and they handle the chunking, embedding, retrieval, and scaling that the last nine articles described in detail. If your needs are ordinary, reproducing all of that in-house is a way to spend a year rebuilding a commodity.
So the real question is not “can we build it” but “is there anything about our situation that a vendor cannot handle.” And the value of having read this series is that you now know exactly what those things are:
Buy unless one of these is true:
Your corpus or access model is genuinely unusual. Deeply nested permissions, an ownership model no vendor anticipated, document types nobody supports. The access-control article was long for a reason: this is where off-the-shelf tools most often break.
Your evaluation or quality bar is something a black box cannot meet. If you need to measure faithfulness and abstention against a bespoke golden set and gate deployments on it, and the vendor gives you a similarity score and a shrug, you may have to own the pipeline to own the quality.
Your data cannot go where the vendor needs it. Regulatory or contractual constraints that force self-hosting. The security article’s hybrid architecture was this case.
Notice that none of these three is “we have good engineers.” Good engineers are a reason you can build, not a reason you should. Build when your situation is genuinely unusual, and the series you just read is the map of exactly how unusual it would have to be.
Trust is the real product, and it is asymmetric
Here is the failure mode that kills RAG systems that are technically excellent. The system is accurate ninety-five percent of the time. In week one it hallucinates two confident, cited, wrong answers in front of the wrong people. It is now dead. Nobody trusts it, nobody uses it, and the ninety-five percent never gets a chance to matter.
Trust is asymmetric. One confident wrong answer costs more than a hundred right ones earn. This is not fair and it is not going to change, because it is how humans relate to tools that claim authority. A calculator that is wrong one time in twenty is not a calculator anyone uses. And a RAG system, with its fluent prose and its citations, claims authority with every answer, which raises the bar rather than lowering it.
This reframes what you are optimising. You are not optimising average accuracy. You are optimising the worst case the user actually sees, because that is what sets trust. Which means:
Knowing when to say “I don’t know” is worth more than another point of accuracy. The abstention discipline from the generation article is not a nicety. It is the core trust mechanism. A system that reliably says “that is not in my documents” is trusted far more than one that is slightly more accurate but occasionally fabricates. Calibrated humility beats confident brilliance.
Show your work, always. Citations are not decoration. They let the user verify, which converts “trust the machine” into “check the source,” which is the only trust that survives contact with a wrong answer. A cited wrong answer is caught. An uncited wrong answer is believed.
Set expectations honestly at the door. A system introduced as “an assistant that helps you find documents, and can be wrong” survives mistakes that a system introduced as “ask it anything” does not. Overpromising at launch is how you spend your trust before you have earned it.
The teams that win treat trust as the product and accuracy as one input to it. The teams that lose optimise a benchmark and wonder why nobody uses the thing that scored so well.
Know your ceiling
Every RAG system has a maximum quality it can reach, set by things you mostly do not control, and pretending otherwise wastes quarters of effort on the last few points that were never available.
Your ceiling is set by the quality of your corpus, the answerability of the questions people actually ask, the inherent ambiguity of natural language, and the limits of the models you can afford to run. If your corpus is incomplete, no model reaches a hundred percent, because the information is not there. If users ask questions the documents were never meant to answer, retrieval cannot invent the answer. These are ceilings, not bugs.
Knowing your ceiling changes what you do. Below it, engineering pays: better chunking, reranking, and evaluation move the number. Near it, engineering stops paying, and the only lever left is content (write the missing docs) or scope (be honest about what the system does not cover). Teams that do not know where their ceiling is keep throwing model changes at a content problem, and the number does not move, and nobody understands why.
The mature move is to measure the ceiling deliberately. Take a set of questions whose answers you know are in the corpus and answerable, and measure against those. That tells you how good your system is. The gap between that and your overall score tells you how much of your problem is content, not engineering, and therefore where the next quarter should actually go.
What the whole series was really about
Step back from ten articles and the shape becomes clear. The techniques, all of them, are in service of one thing: a system that is trusted because it is reliably right, honest when it is uncertain, and used by people who understand what it is for.
Chunking and retrieval get the right information in front of the model. Evaluation keeps it honest. Scale, cost, and security keep it alive under real conditions. And this article is the reminder that all of that is necessary and none of it is sufficient, because the last mile is not technical. It is a good corpus, an honest scope, a system that knows what it does not know, and users who trust it because it has earned it.
The model is a commodity. The engineering is learnable, and this series taught it. The corpus, the judgement, and the trust are the hard part, and they are the part that is actually yours to get right.
That is what nobody tells you. Now you know, and you have the whole map.
This is the final article in the Building Production RAG series. The earlier articles cover the techniques this one assumes: chunking, retrieval, generation, evaluation, scaling, cost and latency, and security.