Vol. 5 - Evaluating and Improving RAG in Production
The systematic loop for keeping a RAG system accurate, and making it better, over time.
So far in this series we’ve built the pipeline: chunking and embeddings, retrieval stack and generation. All of that rests on an assumption we haven’t tested yet, that the answers coming out the other end are actually right. This article is about how you know that, and how you use that knowledge to make the system better over time.
Building a RAG system is the easy part. Knowing whether it’s still right on any given day is the hard part. A demo answers ten questions you chose; production answers thousands you didn’t. Models drift, data changes, someone edits a prompt, and an answer that was correct last week is quietly wrong this week, with no error to flag it.
So evaluation can’t be a one-time gate before launch. It’s a continuous loop: measure quality, find where it breaks, fix it, redeploy, then measure again. Everything below is a piece of that loop, applied to the RAG system we’ve built across this series and the corpus of roughly 50,000 documents it answers from.
The three layers of measurement
You can’t improve what you can’t measure, so we start there. Measurement runs in three layers, each catching what the others miss.
1. Offline, static data
We keep a fixed golden dataset: a curated set of questions, each paired with its known-good answer and the documents that should be retrieved to support it. These are the cases the system must always get right: common questions, known edge cases, and the ones that have burned us before.
Size it by coverage, not volume. There’s no magic number; what matters is that every query type is represented across every document category (version, documentation, feature), along with your known failures. For a corpus this size that lands in the low hundreds; we use around 500. A tight, well-chosen set beats a large random one, because every example has to earn its place. The same inputs run every time, so the only variable is the change you’re testing. This is your regression test.
2. Online, dynamic data
The golden set is fixed by definition, so it can't cover the live distribution of what people actually ask. For that we evaluate on real production traffic, where there's no ground-truth answer to compare against. The first is LLM-as-a-judge: a separate model scores live responses against the four KPIs defined in the next section (judging every response is expensive, so we sample).. The second is user signals: explicit ones like thumbs, corrections, and support escalations, and implicit ones like follow-up questions, reformulations, and abandoned sessions. Together they catch failures that only appear against real, messy, unanticipated questions. This is the layer that surfaces problems in production, and where the improvement loop later begins.
3. Human-in-the-loop, the living dataset
Automated scoring is fast but imperfect. So a sample of those scored production answers, weighted toward the ones the judge rated low or users flagged, gets reviewed by people, who correct what the automated scoring got wrong. The corrected examples are then added back into the golden dataset. Today’s hard production case becomes tomorrow’s regression test, and offline evaluation gets stronger every cycle. The golden set isn’t static; it grows toward the questions that actually matter.
online = where problems surface, offline = verify/regression, human = feedback
What we measure: four core metrics
We score along the RAGAS framework. RAGAS offers more than a dozen metrics (answer correctness, semantic similarity, noise sensitivity, context entity recall, and others), but more isn’t better. We rely on four, because together they cover the distinct ways a RAG answer fails, and each one points at a specific part of the pipeline. The fourth column is why this matters: a low score doesn’t just say the system is wrong, it says where it’s wrong.
The first two judge the generated answer; the last two judge retrieval. (Add a fifth metric only when you hit a failure mode these four don’t capture, not by default.)
A quick word on “upstream,” since it runs through the rest of this piece. RAG works in stages: retrieve, then generate. Retrieval sits upstream of generation, so anything retrieval gets wrong flows downstream into the answer. Take a concrete question against our corpus: “What’s the default request timeout in v4.2?” The corpus holds a config doc for every release. If retrieval pulls the v3.9 doc instead of v4.2, context recall fails. If v4.2’s doc comes back buried under unrelated config pages, context precision drops. If the model states a number the doc doesn’t contain, faithfulness fails. If it explains timeouts in general instead of giving the value, answer relevancy fails. Here’s the trap: the answer can be perfectly faithful to the v3.9 doc and still be wrong, because the failure was upstream, in retrieval, and the generator faithfully summarised the wrong document. That’s why we score all four and localise the failure rather than collapsing it into one number.
Offline and online don’t measure the same set. Offline, against the golden set, we have ground truth (the expected answer and the documents that should have been retrieved), so all four are available, including context recall, which by definition needs to know what should have come back. Online there’s no reference, so we’re limited to the reference-free metrics: the ones a judge can score from just the question, the retrieved context, and the answer, with no correct answer to compare against. Faithfulness, answer relevancy, and context precision are reference-free. Context recall needs a reference, the complete set of documents that should have been retrieved, so it only runs offline.
Evaluation as a pipeline gate (CI/CD)
None of this works if running it depends on someone remembering to. So the golden-set evaluation runs automatically in CI, like a unit test suite. Every change (code, config, prompt, model) triggers the suite, and a change that drops any KPI below its threshold fails the build and doesn’t merge.
This is the difference between evaluation as an aspiration and evaluation as a guarantee. It also makes the metrics actionable rather than decorative: if a commit tanks context recall, the engineer sees it in the same place they’d see a failing test, before it ever reaches a user.
Everything is a versioned artifact
Prompts change most often, so versioning tends to start and stop with them. But almost every part of a RAG system shapes output quality and can change on its own, and each deserves the same treatment: a versioned artifact with a tagged version, its evaluation scores attached, and rollback always available.
The discipline is identical whether you're swapping an embedding model, tuning a reranker, or rewriting a prompt. On top of that automatic golden-set check in CI, each change is then validated in two more stages.
Shadow first. The new version under test, the candidate, runs in parallel with the current system on the same live user questions. Both answer every real query; only the current system’s answers are served. The candidate’s answers are logged, scored on the reference-free KPIs, and compared against the live system’s. This tests the change against the real distribution of what people actually ask, with zero user impact. If it regresses, no one ever saw it.
Then a real A/B test. Once it clears shadow, we serve the candidate to a slice of traffic and keep the rest on the current version as control. We compare the two on the KPIs and on product guardrails (latency, cost, and implicit signals like follow-up and reformulation rates) until the difference is statistically meaningful rather than noise. Only then does it roll out fully.
The rule across both stages: a change is promoted only if it improves the KPI it targets without regressing the others. A retrieval tweak that lifts context recall but quietly drops faithfulness is not a win.
Closing the loop: from signals to improvement
Measurement is half the system. Evaluation that nobody acts on is just a dashboard. The other half is turning what you measure into what you fix. That improvement loop doesn’t introduce anything new; it runs on top of the three layers above, each playing a different role.
It begins at the online layer, because production is where live problems show up, as the signals described earlier: the judge scores and the explicit and implicit user signals. Signals tell you that something is wrong. They don’t tell you what.
That’s where the four KPIs earn their keep a second time. Every flagged case gets categorised by which metric it failed, and the “fix it in” column of the table above tells you where to go. Our v4.2 timeout case is a context-recall failure: the answer was faithful, but to the wrong version’s document, so the fix is in retrieval, not a prompt reword. Categorising by KPI is what keeps you from fixing the wrong layer.
You then verify the fix at the offline layer (re-run it against the golden set, the regression check) before shipping it through shadow and A/B. And the human layer closes the cycle: reviewed corrections become new golden-set cases, so the next pass starts already protected.
So the loop is concrete, and it maps straight onto the three layers: a problem surfaces online (signals), you categorise it by KPI, prioritise by frequency × impact, fix the stage the KPI points to, verify offline against the golden set, then redeploy through shadow and A/B. Online detects, offline verifies, humans feed the corrections back. Then it repeats.
The same discipline applies to chunking
Notice where two of those failures route: chunking. If categorisation keeps pointing at context recall, the fix isn’t a better prompt; it’s the chunking strategy we covered in Article 2. The same eval discipline applies there. A new chunking approach is just another versioned change: A/B it against the golden set on context recall and precision before adopting it. Evaluation isn’t a stage bolted onto the end of the pipeline; it’s the instrument you use to tune every stage of it.
The loop
Put together, this is a cycle, not a checklist. Offline catches regressions. Online catches the unknowns. Humans correct both and grow the golden set. CI enforces the gate. The four KPIs tell you not just whether the system is right but where it’s wrong, and the feedback loop turns that into prioritised fixes that get measured before they ship.
None of this makes the system perfect. What it makes it is accountable and improvable: you can say at any point how good it is, prove a change is an improvement before users see it, and know exactly where to look when it isn’t. For a system people make decisions on, that’s the whole game.
Next in this series: From Prototype to Production, on what changes when the corpus, the traffic, and the team all scale up, and how this evaluation loop has to scale with them.