So far in this series we’ve built the pipeline: chunking and embeddings, retrieval stack and generation. All of that rests on an assumption we haven’t tested yet, that the answers coming out the other end are actually right. This article is about how you know that, and how you use that knowledge to make the system better over time.
Building a RAG system is the easy part. Knowing whether it’s still right on any given day is the hard part. A demo answers ten questions you chose; production answers thousands you didn’t. Models drift, data changes, someone edits a prompt, and an answer that was correct last week is quietly wrong this week, with no error to flag it.
So evaluation can’t be a one-time gate before launch. It’s a continuous loop: measure quality, find where it breaks, fix it, redeploy, then measure again. Everything below is a piece of that loop, applied to the RAG system we’ve built across this series and the corpus of roughly 50,000 documents it answers from.
The three layers of measurement
You can’t improve what you can’t measure, so we start there. Measurement runs in three layers, each catching what the others miss.
1. Offline, static data
We keep a fixed golden dataset: a curated set of questions, each paired with its known-good answer and the documents that should be retrieved to support it. These are the cases the system must always get right: common questions, known edge cases, and the ones that have burned us before.
Size it by coverage, not volume. There’s no magic number; what matters is that every query type is represented across every document category (version, documentation, feature), along with your known failures. For a corpus this size that lands in the low hundreds; we use around 500. A tight, well-chosen set beats a large random one, because every example has to earn its place. The same inputs run every time, so the only variable is the change you’re testing. This is your regression test.
2. Online, dynamic data
The golden set is fixed by definition, so it can't cover the live distribution of what people actually ask. For that we evaluate on real production traffic, where there's no ground-truth answer to compare against. The first is LLM-as-a-judge: a separate model scores live responses against the four KPIs defined in the next section (judging every response is expensive, so we sample).. The second is user signals: explicit ones like thumbs, corrections, and support escalations, and implicit ones like follow-up questions, reformulations, and abandoned sessions. Together they catch failures that only appear against real, messy, unanticipated questions. This is the layer that surfaces problems in production, and where the improvement loop later begins.
Between the fixed golden set and live traffic sits a third option that is easy to miss: replay. Take a sample of real logged queries, freeze them, and run any candidate through them offline. It is not the golden set, because these questions have no curated answer. You are not scoring correctness, you are comparing two systems' outputs and asking which is better, usually with the judge. And it is not live evaluation, because the query set is fixed, so four chunking strategies can be compared against the identical questions instead of each meeting a different day's traffic. That reproducibility is the point: most evaluation is comparing options, not validating one finished thing. The one catch is that it needs logged traffic, so it is a tool you earn once the system is live, not one you have on day one.
3. Human-in-the-loop, the living dataset
Automated scoring is fast but imperfect. So a sample of those scored production answers, weighted toward the ones the judge rated low or users flagged, gets reviewed by people, who correct what the automated scoring got wrong. The corrected examples are then added back into the golden dataset. Today’s hard production case becomes tomorrow’s regression test, and offline evaluation gets stronger every cycle. The golden set isn’t static; it grows toward the questions that actually matter.
online = where problems surface, offline = verify/regression, human = feedback
Before metrics: build the ground truth
Layer 1 called for a golden set: questions paired with known-good answers and the documents that should be retrieved. That sentence hides a lot of work, and the work is what decides whether the metrics that follow mean anything. A metric is only as honest as the ground truth under it: score against a weak reference and you get a confident measure of the wrong thing.
So for each benchmark question, record what a correct answer must contain and where the evidence lives. Not a polished answer, the ingredients of one:
question
reference answer (one acceptable phrasing, not the only one)
supporting document IDs
supporting chunk IDs
document version
answerable / unanswerable
question type
required facts (must be present)
optional facts (may be present)
forbidden / outdated facts (must not appear, usually another version's value)For example:
Question: Is collector v4.2 compatible with gateway v3.9?
Required evidence: - collector v4.2 uses protocol 3
- gateway v3.9 supports protocol 2
Conclusion: No
Supporting chunks: collector_release_notes_v4.2
gateway_compatibility_matrix_v3.9Building the set
Each step below exists to catch a specific blind spot. Skip a step and you go blind to the thing it was there to see.
What we measure: four core metrics
We score along the RAGAS framework. RAGAS offers more than a dozen metrics (answer correctness, semantic similarity, noise sensitivity, context entity recall, and others), but more isn’t better. We rely on four, because together they cover the distinct ways a RAG answer fails, and each one points at a specific part of the pipeline. The fourth column is why this matters: a low score doesn’t just say the system is wrong, it says where it’s wrong.
The first two judge the generated answer; the last two judge retrieval. (Add a fifth metric only when you hit a failure mode these four don’t capture, not by default.)
A quick word on “upstream,” since it runs through the rest of this piece. RAG works in stages: retrieve, then generate. Retrieval sits upstream of generation, so anything retrieval gets wrong flows downstream into the answer. Take a concrete question against our corpus: “What’s the default request timeout in v4.2?” The corpus holds a config doc for every release. If retrieval pulls the v3.9 doc instead of v4.2, context recall fails. If v4.2’s doc comes back buried under unrelated config pages, context precision drops. If the model states a number the doc doesn’t contain, faithfulness fails. If it explains timeouts in general instead of giving the value, answer relevancy fails. Here’s the trap: the answer can be perfectly faithful to the v3.9 doc and still be wrong, because the failure was upstream, in retrieval, and the generator faithfully summarised the wrong document. That’s why we score all four and localise the failure rather than collapsing it into one number.
Offline and online don’t measure the same set. Offline, against the golden set, we have ground truth (the expected answer and the documents that should have been retrieved), so all four are available, including context recall, which by definition needs to know what should have come back. Online there’s no reference, so we’re limited to the reference-free metrics: the ones a judge can score from just the question, the retrieved context, and the answer, with no correct answer to compare against. Faithfulness, answer relevancy, and context precision are reference-free. Context recall needs a reference, the complete set of documents that should have been retrieved, so it only runs offline.
Two things the four metrics miss: citation and abstention
The four metrics score whether an answer is grounded and relevant. They do not directly score two behaviours this series treats as central to trust, so add them explicitly.
Citation correctness. An answer can be faithful and still cite the wrong source, and a wrong citation is worse than none, because it survives a spot check. Measure two things against the ground truth: does every claim carry a citation, and does the cited chunk actually support the claim. On the golden set you have the supporting chunk IDs, so this is checkable directly. A drop here is a regression even when faithfulness holds.
Abstention. The golden set includes unanswerable questions for one reason: to measure whether the system declines when it should. Two error rates, and they trade off. The false-answer rate is how often it answers an unanswerable question, the expensive failure. The over-abstention rate is how often it declines a question it could have answered, which erodes usefulness. Track both, because a system tuned to never be wrong by always saying “I don’t know” scores perfectly on the first and is useless.
Evaluation as a pipeline gate (CI/CD)
None of this works if running it depends on someone remembering to. So the golden-set evaluation runs automatically in CI, like a unit test suite. Every change (code, config, prompt, model) triggers the suite, and a change that drops any KPI below its threshold fails the build and doesn’t merge.
This is the difference between evaluation as an aspiration and evaluation as a guarantee. It also makes the metrics actionable rather than decorative: if a commit tanks context recall, the engineer sees it in the same place they’d see a failing test, before it ever reaches a user.
For the gate to catch a bad change, it first has to know the change happened. That's why every part that affects quality gets versioned, not just the prompt. Next part talks about that.
Everything is a versioned artifact
Prompts change most often, so versioning tends to start and stop with them. But almost every part of a RAG system shapes output quality and can change on its own, and each deserves the same treatment: a versioned artifact with a tagged version, its evaluation scores attached, and rollback always available.
The golden set is versioned for a different reason than the rest. A new reranker changes the system; a new golden set changes the yardstick. Pin every score to the golden-set version it was measured against, or you will not be able to tell a system that got worse from a test that got harder.
Evaluation Cascade
The discipline is identical whether you're swapping an embedding model, tuning a reranker, or rewriting a prompt. Every change runs through four checks, cheapest first. Each has to pass before the next is worth running.:
Offline golden set, in CI, in seconds. Catches regressions on the cases you already know matter. Every change runs it.
Replay on real logged queries, offline, in minutes. Catches what the golden set’s fixed questions miss, and lets you compare candidates on the same real distribution.
Shadow on live traffic, no user impact. The candidate runs in parallel with the current system on the same live questions; both answer every real query, only the current system’s answers are served. The candidate’s answers are logged, scored on the reference-free KPIs, and compared against the live system’s. This tests the change against the real distribution of what people actually ask, with zero user impact. If it regresses, no one ever saw it.
A/B test. Once it clears shadow, serve the candidate to a slice of traffic and keep the rest on the current version as control. Compare on the KPIs and on product guardrails until the difference is statistically meaningful rather than noise. Only then does it roll out fully.
Concretely, a change does not ship if any of these regress against the current version:
Do not deploy if:
- context recall drops materially (retrieval got worse)
- faithfulness falls below threshold (answers less grounded)
- citation correctness regresses (sources wrong or missing)
- false-answer rate on unanswerables rises (abstention got worse)
- p95 latency exceeds the product SLO (too slow, regardless of quality)
- cost per query exceeds budget (too expensive to serve)The last two matter as much as the first four. A change that lifts every quality KPI but pushes p95 past the SLO is not a win, it is a regression the KPIs alone would have waved through. Quality and product guardrails gate together.
Closing the loop: from signals to improvement
Measurement is half the system. Evaluation that nobody acts on is just a dashboard. The other half is turning what you measure into what you fix. That improvement loop doesn’t introduce anything new; it runs on top of the three layers above, each playing a different role.
It begins at the online layer, because production is where live problems show up, as the signals described earlier: the judge scores and the explicit and implicit user signals. Signals tell you that something is wrong. They don’t tell you what.
That’s where the four KPIs earn their keep a second time. Every flagged case gets categorised by which metric it failed, and the “fix it in” column of the table above tells you where to go. Our v4.2 timeout case is a context-recall failure: the answer was faithful, but to the wrong version’s document, so the fix is in retrieval, not a prompt reword. Categorising by KPI is what keeps you from fixing the wrong layer.
You then verify the fix at the offline layer (re-run it against the golden set, the regression check) before shipping it through shadow and A/B. And the human layer closes the cycle: reviewed corrections become new golden-set cases, so the next pass starts already protected.
So the loop is concrete, and it maps straight onto the three layers: a problem surfaces online (signals), you categorise it by KPI, prioritise by frequency × impact, fix the stage the KPI points to, verify offline against the golden set, then redeploy through shadow and A/B. Online detects, offline verifies, humans feed the corrections back. Then it repeats.
The same discipline applies to chunking
Notice where two of those failures route: chunking. If categorisation keeps pointing at context recall, the fix isn’t a better prompt; it’s the chunking strategy we covered in Article 2. The same eval discipline applies there. A new chunking approach is just another versioned change: A/B it against the golden set on context recall and precision before adopting it. Evaluation isn’t a stage bolted onto the end of the pipeline; it’s the instrument you use to tune every stage of it.
The loop
Put together, this is a cycle, not a checklist. Offline catches regressions. Online catches the unknowns. Humans correct both and grow the golden set. CI enforces the gate. The four KPIs tell you not just whether the system is right but where it’s wrong, and the feedback loop turns that into prioritised fixes that get measured before they ship.
None of this makes the system perfect. What it makes it is accountable and improvable: you can say at any point how good it is, prove a change is an improvement before users see it, and know exactly where to look when it isn’t. For a system people make decisions on, that’s the whole game.
Next in this series: From Prototype to Production, on what changes when the corpus, the traffic, and the team all scale up, and how this evaluation loop has to scale with them.