Vol. 4 - Why Your RAG Generation is Failing -Augmentation, Prompt Construction & Synthesis

Series: Building Production RAG

In the previous articles we mapped where retrieval fails and the strategies that fix it: chunking, embeddings, vector search, hybrid search, RRF, and reranking. It answered one question well: did the right chunk make it into the candidate set? This article moves to the next leg of RAG, generation, and a harder truth. Retrieval only sets a ceiling. It decides what is possible. Generation decides how much of that ceiling you actually reach, and it is where most of the loss quietly happens.

The reason is structural. A correct chunk sitting in the context is necessary but not sufficient. The model still has to notice it, prefer it over what it learned in training, and combine it with the other chunks. Each of those steps can fail on its own, with the right answer already in front of the model. So the failures in this article are not retrieval misses. They are the failures that happen after retrieval succeeded.

A note on the structure of this piece. It is organised by failure, not by pipeline step. The question a reader actually has is “my RAG gives wrong answers, why, and where do I fix it?”, so each section names a way the answer goes wrong and points at the lever that fixes it.

First, the vocabulary

Most “RAG is wrong” complaints stay vague because the failures do not have names. Two pieces of vocabulary make every later point sharper.

The first is the distinction between parametric memory and in-context knowledge. Parametric memory is what the model learned during training, baked into its weights. In-context knowledge is what you hand it at query time: your retrieved chunks. The entire bet of RAG is that the model will prefer the chunks you retrieved over what it already “knows”. It does not always honour that bet. We will come back to this, because it is the central generation failure and several of the levers below exist only to enforce it.

The second is a small taxonomy of wrong answers. Wrong answers come in two flavours worth separating. Some contradict your documents (intrinsic: the chunk says 55°C, the model says 70). Some are simply invented, not in your documents at all (extrinsic). And the most dangerous answer is the one that sounds completely reasonable (plausible) but does not actually match your source (unfaithful), because it survives a casual read and only fails when someone checks. That last kind, plausible but unfaithful, is what most of this article is about.

One line ties the whole article together: retrieval succeeding does not mean generation will use what was retrieved.

Part 1: What the model sees (Assembly)

Before the model writes anything, you assemble a context block out of the retrieved chunks. Three failures live here, and all three are pure engineering that you control.

Top-k is a precision dial, not a recall dial

You do not pass every reranked chunk to the model. You pass the top few, typically three to five. The instinct is that more chunks is safer, since you are less likely to drop the relevant one. This is wrong, and the way it is wrong matters. Every chunk past the relevant ones is a distractor, and distractors measurably pull the answer off. The model cannot reliably tell your strong chunks from the plausible-looking noise you padded around them, so accuracy drops. Too few chunks and you starve a question that needed the fourth one. Too many and you bury the answer in irrelevance. Top-k is the dial between those failures, and it controls precision, not recall.

Lost in the middle

Even with the right chunks selected, position matters. Models attend most strongly to the start and end of a long context and under-read the middle, and the effect gets worse as the context grows longer. This connects straight back to top-k: more chunks means a longer context means a deeper, deader middle.

The fix is to order, not just select. Put the highest-scored chunks at the top and bottom of the context, where attention is strongest, and place lower-scored chunks in the middle, where a chunk the model underweights does the least damage. Ranking the chunks is half the job; placing them is the other half.

Because the effect is positional and continuous, it does not respect chunk boundaries: the weak zone is a region in the middle of the token sequence, so it can wash out the tail of an early chunk or the opening of a late one, not just a tidy “middle chunk.” Placement is about where a chunk’s tokens land on the curve, not which slot it occupies.

The context window, and silent truncation

Everything competes for the same context window: the system prompt, instructions, few-shot examples, the user question, and the chunks. People remember the chunks and forget the rest, then overflow the budget. The failure here is not an error message. When you exceed the window, something gets silently dropped, usually the tail of your context, and you get a confident answer built on truncated evidence with no warning that evidence went missing. Budget the whole prompt, not just the chunks, and treat overflow as a bug rather than hygiene.

One mitigation bridges all three of these failures: context compression. Trim each retrieved chunk to the span that is actually relevant before sending it. Shorter chunks dilute less, shorten the middle, and ease window pressure at once.

Part 2: How it is told to use what it sees (Prompt construction)

The context block is assembled. Now you wrap it in a prompt that tells the model how to use it. This is where the most-omitted instruction in production RAG lives.

Grounding and abstention

Two instructions, and the second is the one people drop. First, instruct the model to answer only from the provided context. Second, and more important, explicitly permit it to say “this is not in the documents.” Without that permission, a model that cannot find the answer in the chunks will reach for something anyway, and you get a confident fabrication instead of a useful refusal. The abstention instruction is the difference between a wrong answer that ships and an “I don’t know” that tells you retrieval came up short. It costs one sentence in the prompt and prevents a whole class of plausible-but-unfaithful answers.

Prompt injection from retrieved content

Retrieved chunks are untrusted input. A document in your corpus can contain text like “ignore previous instructions and …”, and if your prompt treats the chunk as instructions rather than data, that text becomes an attack on your own system. The defence is structural, not the model politely declining. Delimit the context clearly so the model knows where instructions end and retrieved data begins, and design the prompt so context cannot override the system instructions. This is a real production surface that most generation write-ups skip entirely.

Chunk demarcation

Closely related, and the foundation for citations: separate the chunks with clear delimiters, such as XML-style tags. Without them, chunks bleed together, the model loses track of where one source ends and the next begins, and any attempt at attribution breaks. Demarcation is cheap and load-bearing.

<context>
  <chunk id=”1” source=”refund-policy.pdf” date=”2024-11”>
  ...chunk 1 text...
  </chunk>
  <chunk id=”2” source=”returns-faq.md” date=”2025-03”>
  ...chunk 2 text...
  </chunk>
</context>

“Chunks bleeding together” means the model can’t tell where one chunk ends and the next begins, so it reads several separate chunks as if they were one continuous passage.

Output control

The smaller cluster: a few-shot example or two to fix the answer format, a tone instruction, and a reminder to stay on the business question rather than drifting into the model’s generic knowledge. Useful, but secondary to grounding and injection defence.

Metadata and citations

Pass the metadata of each chunk into the prompt: source, title, date. This buys two things. It lets the model produce citations, which gives the user a way to trust the answer. And it lets the model reason about authority and recency, for instance preferring the newer document when two chunks conflict. There is a quieter benefit too: a citation is your cheapest handle on faithfulness, because a claim with no attachable source is a candidate hallucination. Citations are not decoration; they are a verification surface.

Part 3: Whether the model actually complies (Generation)

The context is assembled and the prompt is written. Whether the model does what you intended is the last place things break, and the two failures here are the ones top-k cannot touch, because both happen after the chunks are already in the context.

The model answers from training, not from your chunks

This is the central failure, and the rest of the article is partly about preventing it. You built RAG specifically so the model would answer from your current, private, authoritative documents. But the model also carries its training knowledge, and sometimes it answers from that instead, even when the chunk in front of it says otherwise.

Example, a user asks for the maximum operating temperature of product ISC290ABGQ. The retrieved chunk, from that product’s datasheet, says 55°C. But during training the model saw countless spec sheets quoting ranges “up to 70°C,” so that figure is baked into its weights. The chunk says 55; the model answers 70. The right answer was in the context. The model preferred what it already knew. The failure is insidious because 70°C reads as a perfectly normal spec, so the user designs the deployment around it, the hardware throttles in the field, and nobody suspects the assistant used a generic default over the product’s own datasheet.

This is why grounding instructions, low temperature, and edge placement all exist. They are not three unrelated tips. They are three defences against this one failure: forcing the model to prefer your chunks over its parametric memory. Fighting it comes down to making the chunk impossible to ignore and the model accountable to it. Grounding instructions tell the model to answer only from the provided context and to treat it as authoritative over anything it thinks it knows. A citation requirement is the strongest lever here: force the model to attach every claim to a specific chunk, because it cannot cite a chunk it did not read, and a number with no source is a visible red flag rather than a silent fabrication. Edge placement keeps the contradicting chunk where attention is strongest so it is harder to skip, and low temperature stops the model from drifting toward the fluent, familiar training answer. None of these are guarantees; together they tilt the model back toward your datasheet and away from the 70°C it half-remembers.

“Edge placement” is the strategy from the lost-in-the-middle section: deliberately putting your most important chunks at the edges of the context (the very start and the very end).

Multi-chunk synthesis

The second failure is subtler. Sometimes the answer is distributed across two or more chunks, and the model uses some of them but not all. Note that this is not about documents; the chunks can come from the same document or different ones. What matters is that the answer lives in more than one piece of retrieved context and the model has to stitch them together.

An example. A user asks whether product ISC290ABGQ is compatible with module XR-12. Chunk one, the ISC290ABGQ datasheet, lists the backplane standard the host supports. Chunk three, the XR-12 datasheet, lists the standard the module requires. The answer needs both compared. The model answers from chunk one alone, concluding “yes” from the host’s capability without checking the module’s requirement in chunk three. Both chunks were retrieved, both in context, top-k did its job. The correct answer needed chunk one and chunk three read together. Nothing in top-k catches this, because top-k guarantees the chunks are present, not that the model integrates them.

It is worth naming why this failure is so common. Chunking can split a single coherent fact across a boundary, so the thing you would have read in one sentence in the source now lives in chunk four and chunk five. Careless chunking does not just hurt retrieval. It splits a single fact across two chunks, and that scattered fact becomes a synthesis problem the model has to reassemble at generation time. The fix lives partly back in your chunking strategy and partly in prompting the model to combine across all provided chunks rather than answer from the best single one.

Mitigating this runs across both retrieval and generation. On the retrieval side, chunk so that a self-contained fact is not split across a boundary, and prefer retrieving complete units (a full spec table, a full compatibility section) over fragments, so a single answer is less likely to be scattered in the first place. On the prompt side, instruct the model explicitly to use all provided chunks and to combine information across them, rather than answer from the single most relevant one; for comparison questions, tell it to check each requirement against each source before concluding. Demarcating chunks with ids helps here too, because it lets the model track that it has consulted both the ISC290ABGQ datasheet and the XR-12 datasheet rather than stopping at the first. And the honest backstop: some genuinely multi-hop questions are better decomposed before generation, retrieved per sub-question, then synthesised, which is where this leg starts to shade into agentic RAG.

The through-line

Read the three parts together and the shape is clear. Assembly decides what the model sees. Prompt construction decides how it is told to use it. Generation decides whether it complies. Each stage has a signature failure that looks like “the model is not smart enough” and is actually a fixable handoff problem. The two failures that survive a flawless retrieval, answering from training and using only one chunk, are the ones to watch most, because they happen with the right answer already on screen.

The levers in this article, citations, abstention, all-chunks prompting, edge placement, are only trustworthy once you can tell whether they are working. Which raises the obvious next question. If these failures are invisible from the outside, the answer just quietly wrong, how would you even know they are happening? You cannot improve what you cannot measure, which is the subject of the next piece: faithfulness, groundedness, context utilisation, and abstention calibration.