Vol. 9 - Agentic RAG: The Pipeline Stops Being a Pipeline
Series: Building Production RAG
For eight articles, we’ve been talking about RAG as a pipeline. Query comes in, gets embedded, hits a retriever, chunks come back, get reranked, get stuffed into a context window, and a generator produces an answer. Every improvement we’ve discussed — better chunking, hybrid search, reranking, multi-hop decomposition, and in the last piece, locking the whole thing down against prompt injection and data leakage — has been an improvement within that shape. Linear. Deterministic. One retrieval, one generation, done.
Agentic RAG breaks that shape on purpose.
Why the pipeline metaphor stops working
A pipeline assumes you know the steps in advance. But a lot of real queries don’t fit a fixed number of steps. Consider the corpus we’ve been building against throughout this series — product documentation, the company wiki, and support ticket history — and a query like: “A customer says the export feature is broken on the Enterprise plan — is this a known issue, and what’s the workaround?”
A fixed pipeline retrieves once, based on the embedding of that whole sentence, and hopes the top-k chunks contain both a known-issue entry and a workaround. Most of the time, they don’t — not because retrieval is broken, but because the question secretly has two sub-questions across two different sources: “is this known” lives in support tickets or the wiki’s known-issues page, and “what’s the workaround” lives in product documentation, and the two aren’t in the same chunk, sometimes not even the same source. We touched on this in the multi-hop retrieval discussion earlier in the series: some queries need to be decomposed before retrieval even starts — and here, decomposition isn’t just splitting the question, it’s routing each half to the right source.
Agentic RAG takes that observation and generalizes it. Instead of hardcoding “decompose, then retrieve twice, then merge,” you hand the model a set of tools — a retriever, maybe a calculator, maybe a SQL executor — and let it decide, at inference time, how many times to call what, and in what order. The pipeline doesn’t disappear; it just stops being written down in your code and starts being written down, dynamically, by the model, per query.
This is the same shift that happened to agents generally: from “chain of fixed steps” to “loop with tool calls and a stopping condition.” RAG is just one of the tools now, not the whole architecture.
The three things that actually change
1. Retrieval becomes a tool, not a stage.
In a classic pipeline, retrieval happens once, at a fixed point, before generation. In agentic RAG, retrieval is a function the model can call — zero times, once, or five times — whenever it decides it doesn’t have enough grounding to answer. This sounds like a small change. It isn’t. It means your system prompt now needs to teach the model when to retrieve, not just what to do with retrieved content. A model that retrieves too eagerly burns latency and money on unnecessary lookups. A model that retrieves too conservatively hallucinates confidently instead of admitting it needs another lookup. Getting this threshold right is mostly prompt and tool-description work, and it’s genuinely harder to get consistent than tuning a reranker.
2. Decomposition happens live, not in a preprocessing step.
Some of you built query decomposition as a separate LLM call before retrieval — classify the query, split it if needed, retrieve for each sub-query, merge. That’s a reasonable design, and it still works fine for known query shapes. What agentic RAG adds is decomposition for query shapes you didn’t anticipate. The model isn’t running a decomposition template; it’s reasoning about the question, noticing it can’t answer directly from one retrieval, and deciding to break it up on the fly — sometimes into two parts, sometimes into four, sometimes not at all. You trade a predictable, testable decomposition step for a more general one that handles novel questions but is harder to unit test.
3. Self-correction becomes possible — and expensive.
This is the one people get most excited about, for good reason. In a fixed pipeline, if the first retrieval misses the mark, the generator just does its best with bad context — you get a plausible-sounding wrong answer. In an agentic setup, the model can notice its own retrieval was insufficient (”these chunks don’t mention the pricing change date”) and issue another retrieval with a reformulated query, or try a different tool entirely. This is real self-correction, not a retry loop bolted on from outside. It measurably improves answer quality on ambiguous or compound queries, especially ones you haven’t seen in eval data.
It’s also where the honest costs live.
What the loop actually looks like
Go back to the export-feature question and walk it through an agentic setup with three retrieval sources — product docs, the wiki, and support tickets — exposed as tools the model can choose between.
Step 1 — the model decides it can’t answer from the question alone. No retrieval has happened yet. The model reads the query, recognizes two implicit sub-questions (”is this known” and “what’s the workaround”), and decides to decompose rather than fire a single broad retrieval.
Step 2 — first tool call: search support tickets. The model queries the support ticket index for “export feature Enterprise plan broken.” It gets back three tickets, all confirming customers have hit this, but none of them contain an actual fix — just “escalated to engineering.”
Step 3 — self-correction. This is the part a fixed pipeline can’t do. The model notices the tickets confirm the issue is known but don’t answer the workaround half of the question at all. Instead of generating a half-answer, it recognizes the gap and issues a second, different tool call.
Step 4 — second tool call: search product documentation. Now it queries product docs for “export feature workaround” or “manual export alternative.” This time it finds a doc page describing a CSV export fallback for the affected plan tier.
Step 5 — a third, smaller check. Suppose the doc page is undated and the model isn’t sure if it’s still current. It queries the wiki for “export feature known issues” to see if there’s a status page confirming the workaround is still valid, or if it’s since been deprecated.
Step 6 — stopping condition. With confirmation from a ticket (yes, known), documentation (here’s the workaround), and the wiki (still current), the model decides it has enough grounding and generates the final answer, citing where each part came from.
Notice what happened: one user question turned into three tool calls across three different sources, one self-correction, and a decomposition that wasn’t decided in advance — it was decided because the first retrieval came back incomplete. A fixed pipeline would have retrieved once, probably against whichever source scored highest for the whole query as a single embedding, and either returned “yes, this is a known issue” with no workaround, or returned a workaround with no confirmation it’s a real, acknowledged issue. Either answer is incomplete in a way the user wouldn’t necessarily notice — which is exactly the failure mode agentic RAG is meant to close.
It’s also, not coincidentally, the same query shape that makes the cost and safety sections below non-optional: three tool calls instead of one, across three different data sources with three different trust levels — and support tickets, being user-submitted, are exactly the kind of source where you should assume some content could be adversarial or manipulated, not just noisy.
The cost nobody puts in the demo
Latency stops being a number and becomes a distribution.
A classic RAG pipeline has a latency you can quote: embed the query, retrieve, rerank, generate. You can benchmark it, you can put an SLA on it. Agentic RAG doesn’t give you that. A simple factual query might resolve in one retrieval and look identical to your old pipeline. A compound query might trigger three retrieval calls, a self-correction pass, and a second generation attempt — and now you’re 4-6x slower, with no way to know in advance which queries will hit which path. You don’t get to report “p50 latency is 800ms” with a straight face anymore; you have to report a distribution shaped by query complexity, and that distribution is genuinely unpredictable until you’ve seen enough production traffic to characterize it.
This matters operationally more than it sounds like it should. Timeout configuration, autoscaling, and cost forecasting all assume some kind of stable latency envelope. Agentic RAG asks you to plan for a long tail you can’t fully bound in advance, because the tail length is a function of query difficulty, which you don’t control.
Cost compounds the same way.
Every retrieval-as-tool call and every self-correction pass is another round trip to the model, and often another retrieval against your vector store or search index. A query that used to cost one generation call can now cost three or four LLM calls plus multiple retrievals. Multiply that by production volume and the fixed-pipeline RAG system you budgeted for is not the system you’re actually running. This is the same warning we gave in the earlier article on RAG costs, sharpened: agentic behavior doesn’t just add cost, it adds variable cost that scales with question difficulty rather than query volume, which makes it much harder to forecast.
Unpredictability isn’t just a UX problem — it’s a safety problem.
This is where Article 8’s security discussion connects directly. A fixed pipeline has a small, auditable attack surface: one retrieval call, against one index, with content going into one generation prompt. An agentic system that can decide to call retrieval multiple times, potentially against different sources, with the model choosing the query text each time, has a much larger surface. Look back at the walkthrough above: step 3’s second tool call was a query the model wrote itself, based on what it read in step 2’s support tickets. Support tickets are user-submitted — the least trusted of the three sources in that example. If a ticket contained crafted text designed to look like a legitimate escalation note but actually steered the model’s next query toward a malicious or misleading doc page, the model would have no way to distinguish that from a genuine ticket, because it’s reasoning over the content, not the provenance, when it decides what to search next. This is a classic pattern in multi-hop or self-correcting loops: you’ve built a channel where injected content in one retrieval step can influence what gets retrieved or queried in the next one. The isolation and sanitization practices from Article 8 aren’t optional add-ons to agentic RAG — they need to be applied at every tool call in the loop, not just at the entry point, and they need to be strictest at exactly the sources with the weakest provenance, like user-submitted tickets, since every loop iteration is a new opportunity for untrusted content to steer the model’s next action.
This is also why “just let the model decide” is a much bigger claim in agentic RAG than it sounds. You’re not just trusting the model to write a good answer; you’re trusting it to make good decisions about its own tool use, repeatedly, based on content that might be adversarial. That trust needs guardrails: retrieval budgets (cap the number of tool calls per query), scoped tool permissions (not every retrieval tool should be callable in every context), and logging every intermediate call so you can actually debug why a query took the path it did.
Where this leaves you
Agentic RAG is the right upgrade when your query distribution genuinely includes compound, ambiguous, or comparative questions that a fixed pipeline handles poorly — and wrong when it doesn’t. If most of your traffic is single-hop factual lookup, adding agentic decision-making is pure cost and latency variance with no quality upside. The honest way to decide is to look at your actual failure cases from the fixed pipeline: if they’re failing because retrieval genuinely needed to happen more than once, agentic RAG addresses a real gap. If they’re failing for other reasons — bad chunking, missing documents, weak reranking — agentic behavior won’t fix that, it’ll just make the same failure slower and more expensive to reproduce.
The pipeline stops being a pipeline. That’s the point. But a pipeline was also the thing that made your system predictable, testable, and boundable — and you don’t get those properties back for free.