Vol. 1 - The Problem with RAG is That We Reach for It Too Fast

Series: Building Production RAG

Jun 05, 2026

January 2026. My team was handed a problem that sounded simple: build a chatbot so engineers can ask questions across 50,000 internal documents ( technical manuals, product guides, release notes, support tickets, internal wikis accumulated across years of product launches and version cycles).

Engineers were spending hours hunting for answers that existed somewhere, in some document, if only they knew where to look.

The instinct in every room when you hear this problem is immediate: RAG. Obviously RAG.

We went with RAG. And it was the right call. But we got there for the wrong reasons: instinct and familiarity, not deliberate elimination of alternatives. That gap between correct conclusion and correct reasoning matters, because the next team with a similar-sounding problem will reach for RAG on instinct too, and their problem may not be the same as ours.

Most teams skip straight to LangChain + Pinecone without answering any of these. That’s how you end up with an $8,000/month system that a keyword search could have replaced.

Here are the three questions most teams skip:

Do we actually need retrieval?
Do we actually need generation?
Which type of RAG fits this specific problem — if RAG is even the right frame?

If you cannot answer all three before your first architecture meeting, you are building before you understand the problem.

TRADITIONAL APPROACHES

Before RAG existed, teams solving this class of problem had a short menu of options. Understanding why each approach failed is more useful than dismissing them as legacy.

Manual curation + business knowledge. Maintain a dictionary mapping query types to document collections. Works when the corpus is small and stable. Breaks at scale , 50,000 documents with version histories cannot be manually curated without a dedicated team and constant drift. In our case, documents spanned more than 100 product lines each with 3–5 active versions. Any manual mapping was outdated within weeks of a release.

Exact keyword search. BM25, inverted index, ElasticSearch/OpenSearch. Fast, interpretable, zero hallucination risk. Fails when the user’s vocabulary doesn’t match the document’s vocabulary. “How do I reset the auth token” returns nothing if the manual says “authentication token refresh procedure.” In our corpus, engineers querying in conversational language consistently missed documents that existed, recall collapsed on natural language variation.

Fuzzy matching. Extends keyword search with edit distance and phonetic similarity. Recovers some recall on typos and close variants. Breaks on conceptual variation, “restart” and “reboot” are semantically identical but fuzzy match treats them as distant. Also introduces significant latency at large corpus sizes; at 50,000 documents, response times became impractical for interactive use.

Semantic similarity without generation. Embed queries and documents, retrieve top-K by cosine similarity, surface chunks directly. Solves the vocabulary mismatch problem , “auth token reset” and “authentication token refresh” land close in embedding space. Still fails when the answer requires connecting information across multiple documents. A query like “why is my setup failing after following the installation guide” needs context from the guide, the known issues page, and a related support ticket simultaneously. Retrieval surfaces all three. The engineer still has to read and assemble them manually.

Each approach has a failure mode. The failure modes are not bugs, they are signals about what the problem actually requires.

MENTAL MODEL

The senior practitioner’s reframe: RAG is not a technology. It is a composition of three distinct capabilities, each of which you may or may not need.

Most teams treat RAG as atomic. They hear the use case, they reach for the pattern. But RAG is R + A + G:

Retrieval - finding relevant content from a corpus too large for a context window, where keyword match fails. In our case: 50,000 documents across multiple product lines. No context window fits this. No keyword search bridges the vocabulary gap between how engineers ask questions and how documentation is written. Retrieval is the only path to getting the right 3–5 chunks in front of the model, out of 50,000 possible sources.

Augmentation - constructing the prompt. Take retrieved chunks, insert them into a prompt template alongside the user query. No LLM involved at this step, this is string assembly. The quality of augmentation determines what the LLM sees, which directly determines output quality. A poorly constructed prompt with the right chunks still produces a bad answer.

Generation - passing the augmented prompt to an LLM to produce a response. This is where synthesis, hallucination risk, and latency cost live. In our case, generation was necessary because the answer to most engineer queries was never a single chunk , it was a synthesised response drawn from an installation guide, a known issues page, and a support ticket that happened to describe the same failure three months earlier. No human would read all three and manually assemble the answer every time. Generation does that assembly.

Each component carries cost: infrastructure complexity, retrieval failure modes, hallucination risk, latency. Not every problem needs all three.

The reframe that changes how you approach this: treat each component as guilty until proven innocent. Don’t assume you need generation just because you have an LLM available. Don’t assume you need retrieval just because your corpus is large. Question each one independently, and only build what the problem actually demands

This forces you to size the solution to the problem rather than defaulting to the most complex architecture.

Most teams arrive at architecture by asking “how do I implement RAG?” The better question is “what does my problem actually require?” then check whether each component earns its place. The framework below is that check.

FRAMEWORK

Framework 1 — Do you even need Retrieval?

Ask these three questions first. If any answer is yes, you do not need a retrieval pipeline for this problem.

┌─────────────────────────────────────────────────────────────┐
│  STOP. Do you even need retrieval?                          │
│                                                             │
│  1. Does your entire corpus fit in a context window?        │
│     (<200K tokens)                                          │
│     → Pass it all in. No chunking, no vector DB.            │
│       Eg: Single-product FAQ, 50 pages.                     │
│       Re-evaluate if cost per query or privacy rules bite.  │
│                                                             │
│  2. Is the query general knowledge the LLM already knows?  │
│     → Skip retrieval. Go straight to generation.            │
│       Eg: “What is a JWT token?”                            │
│       “What does HTTP 429 mean?”                            │
│       Signal: no proprietary nouns, no product names,       │
│       no version numbers, no internal system references.    │
│                                                             │
│  3. Did the user bring the content with them?               │
│     → LLM reasons over what it was handed. No retrieval.    │
│       Eg: Pasted stack trace: “What is wrong with this?”    │
│       Pasted config: “Is this set up correctly?”            │
│       Watch for: context window overflow on large pastes.   │
│                                                             │
│  None of the above? → Retrieval is justified.               │
│  Proceed to Framework 1.                                    │
└─────────────────────────────────────────────────────────────┘

In all three cases: skip retrieval, go straight to generation.

In our case: 50,000 documents across multiple product lines. No context window fits this. Queries are proprietary and product-specific. Content never arrives with the query. All three exits closed : retrieval is the only path.

Framework 2 — What Kind of Answer Does This User Query Need?

Retrieval is justified. Now decide what pipeline to build based on what the answer looks like , not based on what tools you have available.

What does the answer look like?
│
├── A single fact, spec, price, or definition?
│   The answer IS the chunk. It exists verbatim in one place.
│   └── RETRIEVAL-ONLY. Do not pass to LLM.
│       Surface the chunk directly.
│       Adding generation introduces hallucination risk
│       on queries that already have a correct exact answer.
│       Eg: “What is the rate limit for API v3.2?”
│       Eg: “What is the price of product ISC290ABGQ?”
│
├── A synthesised response drawn from multiple chunks?
│   No single document has the full answer.
│   Answer must be constructed, not located.
│   └── FULL RAG — Retrieval + Augmentation + Generation.
│       Eg: “Why is authentication failing when I follow
│       the setup guide?”
│       Needs: setup guide + known issues page + support ticket.
│       Generation assembles them into a coherent response.
│
└── Exact text from a document — not paraphrased?
    Retrieve the chunks AND pass them to the LLM — but instruct the LLM to 
    quote verbatim, not synthesise
    └── RETRIEVAL + CONSTRAINED EXTRACTION.
        Retrieve chunks. Instruct LLM: quote, do not synthesise.
        Eg: “What does section 4.2 of the data retention
        policy say?”
        Compliance wants the policy text, not an interpretation.
        Prompt: “Return only the exact text from the retrieved
        passage. Do not rephrase, summarise, or infer.”

Framework 3 — If Full RAG, Which Type?

You have decided Full RAG is the right pipeline. Now the question is which RAG architecture fits the nature of your data and queries.

Where does the answer live?
│
├── In structured data — tables, metrics, records?
│   └── SQL-RAG.
│       Query → LLM generates SQL → DB returns rows → LLM formats answer.
│       Retrieval reference: semantic model (table + column descriptions),
│       not document chunks.
│       Eg: “What was average P1 ticket resolution time in Q4?”
│       Tool pattern: Snowflake Cortex Analyst.
│
├── In relationships between entities across systems?
│   Queries require linking multiple entities, not synthesising passages.
│   └── GRAPH-RAG.
│       Entities as nodes, relationships as edges.
│       Retrieval traverses the graph, not a vector index.
│       Eg: “Which downstream services break if the auth module
│       in Product X is deprecated?”
│       Skip if: queries are passage-level synthesis.
│       Use if: queries require multi-entity reasoning.
│
├── In unstructured text across multiple documents?
│   └── TEXT-RAG.
│       Embed → chunk → retrieve → augment → generate.
│       The core pipeline. Articles 2–7 cover this in depth.
│       Eg: “Why does my installation fail at step 3 on Ubuntu 22?”
│       Answer lives across a setup guide, a known issues page,
│       and a support ticket from six months ago.
│
├── In images, diagrams, or slides?
│   └── MULTIMODAL RAG.
│       Separate ingestion pipeline for non-text content.
│       Eg: Architecture diagrams in product manuals,
│       annotated screenshots in support tickets.
│
└── Stable domain knowledge + dynamic document corpus?
    └── HYBRID — Fine-Tune + RAG.
        Fine-tune for domain vocabulary and answer patterns.
        RAG layer for current document retrieval.
        Fine-tune handles: how to answer.
        RAG handles: what the current answer is.
        Eg: Medical coding — stable ICD patterns (fine-tune)
        + current clinical guidelines updated quarterly (RAG).

Fine-Tuning vs RAG vs Hybrid

Fine-tuning is the other fork teams conflate with RAG. Fine-tuning teaches the model domain patterns : vocabulary, answer style, document structure. RAG grounds the model in current documents. They solve different problems and combine well when you have both labeled training data and a dynamic corpus.

Hybrid operationally means: fine-tune the base model on domain vocabulary and answer patterns, then add a RAG layer for current document retrieval. The fine-tuned model handles how this type of question should be answered. RAG handles what the current answer is.

For our use case: 50,000 documents, company-specific domain, infrequent updates, no labeled QA pairs at project start. Text-RAG was the correct call not because RAG beats fine-tuning, but because our constraints eliminated fine-tuning as a near-term option. We will see more about RAG in upcoming articles.

THE ONE-LINER

RAG solves the retrieval-generation gap but that gap only exists for a specific class of problems. Know which class you have before you build.

Ergodic

Discussion about this post

Ready for more?