Vol. 2 - Why Your RAG Retrieval is Failing — Chunking & Embeddings

Series: Building Production RAG

Article 1 gave you the framework to decide whether RAG is right for your problem. This article assumes you've made that call - Text-RAG fits, retrieval is justified. Now the real engineering begins, and the first thing that will break is retrieval.

We thought retrieval was the easy part.

You have documents, you chunk them, you embed them, you search. The whiteboard makes it look like plumbing - connect the pipes and water flows. We gave ourselves a week for retrieval and three sprints for the generation layer where the real intelligence lives.

Three weeks later we were still on retrieval.

The same query - “why does authentication fail after following the setup guide for Product X v4.2” - was returning completely different chunks depending on how we had split the documents. One chunking strategy surfaced the v4.2 setup guide but missed the known issues page entirely. Another returned six chunks from the same paragraph across three document versions - v4.0, v4.1, and v4.2 - each slightly overlapping, none complete. A third returned the right procedure but from v3.8, three releases behind.

The LLM wasn’t the problem. The LLM was doing exactly what it was told. It was generating answers from whatever chunks retrieval handed it - and retrieval was handing it the wrong ones.

This is the article we wish we had before we started building.

Retrieval failure is invisible from the outside. Users see a wrong answer and blame the model. The model is innocent. The problem is upstream - in how you chunked, how you embedded, how you searched, and how you ranked. Fix any one of these incorrectly and the entire pipeline degrades silently.

We will go through each failure point in order: chunking, embedding, vector search, and reranking. For each one - what the decision is, what breaks when you get it wrong, and how to know you’ve got it right.

Part 1 : Chunking Strategies

1. Fixed Chunking Split the document into chunks of a fixed token size - 100, 200, 500 tokens, whatever you decide. Simple to implement, fast to run, no intelligence involved. The problem is that it is completely blind to document structure and semantic meaning. A chunk boundary can land mid-sentence, mid-procedure, or mid-table. The chunk that gets retrieved may be syntactically complete but semantically incomplete - missing the setup step that came just before the boundary, or the error code that appears just after. The LLM receives a fragment and generates from it anyway.

2. Recursive Chunking Instead of a fixed size, you define a hierarchy of separators -paragraph breaks, then newlines, then sentences, then punctuation. The algorithm tries the highest-level separator first. If the resulting chunk is still too large, it moves to the next separator down. This produces more natural boundaries than fixed chunking but is still structure-blind - it doesn’t understand what is in the chunk, only where the separators are. Works well as a default when you have mixed document types and no time to build something more sophisticated.

3. Sentence-Based Chunking Split on sentence boundaries. You decide how many sentences form one chunk - one, three, five. Cleaner than fixed chunking because at least each chunk is grammatically complete. The failure mode is that sentence boundaries don’t respect semantic continuity - a new sentence often continues the thought of the previous one. “This error occurs when the token expires. Refresh it using the endpoint below.” Split across a chunk boundary, neither chunk is complete enough to be useful on its own.

4. Semantic Chunking Instead of splitting on structure, split on meaning. The algorithm embeds each sentence, compares consecutive sentences by cosine similarity, and starts a new chunk when similarity drops below a threshold - meaning the topic has shifted. This is the most adaptive strategy - chunks reflect actual semantic units rather than arbitrary boundaries. More compute-intensive at ingestion time, but produces significantly cleaner retrieval. Best default for unstructured prose documents where topic shifts don’t follow a predictable structure.

5. Document-Structure Based Chunking Respect the document’s native structure. A PDF has pages, sections, headers. An HTML document has tags. A codebase has functions and classes. A legal document has clauses and sub-clauses. This strategy parses the document type first, extracts its structural units, and chunks along those boundaries. The resulting chunks are semantically coherent because they follow the author’s own organisation. Works best when documents have consistent, well-defined structure. Breaks when structure is inconsistent - a PDF that was scanned and OCR’d has no reliable structure to parse.

6. Agentic (Proposition) Based Chunking Instead of splitting the original text, rewrite it first. Each sentence or passage is rewritten by an LLM into a self-contained proposition - a statement that carries its full meaning independently of surrounding context. Then chunk the rewritten propositions. The result is chunks that are always semantically complete regardless of where they appear in the document. The cost is significant - an LLM call per passage at ingestion time makes this expensive and slow at scale. Justified when retrieval quality is critical and ingestion cost is acceptable.

---

When the document is a table

Every strategy above splits on text: characters, sentences, paragraphs, or shifts in meaning. Tables have none of those boundaries in any useful sense, and a corpus of technical documentation is full of them. Configuration references, parameter lists, version comparison matrices, supported-value tables.

Here is what goes wrong. If you take a configuration table and split it with a fixed or recursive chunker, which cuts when it hits a size limit. The header row lands in one chunk and half the data rows land in the next. That second chunk now reads:

| max_retries | 3 | count |
| pool_size | 10 | connections |

A row without its header is meaningless. Three what? Ten of what? The chunk carries the value but not the thing the value describes, and no amount of retrieval quality downstream can recover information that was destroyed at ingestion.

It gets worse before it gets better. That fragment is also close to unretrievable. Embeddings capture semantic meaning, and a pipe-delimited fragment of numbers has very little of it.

The fixes, in escalating order

Stop at the first one that works. Most tables need only the first.

Keep the table whole. The default, and usually the whole answer. Make a table an atomic unit the chunker cannot split, even if the chunk runs over your size target. Structure-aware parsers find table boundaries in Markdown, HTML, and clean PDFs; exempt those regions from the splitter. A modest table with its heading intact is now meaningful and findable. Done.

Repeat the header. For tables genuinely too large to keep whole, split by rows, not by size, and repeat the header at the top of every chunk. Cheap, and it kills the orphaned-row problem outright.

Serialise rows into sentences. Embed “In v4.2, request_timeout defaults to 30 seconds” instead of the raw row, so the chunk matches the register of the question. Targeted, not default. Reach for it when the table is too large to keep whole, when near-identical tables collide (a v3.9 and a v4.2 config table are ninety percent the same text, so their embeddings sit on top of each other and retrieval picks the wrong version; putting the version in every sentence pulls them apart), or when users want single values and you would rather not spend context on fourteen irrelevant rows. Outside those cases, skip it: serialising a fourteen-row table buys you fourteen chunks to manage and a table that must be reassembled when someone asks to see all of it.

Index a summary. For large or complex tables, embed a short description of what the table holds (”default configuration values for v4.2: timeouts, retry limits, pool sizes”) and retrieve on that, returning the full table on a hit.

The last two combine into the production pattern for a documentation corpus: index one representation, return another. Search the serialised rows or the summary; hand the model the whole table. Precision on the way in, completeness on the way out. That is parent-child, applied to tables.

The context around the table

One thing all four techniques depend on: a table almost never means anything on its own. The sentence above it (”The following defaults apply to v4.2 and later”) is frequently the only thing that disambiguates it from the near-identical table three pages earlier for v3.9. Preserve the heading and the surrounding line when you chunk, or you will build a corpus of tables that are individually well-formed and collectively impossible to tell apart.

---

Chunking Modifiers

Whichever strategy you choose, two modifiers improve retrieval quality:

Overlap When creating chunks, include the last N tokens of the previous chunk at the start of the next one. This preserves context across boundaries — a procedure that spans two chunks is partially present in both, reducing the chance that a boundary cut loses critical information. Most effective with fixed, recursive, and sentence-based chunking where boundary placement is arbitrary. Less necessary with semantic chunking where boundaries already fall at meaning shifts. Typical overlap: 10–20% of chunk size. Too much overlap and you’re storing redundant content; too little and boundary cuts still lose context.

Parent-Child Chunking Create two levels of chunks from the same document. Large parent chunks — whole sections or paragraphs — capture broad context. Small child chunks — individual sentences or propositions — enable precise retrieval. At query time, retrieve by child chunks for precision, then return the parent chunk to the LLM for context. The child chunk gets you to the right location in the document. The parent chunk gives the LLM enough surrounding context to generate a complete answer. Particularly effective for technical documentation where a precise fact (child) only makes sense in the context of the procedure it belongs to (parent).

The decision is sequential, not simultaneous:

1. Pick a chunking strategy (structure-based, fixed-token, whatever). The pieces it spits out = your chunks. By default these are also what you search.

2. Then decide: do I want to return something bigger than what I searched when one matches? If no → you’re done, plain chunking (search the chunk, return the chunk). If yes → you’ve got parent-child: the searched piece is now the child, and you define a parent that wraps it and gets returned instead

Part 2 : Embeddings

Chunking decides how documents are split. Embedding decides how those chunks — and the user query — are represented as vectors for search. The fundamental requirement: query and chunks must live in the same vector space, encoded by the same model. If they aren’t, similarity scores are meaningless.

Two retrieval paradigms, each with a different representation:

Sparse Retrieval

BM25 is a statistical scoring function — no neural encoder involved. It scores each chunk against the query based on term frequency (how often query terms appear in the chunk) weighted by inverse document frequency (how rare those terms are across the full corpus). Fast, interpretable, zero training required. Strong for exact term match — product names, version numbers, error codes, API endpoints.

The failure mode is vocabulary mismatch — the same problem exact keyword search always had. “Authentication token refresh” and “reset auth token” score near zero against each other in BM25 because the terms don’t overlap.

One important note: sparse retrieval has no embedding model decision. BM25 matches tokens statistically. If your corpus and queries are in different languages, sparse retrieval is not viable — there is no token overlap to exploit. Dense retrieval with a multilingual model is the only path.

The neural evolution of sparse retrieval is SPLADE — a transformer model that produces sparse vector representations. Better than BM25 at handling vocabulary mismatch while keeping the interpretability and speed of sparse representations. Worth considering when BM25 recall is insufficient but full dense retrieval cost is too high.

Dense Retrieval

Encoding models like BERT or OpenAI embeddings encode the user query and each chunk into dense vectors — independently, in separate passes. This architecture is called a bi-encoder: two separate encoding passes, one for the query, one for the chunk. Similarity between query and chunk is then measured by cosine similarity between their vectors.

Because encoding happens separately, chunk embeddings can be pre-computed and stored at ingestion time. At query time only the query needs encoding — making this fast enough for production use at scale.

Dense retrieval handles vocabulary mismatch well — “authentication token refresh” and “reset auth token” land close together in dense vector space because the model understands semantic equivalence.

The limitation: because query and chunk never interact during encoding, the model captures general semantic similarity but can miss fine-grained relevance. Two chunks can be semantically close to the query without actually answering it. This is where reranking earns its place , but that comes after retrieval.

Embedding model decision within dense retrieval:

This is where language comes in — and it is a dense-only decision. Sparse retrieval has no model to configure here.

Embedding model language decision
│
├── Single language corpus + queries?
│   → Monolingual model
│     Eg: BERT, OpenAI text-embedding-3
│
├── Multi-language corpus AND queries —
│   same language matches same language?
│   → Multilingual model
│     Eg: mBERT
│     French query → French docs ✓
│     French query → English docs ✗ (weak alignment)
│
└── Queries in any language must find
    documents in any other language?
    → Cross-lingual model ← OUR CASE
      Eg: LaBSE, multilingual-e5-large
      German query → English docs ✓
      Any language → any language ✓

Multilingual model — trained on many languages. Good at encoding within a language. A French query finds French documents well. An English query finds English documents well. But a French query may not find English documents , because the model hasn’t explicitly aligned cross-language representations. Languages share the same vector space in theory but alignment across languages is weak.

Cross-lingual model — specifically trained to align languages into a shared vector space. A German query and a French document land close together even though they’re different languages. LaBSE, multilingual-e5-large are cross-lingual models. This is what you actually need for global customer → English documentation retrieval.

Global customers querying in their own language against English documentation. The query language is unpredictable — could be German, Japanese, Arabic, Mandarin. A multilingual model handles each language well in isolation but does not reliably align cross-language queries to English documents. A cross-lingual model like LaBSE or multilingual-e5-large is the correct choice — it is explicitly trained for this alignment.

Conclusion

Chunking and embedding are the foundation. Get them wrong and nothing downstream saves you. Get them right and you have earned the right to the next set of problems — how you store those embeddings, how you search across them, and how you rank what comes back.

Because retrieval does not end when the vector is stored. It ends when the right chunk lands in the right position in the LLM’s context window. Everything between storage and that moment is Article 3.