RAG quality: chunking, reranking, and retrieval evaluation

When a RAG (Retrieval-Augmented Generation) system gives a wrong answer, the blame usually lands on the prompt or the model; but the cause is often simpler: the retrieval layer never handed the model the right chunk. Chunking strategy, hybrid search setup, and reranking — together these three decisions determine RAG chunking reranking quality, each closing off a different failure mode. This post builds, step by step, a quality-engineering loop that runs in production: chunking approaches, two-stage reranking, and measuring retrieval with recall@k and MRR.

💡 Pro Tip: When starting a new RAG project, don't reach for a reranker before you've fixed chunking — no matter how well you rank bad chunks, you can't get a correct answer out of a chunk that never contained the answer.

Table of Contents

• Bad answers usually come from retrieval
• Chunking strategies: fixed-size, sentence, header-aware
• Overlap and metadata: what to store
• When hybrid search (BM25 + vector) wins
• Reranking: the cost of two-stage retrieval
• Measuring retrieval: recall@k, MRR, answer accuracy
• The production improvement loop
• FAQ
• What's the best chunk size for RAG?
• Is a reranker actually necessary, and how much does it gain you?
• How is retrieval quality measured (recall@k, MRR)?
• When does hybrid search beat pure vector search?
• Do you recommend a fixed percentage for chunk overlap?
• Should I fix chunking first, or set up hybrid search first?
• Update (September 2026)
• Conclusion
• Sources

Bad answers usually come from retrieval

A typical RAG pipeline has three steps: embed the query, retrieve the nearest k chunks, optionally rerank them, then hand the context to the model. The most fragile link is usually the middle step. In a baseline from Anthropic's contextual retrieval work, even with a solid embedding model, top-20 retrieval missed the relevant chunk 5.7% of the time — roughly 6 out of every 100 queries answered without the model ever seeing the correct chunk.

That's why, when a RAG answer comes back wrong, the first thing to check isn't the prompt — it's what was actually retrieved. Poking at the prompt without checking retrieval logs usually treats the symptom, not the disease. I answer one question first: "was the answer actually among the retrieved top-k chunks?" If not, the problem is chunking/search; if it was there, the problem is how the model used the context.

Optimizing without separating these two usually goes to the wrong place: lengthening the prompt, hardening the system instruction, or switching models doesn't close the retrieval gap — it just changes how the failure looks. The rest of this post splits retrieval into three separately measurable components (chunking, search, reranking), so an improvement can be verified with numbers, not guessed at.

Chunking strategies: fixed-size, sentence, header-aware

Pinecone's chunking guide groups the approaches used in production under a few main headings:

• Fixed-size chunking: Splits by character or token count; this is the guide's default recommendation — simple, predictable, quick to set up.
• Sentence-based chunking: Splits at natural sentence boundaries; preserves semantic integrity better than fixed-size, but chunk sizes become irregular.
• Header/structure-aware chunking: Follows Markdown/HTML heading hierarchy or document structure; for headed content like technical documentation and articles, this keeps chunk boundaries aligned with the author's own semantic boundaries.
• Semantic chunking: Sets boundaries based on drops in embedding similarity between consecutive sentences; splits when the topic of conversation changes.
• Contextual chunking: An approach Anthropic introduced in 2024 that prepends a short context sentence (which document, which section) to each chunk so it can be interpreted correctly on its own.
• Chunk-expansion: When a chunk is selected at retrieval time, neighboring chunks are also added to the context, completing sentences cut off at the boundary.

For short/structured content like a codebase or FAQ, fixed-size is usually enough; for long, headed documentation, header-aware chunking keeps boundaries aligned with the author's semantic boundaries and produces more coherent chunks. I covered RAG fundamentals (embedding, vector search, similarity scores) in RAG fundamentals: embedding and vector search; this post is a direct continuation of that one.

Overlap and metadata: what to store

The point of chunk overlap is to prevent a sentence or idea from landing right on a chunk boundary and getting cut in half. There's no single ideal ratio: more overlap means less boundary context loss, but more stored data and embedding cost; the right value depends on document type and chunk size, so it's determined experimentally with the recall@k measurement described below.

The metadata you store affects production quality more than retrieval itself: document title, section/heading path, source URL, version number, and indexing timestamp let the model cite sources and enable filtered search (e.g. "search only within this product's documentation"). Keeping these fields schematized and consistent — as you would for LLM output — pays off here too; I detailed that discipline in Structured output and JSON Schema in LLMs. Nailing down the metadata schema up front is far cheaper than rebuilding the whole index later.

When hybrid search (BM25 + vector) wins

Pure vector search is strong at finding text that's semantically close but phrased differently; it can weaken on queries needing an exact match — an error code, a product SKU, a rare proper noun — because "closeness" in embedding space doesn't always mean "same token". Anthropic's measurement makes this concrete: with contextual embeddings alone, top-20 failure drops 35% from baseline to 3.7%; adding contextual BM25 (lexical/hybrid search) pushes the reduction to 49%, down to 2.9%. Hybrid search delivers an additional, measurable gain on top of pure semantic search.

On the vector side, OpenAI's embedding models (text-embedding-3-small: 1536 dimensions, 62.3% average MTEB; text-embedding-3-large: 3072 dimensions, 64.6%; both used with cosine similarity, 8192-token input limit) give a strong semantic foundation, but the data above shows complementing it with a lexical layer like BM25 yields additional accuracy over pure vector search. That 8192-token limit is also a constraint to weigh against context-window cost when choosing chunk size — I covered that trade-off in Tokens, context windows, and cost fundamentals in LLMs:

1def hybrid_score(vector_score: float, bm25_score: float, alpha: float = 0.7) -> float:
2    # alpha: weight given to the vector score, (1-alpha): to BM25
3    return alpha * vector_score + (1 - alpha) * bm25_score
4 
5# example: vector 0.82, BM25 0.41
6score = hybrid_score(0.82, 0.41, alpha=0.7)
7print(round(score, 3))  # 0.7*0.82 + 0.3*0.41 = 0.574 + 0.123 = 0.697

In practice, adjusting alpha by query type (more BM25 weight for short/code-like queries) beats keeping it fixed — for technical-documentation RAGs I generally start alpha at 0.6-0.7 and calibrate from measurement.

Reranking: the cost of two-stage retrieval

Anthropic's pipeline is two-stage: a broad candidate pool (top-150) is retrieved, then a reranker progressively narrows it to a final top-20. Adding this stage reduces the total failure rate 67% from baseline, down to 1.9% — below even the 2.9% from contextual embeddings + BM25 alone. Reranking still delivers a measurable gain on top of hybrid search.

This has a cost: the reranker scores hundreds of first-stage candidates individually or in groups, meaning an extra model call and added latency. Contextual chunking itself isn't free either — generating a context sentence per chunk costs $1.02 per million tokens with prompt caching (this figure is for contextualization, not reranking — don't conflate the two).

In a real-time, low-latency chat interface, two-stage retrieval + rerank might not always make sense; the added latency can become noticeable to a waiting user. But in batch use cases, or ones where accuracy matters more than latency (legal/medical document querying, internal knowledge bases), the cost generally pays for itself, since a wrong answer costs far more than the added latency.

Measuring retrieval: recall@k, MRR, answer accuracy

There are two core metrics for measuring retrieval quality instead of just saying "it looks good":

• Recall@k: The rate at which the relevant (ground-truth) chunk appears within the first k retrieved results. As k increases, recall goes up, but the context window also swells, and the model's attention can get diluted by irrelevant content.
• MRR (Mean Reciprocal Rank): The average of the reciprocal of the rank at which the first relevant result appears; a result in position 1 contributes 1, in position 3 contributes 1/3 — it captures ranking quality more precisely than recall does.

When computing both, I usually look at recall@k first, because it answers a simple yes/no question: is the chunk present within the first k or not.

1def recall_at_k(hits: list[bool]) -> float:
2    # hits[i] = True if the relevant chunk for that query was found within the top k results
3    return sum(hits) / len(hits)
4 
5# 4 queries, was the relevant chunk found within the top-5
6hits = [True, True, False, True]
7print(round(recall_at_k(hits), 3))  # 3/4 = 0.75

Recall@k alone isn't enough, because it doesn't distinguish whether the relevant chunk came in at position 1 or position 5; you need MRR to see ranking quality:

1def mrr(ranks: list[int]) -> float:
2    # for each query, the rank of the first relevant result (1-indexed)
3    reciprocal = [1 / r for r in ranks]
4    return sum(reciprocal) / len(reciprocal)
5 
6# 3 queries: the first relevant result came in at rank 1, 3, and 2 respectively
7print(round(mrr([1, 3, 2]), 3))  # (1 + 0.333... + 0.5) / 3 = 0.611

These metrics answer "did retrieval fetch the right chunk," not "did the model use it correctly" — that needs a separate answer-accuracy evaluation (a golden dataset plus a human or LLM-as-judge); I covered that distinction and the practice of building a golden dataset in Evals for LLM applications: golden datasets and LLM-as-judge. Track both together in production: low recall@k means a retrieval problem; high recall@k with a wrong answer means a prompting or context-use problem.

The production improvement loop

Retrieval quality isn't a one-time setting, it's a continuous loop: every query and its retrieved chunks are logged, recall@k and MRR are recomputed on a golden set at regular intervals, query clusters showing a drop are examined (is the chunk size too small, is the hybrid weight wrong, or was the document never indexed at all), the relevant parameter is changed, the index is rebuilt, and the measurement is repeated.

1# a simple offline evaluation loop (pseudocode)
2# run-tag: CHUNKRUN01
3for query in golden_set.jsonl; do
4  retrieved=$(rag_retrieve --query "$query" --top-k 20)
5  echo "$retrieved" | eval_recall_mrr --ground-truth golden_set.jsonl >> metrics.log
6done
7# if metrics.log trends downward, review the chunk/hybrid/rerank parameters

It's also a mistake to build the golden set once and shelve it: queries users actually ask but the system handles poorly should be added regularly from production logs, since user behavior and document content change over time. Before pushing a chunk-size or hybrid-weight change to production, testing it against a sample of recent real queries — not just the golden set — prevents a setting that looked good in the lab from unexpectedly getting worse live.

This loop can also be automated in agent-based systems instead of run as a one-off manual check: modeling retrieval as a tool the model can invoke again on its own judgment — as in AI integration with MCP (Model Context Protocol) — lets it say "these results aren't good enough, try a different query," producing behavior noticeably more resilient than one-shot retrieval on complex, multi-step questions.

FAQ

What's the best chunk size for RAG?

There's no single "best" size; even Pinecone's guide, while recommending fixed-size chunking as the default, notes that following Markdown heading hierarchy in structured documents produces semantically more coherent chunks. The right approach is comparing a few chunk sizes on your own golden set using recall@k.

Is a reranker actually necessary, and how much does it gain you?

In Anthropic's measurement, the contextual embeddings + BM25 hybrid already reduces the top-20 failure rate 49% from baseline (2.9%); adding a reranker takes that to a 67% reduction (1.9%). So it's not necessary, but it does deliver a measurable additional gain — the real question is whether that gain is worth the added latency.

How is retrieval quality measured (recall@k, MRR)?

Recall@k gives the rate at which the relevant chunk is within the first k results; MRR gives the average of the reciprocal of the rank of the first relevant result. Both require a golden set (query plus correct source match) and should be recomputed at regular intervals.

When does hybrid search beat pure vector search?

When the query contains a term that needs an exact match (an error code, a product name, a rare name), hybrid search outperforms pure vector search; in Anthropic's data, adding contextual BM25 further reduces the failure rate compared to using embeddings alone.

Do you recommend a fixed percentage for chunk overlap?

No — instead of giving a fixed number, I recommend trying a few values on your golden set and looking at recall@k; the right value depends on document type.

Should I fix chunking first, or set up hybrid search first?

They don't block each other, but order matters: if chunk boundaries don't match the document structure, neither hybrid search nor a reranker can fix broken chunks, because the correct content to find may already be cut in half. So establish chunking appropriate to the document type (header-aware or fixed-size) first, then add hybrid search, and only add reranking last if needed — in other words, the order is chunking, then hybrid search, then reranking.

Update (September 2026)

This post was written with the tools and architectures of March 2026; as of September 2026, the field has advanced on two axes. First, approaches that hand chunking largely over to the embedding model: Voyage AI's voyage-context-4, announced on June 29, 2026, encodes the whole document in a single pass with full context, taking the position of "stop worrying about chunking," and in its own measurements reports a +2.08% gain over voyage-context-3 at the chunk level; on the LongEmbed evaluation, using contextualized chunk embedding instead of embedding the same documents as a single vector yields a +7.11% gain (blog.voyageai.com, June 29, 2026). Second, rerankers getting smaller and faster: jina-reranker-v3.5 (Aug 3, 2026), at 0.6 billion parameters, reports going from 62.10 to 63.20 on BEIR compared to the previous version while lowering average latency on long documents by 1.56x, but it reports falling behind Qwen3-Reranker-4B — roughly 7x larger — on multilingual and structured IR tasks (jina.ai) — meaning "small and fast always wins" isn't true; it depends on the task.

On the academic side, the ChunkRank study (arXiv:2609.29828), published September 24, 2026, shares a notable negative finding: across three QA datasets (NaturalQuestions, TriviaQA, HotpotQA), for answer selection (not retrieval reranking), content-based rankers are not reliably better than the model simply taking the first non-empty answer. Also, Microsoft added an automatic retrievalReasoningEffort setting (preview) in August 2026 to the structure it brought to GA as "knowledge base" in Azure AI Search in April 2026, bringing an adaptive flow that escalates from a lightweight initial retrieval pass to LLM-based query planning (up to medium effort) when that pass falls short; skipping reranking entirely comes via a separate feature from the same month, the per-source resultsProcessing: none setting (preview) (learn.microsoft.com/azure/search/whats-new). Vector database selection is out of scope for this post; you can find that comparison in Pinecone, Weaviate, Qdrant vector database comparison.

Conclusion

RAG chunking reranking quality isn't a single "magic" setting — it's a system where chunking matched to your document structure, hybrid search that catches exact-match queries, and reranking added only when needed all work together. You can go deeper on the fundamentals in RAG fundamentals: embedding and vector search, on the metadata schema in Structured output and JSON Schema in LLMs, on measurement discipline in Evals for LLM applications: golden datasets and LLM-as-judge, and on automating the production loop in AI integration with MCP. Move your system forward not on a single "looks good" feeling, but by measuring it regularly on your own golden set with recall@k and MRR.

Sources

• Pinecone — Chunking Strategies — a guide to fixed-size, sentence-based, header-aware, semantic, contextual, and chunk-expansion chunking approaches.
• Anthropic Engineering — Contextual Retrieval — the baseline top-20 failure rate, the contributions of contextual embeddings/BM25/rerank, and the two-stage retrieve→rerank pipeline.
• OpenAI — Embeddings guide — text-embedding-3-small/large dimensions, MTEB scores, and token limit.
• Voyage AI — voyage-context-4 announcement — an embedding model that encodes the document with full context.
• Jina AI — jina-reranker-v3.5 — a comparison of a small/fast reranker on BEIR and multilingual search.
• arXiv — ChunkRank (2609.29828) — a study questioning content-based rankers' superiority over the first non-empty answer in answer selection.
• Microsoft Learn — Azure AI Search what's new — knowledge base GA and adaptive retrieval reasoning effort.