When retrieval beats a longer context

The question is not "RAG or no RAG". It is "what is the cheapest way to put the right 3,000 tokens in front of the model for this question?" Sometimes that is a search index. Sometimes it is the whole document, cached. Sometimes it is a SQL query. Know the tradeoffs and you can choose on numbers.

When a long context wins

If the corpus is small and stable, skip retrieval. Anthropic's contextual retrieval write-up says a knowledge base under about 200,000 tokens (roughly 500 pages) can go straight into the prompt, and prompt caching makes that fast and cheap. Long context also wins when the task needs global reasoning over the whole thing (summarize, find contradictions) because top-k chunks cannot see what they did not retrieve.

When retrieval wins

  • The corpus is big or changing. You cannot cache what does not fit, and re-embedding one changed file is cheaper than rewriting a prompt.
  • Access differs per user. A search index can filter by permission. A giant shared prompt cannot.
  • You need citations. A retrieved chunk has an id and a source.
  • Quality. Models use information at the start and end of a long input better than the middle (Liu et al., "Lost in the Middle"). Fewer, better tokens often beat more tokens.
  • Cost. Take 1,000 questions a day over a 150,000-token corpus at 3.00 per million input tokens:
    • Stuff it every time: $150{,}000 \times 1000 = 150$M tokens, about 450 per day.
    • Stuff it with the prefix cached at a 0.1 read price: about 45 per day, if traffic keeps the cache warm.
    • Retrieve 6 chunks of 400 tokens plus 1,000 tokens of instructions: $3{,}400 \times 1000 = 3.4$M tokens, about 10 per day.

Hybrid search, and why

Keyword search (BM25) is great at rare exact strings: error codes, function names, product SKUs. Dense embeddings are great at paraphrase: "can't log in" matches "authentication failure". Each misses what the other catches, so run both and fuse the rankings.

Reciprocal rank fusion needs no score calibration. For a document $d$ ranked $\text{rank}_r(d)$ by retriever $r$:

$$\text{RRF}(d) = \sum_{r} \frac{1}{k + \text{rank}_r(d)}, \qquad k = 60$$

Worked example. Document A is rank 1 in BM25 and rank 5 in dense search. Document B is rank 3 in BM25 and rank 1 in dense.

  • A: $1/61 + 1/65 = 0.01639 + 0.01538 = 0.03178$
  • B: $1/63 + 1/61 = 0.01587 + 0.01639 = 0.03227$

B wins narrowly, because being good in both lists beats being great in one. A reranker (a cross-encoder that reads the query and the chunk together) then reorders the fused top 50 for precision. Anthropic reports that adding context to each chunk before embedding and BM25 cut failed top-20 retrievals by 49 percent, and by 67 percent with reranking, on their test sets. Treat those as a direction, not a promise; measure on yours.

Measure retrieval before you blame the model

Build 30 questions with the chunk ids that answer them. Then:

$$\text{recall@}k = \frac{|\text{relevant} \cap \text{top-}k|}{|\text{relevant}|}$$

If recall@10 is 0.6, no prompt will save you: the answer never reached the model. Fix chunking, add hybrid search or rerank until recall is high, then tune generation.

In code

import numpy as np
from rank_bm25 import BM25Okapi

bm25 = BM25Okapi([d.lower().split() for d in docs])


def bm25_rank(query, k=50):
    scores = bm25.get_scores(query.lower().split())
    return list(np.argsort(-scores)[:k])


def dense_rank(query, doc_vecs, embed, k=50):
    q = embed([query])[0]
    sims = doc_vecs @ q / (np.linalg.norm(doc_vecs, axis=1) * np.linalg.norm(q))
    return list(np.argsort(-sims)[:k])


def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)


def recall_at_k(ranked, relevant, k=5):
    return len(set(ranked[:k]) & set(relevant)) / len(relevant)

Here docs is your list of chunk strings, doc_vecs their embeddings as a matrix, and embed your provider's embedding call. The whole pipeline is about 30 lines; the hard part is the question set.

Let the agent drive

A one-shot pipeline retrieves once, before the model has thought. An agent can do better: give it search_docs(query) and read_doc(id, start, end) and let it reformulate, narrow and follow references. For code and logs, plain grep and file reads often beat embeddings. The cost is more steps, so keep each result small and cap the number of searches.

When not to use RAG

  • The question is an aggregate ("how many tickets mention billing this month?"). Use SQL.
  • The data is structured. Give the model a query tool, not chunks of rows.
  • The corpus fits in the window and rarely changes. Cache it.
  • You have no retrieval eval. You cannot improve what you cannot score.

Common mistakes

  • Chunking by fixed character count and cutting tables and code in half. Split on structure and keep a little overlap.
  • Embedding the query with one model and the corpus with another.
  • Skipping the keyword leg and then wondering why error codes never match.
  • Returning chunks with no source, so answers cannot be checked.
  • Tuning the prompt while recall@10 is 0.5.

Check yourself

  • Compute RRF for a document ranked 2 and 4. Is it above or below document B?
  • For which of these would you pick long context, and for which retrieval: a 40-page policy, a 4M-token wiki, a customer's private notes?
  • Why is recall@k the first number you measure, not answer quality?
  • Give one query where BM25 beats dense and one where dense beats BM25.