Cut Failed Retrievals by 49%: Add Document Context to Every Chunk Before Embedding

TL;DR: Chunks lose their document identity when split — Anthropic's Contextual Retrieval prepends a 50-100 token LLM-generated context summary to each chunk before embedding AND BM25 indexing, cutting retrieval failures by 49% (67% when you add a reranker).

What you'll be able to do after this:

• Fix the core failure mode of RAG: chunks that retrieve for the wrong query because they carry no document context
• Generate context summaries for thousands of chunks cheaply using prompt caching (90% cost cut vs. naive per-chunk API calls)
• Stack Contextual Embeddings + Contextual BM25 + reranking to hit the full 67% failure reduction

Why chunks fail

When you split a 40-page product spec and embed chunk #18 — "The maximum payload is 2 MB" — the embedding captures a generic constraint. It retrieves for every payload-related query regardless of product, version, or context. The fix: tell each chunk what document it came from before you embed it.

The technique: context-prepend before embed and BM25

For each chunk, ask Claude to generate a short (50-100 token) context summary explaining the chunk's role in its parent document. Prepend that summary to the chunk text. Use the enriched text for both the vector embedding and the BM25 keyword index.

import anthropic

client = anthropic.Anthropic()

CONTEXT_PROMPT = """<document>
{document}
</document>

Situate this chunk for retrieval (50-100 tokens, no preamble):
<chunk>
{chunk}
</chunk>"""

def add_context(document: str, chunk: str) -> str:
    r = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=150,
        messages=[{"role": "user", "content": CONTEXT_PROMPT.format(
            document=document, chunk=chunk)}]
    )
    return r.content[0].text

# Enrich before BOTH your embed call and BM25 index:
context = add_context(full_doc, chunk_text)
enriched = f"{context}\n\n{chunk_text}"

Cut the cost 90% with prompt caching

For a document with 500 chunks, calling Claude 500 times with the full document in every prompt is expensive. Use cache_control on the document prefix — you pay full price once; subsequent chunks hit the cache at ~10% of the input cost:

r = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=150,
    messages=[{"role": "user", "content": [
        {"type": "text", "text": f"<document>\n{document}\n</document>",
         "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": f"Situate this chunk:\n<chunk>\n{chunk}\n</chunk>"}
    ]}]
)

Stack it: Contextual BM25 + vector + reranking

Apply the same context prepending to your BM25 keyword index, not just embeddings. Anthropic's benchmarks across codebases, scientific papers, and fiction:

Configuration	Retrieval failure reduction
Contextual Embeddings + Contextual BM25	49%
+ Reranker	67%

The Claude Cookbook has the full implementation: BM25 indexing (via bm25s), Reciprocal Rank Fusion code, and the evaluation dataset.

Sources: Contextual Retrieval in AI Systems — Anthropic · Enhancing RAG with Contextual Retrieval — Claude Cookbook · DataCamp implementation guide