Your Retriever Returns 10 Chunks. Your LLM Needs One. Contextual Compression Fixes the Gap.

TL;DR: LangChain's ContextualCompressionRetriever wraps your existing retriever and strips irrelevant content from retrieved chunks before they reach the LLM — no architecture change, two compressor options.

What you'll be able to do after this: Add a compression layer that either drops low-relevance chunks entirely (EmbeddingsFilter, no extra LLM call) or extracts only the relevant sentences from each chunk (LLMChainExtractor, one cheap LLM call per chunk).

Three things to know:

1. EmbeddingsFilter is the fast path: embed the query and each chunk, drop anything below a cosine similarity threshold (0.76 is a reasonable starting value). No LLM call, sub-millisecond overhead per chunk.
2. LLMChainExtractor is the quality path: send each chunk to the LLM and ask it to extract only what's relevant to the query. Adds one call per chunk but can compress a 2,000-token chunk to 80 tokens when most of it is noise.
3. DocumentCompressorPipeline chains them: filter first (EmbeddingsFilter removes irrelevant chunks cheaply), then extract (LLMChainExtractor compresses the survivors). Best of both at the cost of both.

Walk-through — EmbeddingsFilter (fast, cheap):

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import EmbeddingsFilter
from langchain_openai import OpenAIEmbeddings

# Your existing retriever — Chroma, FAISS, pgvector, etc.
base_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})

embeddings_filter = EmbeddingsFilter(
    embeddings=OpenAIEmbeddings(),
    similarity_threshold=0.76  # tune per corpus: lower = more permissive
)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=embeddings_filter,
    base_retriever=base_retriever
)

docs = compression_retriever.invoke("What are the failure modes of RAG pipelines?")
# Typically returns 2-4 chunks instead of 10

Walk-through — LLMChainExtractor (highest quality):

from langchain.retrievers.document_compressors import LLMChainExtractor
from langchain_anthropic import ChatAnthropic

extractor = LLMChainExtractor.from_llm(
    ChatAnthropic(model="claude-haiku-4-5-20251001")  # cheap model works fine here
)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=extractor,
    base_retriever=base_retriever
)

Drop-in replacement: use compression_retriever anywhere you'd use base_retriever.

Real limits. similarity_threshold needs per-corpus tuning — a value that works on one dataset may over-filter on another, silently dropping relevant chunks. LLMChainExtractor adds one LLM call per retrieved chunk: at k=10, that's 10 extra calls before the main generation; on high-traffic systems this doubles both latency and cost. Neither technique fixes a bad retriever — if your vector search returns the wrong documents, compression gives you a smaller wrong set.

Sources: LangChain contextual compression how-to · NirDiamant RAG_Techniques: contextual_compression.ipynb · LangChain OpenTutorial: ContextualCompressionRetriever