Your Retriever Is Finding Noise: LangChain's ContextualCompressionRetriever Fixes It

ContextualCompressionRetriever wraps any existing LangChain retriever and sends each retrieved chunk through an LLM that strips everything but the query-relevant sentences — cutting noise, reducing tokens, and improving generator answer quality in three lines of code.

What you'll be able to do after this:

• Wrap any existing LangChain retriever (FAISS, Chroma, pgvector) in a ContextualCompressionRetriever in three lines
• Choose between LLMChainExtractor (extract relevant sentences) and LLMChainFilter (drop irrelevant chunks entirely)
• Understand the quality-vs-latency trade-off and when compression is — and isn't — worth it

Standard RAG retrieves the top-k chunks semantically similar to the query, then passes all of them to the generator. The problem: those chunks were written as complete documents, not query answers. Half the text in each chunk is often surrounding boilerplate, adjacent paragraphs, and headers that happened to be in the same chunk as the one useful sentence. You pay for all of it in tokens, and the generator wades through the noise.

ContextualCompressionRetriever adds a compression step between retrieval and generation: each chunk passes through a compressor that extracts or filters down to only what the current query actually needs.

Set it up in 5 lines

from langchain_anthropic import ChatAnthropic
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor

llm = ChatAnthropic(model="claude-sonnet-5")

# base_retriever is any retriever you already have (FAISS, Chroma, pgvector...)
compressor = LLMChainExtractor.from_llm(llm)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever,
)

# Drop-in replacement — same .invoke() interface
docs = compression_retriever.invoke("What are the auth requirements for the payment API?")

The extractor is called once per retrieved chunk with the query and the chunk text. It returns only the sentences directly relevant to the query, or an empty string if nothing is relevant.

LLMChainFilter: drop whole chunks instead of extracting

For document collections where some chunks are entirely unrelated, LLMChainFilter is cheaper: rather than extracting sentences, it runs a binary relevance decision and either keeps or discards each chunk.

from langchain.retrievers.document_compressors import LLMChainFilter

filter_compressor = LLMChainFilter.from_llm(llm)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=filter_compressor,
    base_retriever=base_retriever,
)

When to use which

• LLMChainExtractor: chunks are long and partially relevant — technical docs, long prose, reference manuals. Extracts the useful sentences; discards the rest.
• LLMChainFilter: chunks are short and either clearly relevant or clearly not — product FAQs, structured records. Cheaper per call than extracting; filters at chunk level.
• EmbeddingsFilter: high-throughput paths where LLM latency per chunk is unacceptable. Uses a cosine similarity threshold — no LLM call, faster, but less precise than extraction.
• Skip compression: chunks are already tightly scoped (code snippets, short Q&A pairs). Compression adds latency without quality benefit.

The trade-off: compression adds one LLM call per retrieved chunk to the pipeline. On a retrieval returning five chunks, that's five extra Claude calls before generation. Worth it when the chunks are noisy (typical for general document corpora); not worth it when chunks were already scoped at write time.

Sources: Improving Document Retrieval with Contextual Compression — LangChain Blog · LangChain Contextual Compression with LLMChainExtractor and FlashRerank — YouTube · ContextualCompressionRetriever — LangChain OpenTutorial