Two-Stage RAG: Add a Cross-Encoder Reranker and Stop Feeding Your LLM Irrelevant Context

Chris Harper

3 min read

Jul 31, 2026 · 20:05 UTC

Tutorial

RAG

Best Practices

Pull 100 candidates with your bi-encoder, pass them through a cross-encoder, and return the precision-ranked top 10 to your LLM — the highest-ROI improvement to most RAG pipelines, in 10 lines of Python.

What you'll be able to do after this:

Understand why bi-encoder vector search recalls well but ranks poorly
Add a cross-encoder reranker to any RAG pipeline — LangChain, LlamaIndex, or raw Python
Choose the right reranker for your latency budget (local model vs. Cohere/Jina/Voyage API)

Why vector search alone isn't enough

When you do ANN search over your vector store, the bi-encoder encodes the query and each document independently — they never see each other. That's what makes bi-encoders fast (no pair computation at query time), but it's also why ranking is noisy: two documents can have similar cosine distances but very different relevance to your specific query.

Cross-encoders fix this. A cross-encoder processes the query and the document together through the full transformer stack — every layer attends over both texts simultaneously. This catches subtle query-specific relevance cues that bi-encoders miss. The cost: you can't pre-compute anything, so you run it once per candidate pair. That's why you only rerank the top-k from your first stage (typically 50–100), not your whole corpus.

The two-stage pipeline

Query → bi-encoder ANN search → top-100 candidates
                               → cross-encoder → top-10 → LLM

Code: 10 lines with sentence-transformers

from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

bi_encoder = SentenceTransformer("BAAI/bge-m3")
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve_and_rerank(query: str, corpus: list[str], top_k=10, candidate_n=100):
    # Stage 1: fast ANN retrieval (replace with your vector store)
    q_emb = bi_encoder.encode(query, normalize_embeddings=True)
    c_embs = bi_encoder.encode(corpus, normalize_embeddings=True, batch_size=64)
    candidate_ids = np.argsort(q_emb @ c_embs.T)[::-1][:candidate_n]

    # Stage 2: rerank the short list
    pairs = [[query, corpus[i]] for i in candidate_ids]
    rerank_scores = cross_encoder.predict(pairs)
    best_ids = candidate_ids[np.argsort(rerank_scores)[::-1][:top_k]]
    return [corpus[i] for i in best_ids]

Pick your reranker

Model	Latency / 100 pairs	Quality	Best for
`cross-encoder/ms-marco-MiniLM-L-6-v2`	~30ms	Good	Prototyping, edge
`cross-encoder/ms-marco-MiniLM-L-12-v2`	~60ms	Better	Server-side
`BAAI/bge-reranker-v2-m3`	~80ms	Best local	Production (multilingual)
Cohere Rerank 3	~150ms (API)	Excellent	Low-ops prod
Voyage Rerank 2	~120ms (API)	Excellent	Claude-heavy stacks

Start with ms-marco-MiniLM-L-6-v2 — small, fast, and widely benchmarked. Upgrade to bge-reranker-v2-m3 when you need multilingual support or higher precision.

When NOT to add a reranker

Ultra-low latency — the second model call adds latency; fix chunking instead
Corpora under ~500 docs — bi-encoder precision is already high; overhead isn't worth it
Pre-filtered retrieval — if metadata filters already narrow candidates at read time, first-stage recall is high

For everything else — general document corpora, multi-tenant knowledge bases, long-context RAG — a reranker is among the highest-ROI retrieval improvements available.