Retrieve 100, Keep 5: Cross-Encoder Reranking Closes the Precision Gap in RAG Pipelines

TL;DR: Add a cross-encoder reranker after your vector search to re-score the top-N candidates jointly — this second stage cuts irrelevant chunks before the LLM ever sees them, typically lifting RAG precision by 5-15 NDCG points with ~10 lines of Python.

What you'll be able to do after this:

• Understand why bi-encoder (vector) retrieval optimizes for recall but needs a second precision stage
• Run sentence-transformers CrossEncoder to rerank top-N vector search results in ~10 lines of Python
• Pick the right open-weight reranker model for your workload (size vs. quality trade-off)

---

The canonical RAG setup retrieves the top-50 or top-100 nearest chunks from a vector index and sends them to the LLM. Two problems: (1) many of those chunks aren't actually relevant to the query, and (2) they may not all fit in the LLM's context window. A bi-encoder (embedding model) optimizes for recall — it encodes query and document separately and compares their vectors. It never sees how the two texts relate to each other. A cross-encoder fixes this at the precision stage.

How a cross-encoder works

A cross-encoder takes a [query, document] pair as one input sequence. Both texts flow through every transformer layer together, so attention can model their interaction directly. The output is a single relevance score. Cross-encoders are slower than bi-encoders — they can't precompute document representations ahead of time — so they run as a second stage only over the small candidate set the bi-encoder already retrieved.

Walk-through

Step 1: Vector search for top-50 candidates

Use any vector store you already have (Chroma, FAISS, pgvector). Nothing changes here.

candidates = vector_store.similarity_search(query, k=50)

Step 2: Rerank with a CrossEncoder

from sentence_transformers import CrossEncoder

# BAAI/bge-reranker-v2-m3 is the best open-weight English + multilingual reranker
# cross-encoder/ms-marco-MiniLM-L-6-v2 is smaller and faster — great default to start
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

pairs = [[query, doc.page_content] for doc in candidates]
scores = reranker.predict(pairs)   # shape: (50,) — one float per pair
ranked = sorted(zip(scores, candidates), reverse=True)
top_k = [doc for _, doc in ranked[:5]]

Step 3: Pass only the reranked top-5 to the LLM

context = "\n\n".join(doc.page_content for doc in top_k)
# ... build your prompt and call the LLM as normal

Choosing a reranker model

Model	Size	Best for
cross-encoder/ms-marco-MiniLM-L-6-v2	22M	Speed-first; great default
BAAI/bge-reranker-v2-m3	568M	Best quality; multilingual
Cohere Rerank API	hosted	Managed option ($2/1,000 queries)

On CPU, expect 100-300ms for 50 candidates with the MiniLM model. On GPU, under 50ms.

Sources: Reranking in RAG From Scratch (2026) — YouTube · sentence-transformers CrossEncoder docs · BAAI/bge-reranker-v2-m3 — HuggingFace