Your Vector Search Gets the Right Documents — Just in the Wrong Order. A Cross-Encoder Fixes That.

Retrieve 50 candidates with a bi-encoder; rerank them with a cross-encoder that reads query + document together — 15 lines that lift NDCG@10 by +5 to +15 points.

What you'll be able to do after this: add a cross-encoder reranking stage to any existing vector-search RAG pipeline and measure its impact on retrieval quality before sending results to the LLM.

Why bi-encoders leave quality on the table

Your bi-encoder (the embedding model behind most vector stores) compresses the query into one vector and each document into another, then compares them at retrieval time. That is fast — sub-millisecond at scale — but it is lossy: the query and the document never "see" each other during encoding. Token-level interactions, negations, and multi-word phrase semantics get flattened into a fixed-size vector.

A cross-encoder reads the query and document together in a single forward pass, producing a relevance score rather than an embedding. It cannot pre-index documents (no embedding to store), so it cannot replace your vector store — but it is the right tool for reordering a short candidate list.

Two-stage pattern

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

# Stage 1: vector store retrieves top-50 candidates (fast)
candidates = vector_store.similarity_search(query, k=50)

# Stage 2: cross-encoder scores each candidate (slower, but only 50 docs)
pairs = [(query, doc.page_content) for doc in candidates]
scores = reranker.predict(pairs)

# Sort by cross-encoder score, keep top-5 for the LLM
reranked = [doc for _, doc in sorted(zip(scores, candidates), reverse=True)]
top_docs = reranked[:5]

BAAI/bge-reranker-v2-m3 is the current recommended general-purpose local model: multilingual, MTEB/BEIR benchmarks show +5 to +15 NDCG@10 over bi-encoder-only retrieval on most open domain tasks.

When this is worth adding

✅ Add it when: your vector retrieval recall@50 is solid (the right documents are in the top 50) but top-5 precision is poor (wrong order → LLM gets suboptimal context)

❌ Don't add it when: recall@50 is low — if the right document is not in the top 50, no reranker can help. Fix your embeddings or chunking first, then add a reranker.

Latency reality check: a cross-encoder on 50 candidates takes 50–400 ms depending on document length and hardware. That is fine for search-augmented pipelines; it may be tight for latency-critical applications. Batch your predictions (reranker.predict(pairs) already does this).

Token limit: most local cross-encoders truncate at 512 tokens (query + document combined). Long documents need chunking before reranking or you need a hosted model with extended context.

Hosted alternatives when you need longer context

• Cohere Rerank 3 — 4 096-token context, $1–2 per 1 000 queries, strong multilingual performance; one API call replaces the local model
• Jina Reranker v2 — 8 192-token context, similar pricing tier; good for long document retrieval

Both are drop-in replacements for the local pattern: same two-stage flow, API call instead of reranker.predict().

Sources: TDS: Advanced RAG Retrieval — Cross-Encoders & Reranking · Local AI Master: BGE, Cohere, Jina reranking guide · BAAI/bge-reranker-v2-m3 on HuggingFace