Why Your RAG Retriever Is Missing Half the Answers: Hybrid Search With BM25, Vectors, and Reciprocal Rank Fusion

TL;DR: Pure vector search misses exact keywords; pure BM25 misses semantic meaning. Hybrid search runs both in parallel and merges ranks with Reciprocal Rank Fusion — consistently better recall than either alone, with no new infrastructure.

What you'll be able to do after this:

• Understand when dense vector search fails and when BM25 catches it (and vice versa)
• Run both retrievers in parallel and merge results with Reciprocal Rank Fusion (RRF)
• Add a cross-encoder reranker as a precision layer on top of hybrid retrieval

The problem with pure vector search

Most RAG pipelines use only dense vector search. That means they fail on exact-match queries — ask "what is the FDA approval date for drug X?" and an embedding model compresses the specific date out of the comparison. BM25 has the reverse problem: it matches exact tokens but misses "vehicle" when the document says "car."

Hybrid search fixes this by running both retrievers independently and fusing their ranked results.

Implementation: rank_bm25 + sentence-transformers + RRF

pip install rank_bm25 sentence-transformers

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, util

# BM25 side
tokenized_docs = [doc.split() for doc in documents]
bm25 = BM25Okapi(tokenized_docs)
bm25_scores = bm25.get_scores(query.split())
bm25_ranked = sorted(range(len(documents)), key=lambda i: bm25_scores[i], reverse=True)

# Vector side
model = SentenceTransformer('all-MiniLM-L6-v2')
doc_embeddings = model.encode(documents, convert_to_tensor=True)
query_emb = model.encode(query, convert_to_tensor=True)
cos_scores = util.cos_sim(query_emb, doc_embeddings)[0]
vec_ranked = sorted(range(len(documents)), key=lambda i: cos_scores[i].item(), reverse=True)

# Reciprocal Rank Fusion (k=60 is standard)
k = 60
rrf_scores = {}
for rank, idx in enumerate(bm25_ranked):
    rrf_scores[idx] = rrf_scores.get(idx, 0) + 1 / (k + rank + 1)
for rank, idx in enumerate(vec_ranked):
    rrf_scores[idx] = rrf_scores.get(idx, 0) + 1 / (k + rank + 1)

hybrid_ranked = sorted(rrf_scores, key=rrf_scores.get, reverse=True)
top_docs = [documents[i] for i in hybrid_ranked[:top_k]]

RRF is rank-only — no score normalization, no tuning the relative weights of BM25 vs vector scores. Both lists vote by rank, and documents that rank well in both float to the top.

Optional third layer: cross-encoder reranking

After retrieving top-50 with hybrid search, pass all 50 to a cross-encoder (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2). The cross-encoder sees the full query + document pair together — far richer context than any bi-encoder. The standard production pattern is: retrieve-1000 → hybrid-50 → rerank-10.

The retrieve-then-rerank approach keeps latency acceptable (the cross-encoder only scores 50 candidates, not thousands) while giving the final ranking the precision of a model that understands the query-document relationship directly.

Sources: Implementing Hybrid Semantic-Lexical Search in RAG — MachineLearningMastery.com · The Complete Guide to Hybrid Search in RAG (BM25 + Embeddings + Reranker) — YouTube · Hybrid Search and Re-Ranking in Production RAG — Towards Data Science