Your Retriever Picks the Wrong Chunk Half the Time — Combine BM25 and Dense Vectors with RRF

TL;DR: Hybrid search combines BM25 (exact-term precision) with dense vectors (semantic recall), fused via Reciprocal Rank Fusion — simpler than score normalization, and reliably better than either retriever alone.

What you'll be able to do after this: Run a retriever that handles both "what is RAG?" and "retrieval-augmented generation latency benchmarks" well, without building two separate pipelines or normalizing scores.

The anchor resource: Hybrid Search in RAG Explained | BM25 + Semantic Search + RRF (May 2026, ~23 min).

Three takeaways:

1. BM25 for precision, vectors for recall. BM25 excels on queries that use the exact terms the document uses — product codes, proper names, technical abbreviations. Dense vectors excel when the user phrases things differently from the document. On the WANDS e-commerce benchmark, a tuned hybrid setup reaches 0.75 NDCG — a 7% lift over BM25 (0.70) or pure vector search (0.70) alone.

2. RRF beats score normalization. Reciprocal Rank Fusion — score = 1/(k + rank), where k=60 is the standard default — ignores raw BM25 and cosine scores entirely, using only rank position. No need to put two differently-scaled scores onto a common axis. It's the default fusion strategy in Weaviate, Elasticsearch, and LangChain's EnsembleRetriever.

3. alpha is your tuning knob. In Weaviate: alpha=0.5 weights both result sets equally; push toward 0 for keyword-heavy corpora (legal, technical docs), toward 1 for open-domain semantic queries. In LangChain, weights=[0.5, 0.5] on EnsembleRetriever does the same.

Code (Python + Weaviate):

results = collection.query.hybrid(
    query="transformer attention mechanism",
    alpha=0.5,           # 0 = BM25 only, 1 = vector only
    return_metadata=MetadataQuery(score=True, explain_score=True)
)

LangChain equivalent:

from langchain.retrievers import EnsembleRetriever
hybrid = EnsembleRetriever(
    retrievers=[bm25_retriever, chroma_retriever],
    weights=[0.5, 0.5]
)

Where hybrid search breaks down:

• Latency: two retrievers run in parallel, adding ~5–10ms at p50 over dense-only — worth profiling before assuming it's free.
• Typos and synonyms: BM25 treats "RAG" and "retrieval-augmented generation" as different terms. Add query expansion or a stemmer if your users abbreviate.
• alpha is dataset-specific. The right weight for financial documents is not the right weight for engineering docs. Tune it on a held-out eval set with actual user queries, not intuition.
• BM25 needs a keyword index. If your vector store doesn't have a built-in sparse index (Chroma, for example, does not), you'll manage a separate BM25 index (rank-bm25, Elasticsearch) — more operational surface.

Sources: YouTube: Hybrid Search in RAG Explained (BM25 + RRF) · Weaviate hybrid search docs · LangChain EnsembleRetriever · MachineLearningMastery: implementing hybrid search in RAG