Stop Sending One Query: RAG Fusion Gets You Better Retrieval for Free

TL;DR: RAG Fusion generates 3–5 variants of the user's query, retrieves for each, then merges the ranked lists with Reciprocal Rank Fusion so documents that consistently score well across multiple phrasings beat any single-query result.

What you'll be able to do after this:

• Generate multiple semantically varied queries from a single user question using an LLM — no manual prompt engineering required
• Apply the RRF formula to fuse ranked retrieval lists so documents appearing consistently across query variants rise to the top
• Wire this into a LangChain chain in under 30 lines, using the provided Colab notebook to test against your own data immediately

Resource: Advanced RAG 06 — RAG Fusion by Sam Witteveen (YouTube, 13 min, with Colab notebook and code repo) — covers full implementation from retriever setup to the final fusion chain.

Why single-query retrieval misses things

A user asking "how do I debug a loop that only fails on the third iteration?" is phrased so specifically that a vector store may miss semantically equivalent documents phrased differently. RAG Fusion sidesteps the query-phrasing problem by generating multiple angles — at least one phrasing will match.

The RRF formula

RRF scores a document by summing its reciprocal rank across all retrieved lists:

score(d) = Σᵢ 1 / (k + rankᵢ(d))

k = 60 is a smoothing constant that prevents low-ranked documents from being penalized too harshly. A document ranked #1 in two query-result lists scores 1/61 + 1/61 ≈ 0.033; a document ranked #100 in one list scores just 0.006. Documents consistently near the top across variants dominate the final ranking.

LangChain implementation

from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain_anthropic import ChatAnthropic
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

# 1. Build your vector store as normal
vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())
base_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})

# 2. Generate multiple query variants + retrieve for each
llm = ChatAnthropic(model="claude-haiku-4-5-20251001")
mq_retriever = MultiQueryRetriever.from_llm(
    retriever=base_retriever,
    llm=llm,  # generates ~3 variants, retrieves for each, deduplicates
)

# 3. For explicit RRF across different retriever types (BM25 + vector)
bm25 = BM25Retriever.from_documents(docs, k=10)
ensemble = EnsembleRetriever(
    retrievers=[bm25, base_retriever],
    weights=[0.5, 0.5],  # EnsembleRetriever applies RRF internally
)

MultiQueryRetriever handles query generation; EnsembleRetriever applies RRF across different retriever types. Combine them for the full pipeline.

When to use it

RAG Fusion pays off most when: users phrase questions unpredictably, your corpus covers the same concept in multiple writing styles, or you're already doing hybrid search and want a principled merge rather than a fixed weighted sum.

Sources: Advanced RAG 06 — RAG Fusion (YouTube) · RAG-Fusion paper — Zackary Rackauckas · LangChain MultiQueryRetriever docs · LangChain EnsembleRetriever docs