Dense Embeddings Flatten a Document to One Vector: ColBERT Keeps Every Token — RAGatouille Makes It Three Lines

ColBERT keeps a separate embedding per token instead of one per document — dense embeddings lose exact-match signals ColBERT catches; RAGatouille makes it three lines.

What you'll be able to do after this:

• Build a ColBERT index and search it in under 20 lines of Python with RAGatouille
• Explain why single-vector (biencoder) embeddings lose per-token precision and when that costs recall
• Use ColBERT as a drop-in reranker on top of any retriever you already have

Why dense-only fails on precision queries

A biencoder embedding model flattens every document to a single 768-d vector via mean pooling. Two documents mentioning "HNSW" and "pgvector" score almost identically against a query for either — the averaging hides token-level specificity. ColBERT's late interaction keeps one embedding per token for every document. At query time it computes MaxSim: for each query token, find its highest-scoring match across all document tokens, then sum those per-token scores. The "HNSW" token in a query gets a strong match against the document that uses it, not the one that doesn't.

Walk-through with RAGatouille

pip install ragatouille

from ragatouille import RAGPretrainedModel

# Step 1: Index (downloads ColBERTv2 weights ~500MB on first run)
RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")

documents = [
    "pgvector stores vector embeddings natively inside PostgreSQL.",
    "HNSW is a graph-based approximate nearest-neighbor index algorithm.",
    "ColBERT keeps a separate embedding per token for late interaction scoring.",
]

RAG.index(
    index_name="my-docs",
    collection=documents,
    document_ids=["doc-0", "doc-1", "doc-2"],
)

# Step 2: Search
results = RAG.search("how does HNSW differ from flat vector search?", k=3)
for r in results:
    print(r["rank"], round(r["score"], 2), r["content"][:70])

Use it as a reranker (no index needed)

If you already have a retriever, ColBERT can re-score its top-N candidates on the fly without a pre-built index:

# Pass any candidate strings from your existing retriever
reranked = RAG.rerank(
    query="pgvector vs HNSW: which index is faster?",
    documents=initial_candidates,  # list[str] from your dense retriever
    k=5,
)

This is the lowest-friction entry point: keep your existing embedding pipeline, add .rerank() as a second-stage pass to improve precision at virtually no infrastructure cost.

When to reach for ColBERT: high-precision technical docs, code search, legal text, or any domain where exact terminology matters. For general semantic similarity, dense embeddings remain fast and sufficient.

Sources: RAGatouille — GitHub · Late Interaction Overview — Weaviate · Supercharge Your RAG with Late Interactions — YouTube