Your Retriever Doesn't Know Which Documents Are Relevant — Metadata Filters Tell It Before Similarity Search Runs

What you'll be able to do after this: query your vector store with natural language like "show me Python tutorials from 2024" and have the retriever automatically apply a structured filter — language=Python AND year=2024 — before similarity search runs inside that subset.

Why cosine similarity alone isn't enough

Cosine similarity compares embedding vectors. It does not understand dates, categories, or document sources. A query like "what were the Q3 revenue numbers?" can return semantically close content from Q1 or Q2 if those happen to be nearest in embedding space. Metadata filtering adds an explicit pre-filter step: apply a WHERE clause first, then run similarity search inside the filtered subset.

Three takeaways

• Metadata filters run before similarity scoring — they narrow the haystack, they don't re-rank it.
• You can mix structured filters (year > 2023, category = "finance") with free-text semantic similarity in the same query.
• LangChain's SelfQueryRetriever uses a small LLM call to translate natural language into filter predicates automatically — your users write plain queries, not structured filters.

Walk-through with Chroma and LangChain

First, store documents with metadata at index time:

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

# Each doc.metadata carries fields like:
# {"source": "blog", "year": 2024, "topic": "rag", "language": "python"}
vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())

Declare the metadata fields the retriever is allowed to filter on:

from langchain.chains.query_constructor.base import AttributeInfo
from langchain.retrievers.self_query.base import SelfQueryRetriever
from langchain_openai import ChatOpenAI

metadata_field_info = [
    AttributeInfo(name="source",   description="Document source: blog, docs, or wiki",        type="string"),
    AttributeInfo(name="year",     description="Year the document was published",               type="integer"),
    AttributeInfo(name="topic",    description="Primary topic tag, e.g. rag, embeddings, llm", type="string"),
    AttributeInfo(name="language", description="Programming language featured, if any",         type="string"),
]

retriever = SelfQueryRetriever.from_llm(
    llm=ChatOpenAI(model="gpt-4o-mini"),   # or claude-haiku-4-5 — filter construction is cheap
    vectorstore=vectorstore,
    document_contents="Technical articles about building AI applications",
    metadata_field_info=metadata_field_info,
)

Query with plain English — the LLM translates it into a Chroma filter automatically:

docs = retriever.invoke("Python RAG articles from 2024")
# Chroma filter applied: {"year": {"$eq": 2024}, "topic": {"$eq": "rag"}, "language": {"$eq": "python"}}

Three limits to know before you ship

1. Latency: the filter-construction LLM call adds 100–500 ms. For latency-critical paths, build filters explicitly from structured query parameters (a date picker, a category dropdown) and skip the LLM translation entirely.

2. Comparator support varies by store: Chroma supports $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin. Qdrant, Pinecone, and Weaviate support overlapping but non-identical sets. Check your store's comparator list before designing the metadata schema — a filter your store can't execute silently falls back to full scan.

3. Filter accuracy depends on field descriptions: the LLM generates filters from the AttributeInfo descriptions. Ambiguous descriptions produce wrong filters at the edges. Test queries that cross multiple fields before going to production.

Sources: LangChain self-querying retrieval — official docs · How to Build a RAG System with a Self-Querying Retriever — Towards Data Science · Advanced RAG techniques Part 7 — Medium