Retrieve Small, Return Big: Better RAG With LangChain's ParentDocumentRetriever

Small child chunks find the right passage; large parent chunks give the LLM the context it needs — LangChain's ParentDocumentRetriever indexes both and returns the right one automatically.

What you'll be able to do after this:

• Understand why one chunk size can't satisfy both retrieval precision and LLM context quality
• Configure a dual-granularity index: child chunks for embedding, parent chunks for the LLM
• Wire up ParentDocumentRetriever with Chroma + InMemoryStore in under 30 lines

The core tradeoff

Small chunks (~200-400 tokens) make precise embeddings — a single sentence produces a vector that reliably matches narrow queries. But when the LLM gets a 200-token snippet as context, it's missing the setup paragraph, the surrounding equations, the follow-on example. Answer quality suffers.

Large chunks (~1,500-2,000 tokens) preserve narrative context but dilute the embedding: the vector becomes an average of several topics and won't reliably surface on specific queries.

ParentDocumentRetriever maintains two granularities simultaneously:

• Child chunks (small, ~400 tokens): embedded and stored in the vector store; used for similarity search only
• Parent chunks (large, ~2,000 tokens): stored in a key-value docstore keyed by parent ID; returned to the LLM

Query flow: similarity search finds the best child chunks → look up their parent IDs → return the full parent documents. Precise matching AND rich context.

Runnable setup

from langchain_community.vectorstores import Chroma
from langchain.storage import InMemoryStore
from langchain.retrievers import ParentDocumentRetriever
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings

# Small splitter for retrieval signal, large for LLM context
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)

# Child embeddings → vectorstore; parent text → docstore
vectorstore = Chroma(
    collection_name="split_parents",
    embedding_function=OpenAIEmbeddings()
)
store = InMemoryStore()

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

# add_documents splits, indexes children in vectorstore,
# stores parents in docstore — all in one call
retriever.add_documents(docs)

# Returns full parent chunks (~2,000 tokens), not child snippets
results = retriever.invoke("quantum decoherence in superconductors")
print(len(results[0].page_content))  # → ~2000 chars

Two usage modes

Full document as parent (omit parent_splitter): embed child chunks, return the entire source document. Best for short documents where you want the complete text.

Large chunk as parent (pass parent_splitter): embed children, return medium parent chunks. Best for long documents — gives the LLM a bounded window, not everything.

For production, swap InMemoryStore with a persistent store (RedisStore or a custom ByteStore) so documents survive restarts. The vectorstore (Chroma, pgvector, Pinecone) can also be persisted independently — the two stores are decoupled by design.

Sources: How to use the Parent Document Retriever — LangChain docs · LangChain ParentDocumentRetriever tutorial — YouTube · Parent Document Retrieval with MongoDB — MongoDB Docs