Embed Sentences, Answer With Paragraphs: Parent-Document Retrieval Fixes the Chunk-Size Tradeoff

Stop tuning chunk size. Embed small chunks for precise retrieval, then send the LLM the larger passage each chunk came from.

Every RAG pipeline hits the same wall. Small chunks embed cleanly and match queries precisely, but hand the model a sentence with no surrounding context. Large chunks carry context but dilute the embedding, so retrieval gets vague. Tuning the number trades one failure for the other.

What you'll be able to do after this:

• Retrieve on precise, sentence-sized embeddings without starving the model of context
• Wire it up in LangChain with two splitters instead of one
• Know what it costs, so you can decide whether it's worth it

How it works. You store two representations. Small child chunks go in the vector store and are the only thing you search. Each child keeps the id of the larger parent passage it came from, and parents live in a separate docstore. At query time: similarity search hits a child → read its parent id → fetch the parent from the docstore → send the parent to the model. You search the needle and answer with the paragraph.

In LangChain this is ParentDocumentRetriever, built from a child_splitter, a parent_splitter, a vectorstore, and a docstore. Rushank Savant's walk-through uses a 100-character child splitter with no overlap against a 400-character parent splitter with 50-character overlap — a useful starting ratio. LlamaIndex does the same thing differently: SentenceWindowNodeParser embeds each sentence but stashes the neighbouring sentences in metadata, then MetadataReplacementPostProcessor swaps the window back in before the model sees it.

What it costs. That write-up is honest about the tradeoffs, and they're real: you store your corpus twice, the id lookup adds a hop of latency, overlapping parents mean you resend some tokens, and the vector store and docstore now have to stay in sync. A fixed window also can't help when the context you need sits outside it.

Sources: Sentence window retrieval — Rushank Savant · LangChain ParentDocumentRetriever reference · Chunking for RAG — Vectara