The Cut That Breaks Your RAG: A Beginner's Guide to Chunking Strategies

TL;DR: Document chunking is the most-overlooked RAG variable — the wrong split size or strategy silently tanks retrieval quality before the LLM ever runs. Here's how to get it right from the start.

What you'll be able to do after this:

• Pick the right splitting strategy (fixed-size, recursive, semantic) for your document type and query mix
• Configure chunk_size and chunk_overlap with numbers that actually work for your use case
• Avoid the three most common chunking mistakes that silently lower retrieval recall by 20–30%

Resource: Why Your RAG Gives Wrong Answers (And 4 Chunking Strategies to Fix It) — YouTube. Covers the strategies below in video form with live demos; pair it with this walkthrough.

---

The three strategies

1. CharacterTextSplitter (fixed-size baseline)

Split every N characters regardless of content. Fast and deterministic, but splits mid-sentence and destroys semantic context at boundaries.

from langchain_text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(chunk_size=512, chunk_overlap=64)
chunks = splitter.split_text(raw_text)

Use this only as a baseline to verify your pipeline end-to-end; don't ship it.

2. RecursiveCharacterTextSplitter (the production default)

Tries separators in priority order: `

→ →. → ` → character. Steps down only when the current separator doesn't fit the budget. Respects natural document boundaries while still hitting your size limit.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["

", "
", ". ", " ", ""]
)
docs = splitter.create_documents([raw_text])

This is the right default for most corpora. Start here.

3. SemanticChunker (embedding-based grouping)

Embeds each sentence and groups adjacent sentences whose cosine similarity passes a threshold. Produces semantically coherent chunks — ideal for technical reports and research papers where topics shift gradually.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
chunker = SemanticChunker(embeddings, breakpoint_threshold_type="percentile")
docs = chunker.create_documents([raw_text])

Cost: one embedding call per sentence. Worth it for long-form structured documents; overkill for short mixed corpora.

---

The two numbers that matter

• chunk_size: 256–512 tokens for factoid/lookup queries; 1024–2048 for analytical queries that need surrounding context. Start at 512 and tune by measuring retrieval recall on 20–30 representative questions.
• chunk_overlap: 10–20% of chunk_size. Overlap prevents a key sentence from spanning two chunks and appearing in neither. At 512 chunks, 64 characters of overlap is a solid starting point.

The silent killer

Splitting code or Markdown at fixed character boundaries. Use structure-aware splitters for these:

from langchain_text_splitters import MarkdownTextSplitter, PythonCodeTextSplitter

md_splitter = MarkdownTextSplitter(chunk_size=1000)
py_splitter = PythonCodeTextSplitter(chunk_size=1000)

These preserve headings and function boundaries so a chunk never starts mid-code-block or mid-header.

Sources: Why Your RAG Gives Wrong Answers (YouTube) · LangChain Text Splitters Reference · Best Chunking Strategies for RAG 2026 — Firecrawl