Your Embedding Model Sees Chunks, Not Documents: Three Splitting Strategies Every RAG Engineer Needs

Fixed-size splitting is fast but breaks prose at arbitrary boundaries. Recursive splitting respects paragraph structure by default. Semantic splitting places boundaries where topics actually shift — three strategies that cover 90% of RAG pipelines.

What you'll be able to do after this:

• Pick the right chunking strategy for your document type (prose, code, markdown, PDFs)
• Wire LangChain's RecursiveCharacterTextSplitter with the right chunk_size and chunk_overlap for your use case
• Add a SemanticChunker to split at topic boundaries without embedding overhead during query time

Before any text reaches your embedding model, you've already made the most consequential decision in your RAG pipeline: how to split it. Too small — the model has no context. Too large — the model gets a wall of text. Wrong boundaries — a sentence is cut mid-thought and retrieval fails for exactly the queries it should handle.

Strategy 1: Fixed-size (CharacterTextSplitter)

Split every N characters with an M-character overlap. Fast, simple, but cuts at arbitrary positions regardless of sentence or paragraph structure.

from langchain_text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separator="
",
)
chunks = splitter.split_text(document_text)

Use for: structured data exports (CSV rows, JSON dumps), logs, or text with no prose structure. Avoid for: any natural-language document — it splits sentences mid-way.

Strategy 2: Recursive splitting (the default for most RAG pipelines)

RecursiveCharacterTextSplitter tries separators in order: `["

", " ", " ", ""]. It keeps paragraphs together first, then sentences, then words — only falling back to finer splits when the chunk would exceed chunk_size`. No embedding API calls during ingestion.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,      # tokens roughly; embedding models prefer 256-512
    chunk_overlap=64,    # 10-15% overlap captures cross-boundary context
    length_function=len,
    separators=["

", "
", " ", ""],  # default; works for prose
)
chunks = splitter.create_documents([document_text])

Chunk size guidance: 256–512 tokens for Q&A retrieval, 512–1024 for summarization tasks. Smaller improves precision; larger improves recall. Measure both with RAGAS evals on a held-out question set.

For Markdown or code, use language-aware separators:

from langchain_text_splitters import RecursiveCharacterTextSplitter, Language

# Markdown (splits on headings, code blocks, paragraphs)
md_splitter = RecursiveCharacterTextSplitter.from_language(
    Language.MARKDOWN, chunk_size=512, chunk_overlap=64
)

# Python (splits on class/function definitions)
py_splitter = RecursiveCharacterTextSplitter.from_language(
    Language.PYTHON, chunk_size=512, chunk_overlap=64
)

Strategy 3: Semantic splitting (topic-boundary detection)

SemanticChunker uses sentence embeddings to detect where topics shift. It embeds adjacent sentence groups, computes cosine similarity, and places a chunk boundary when similarity drops below a percentile threshold.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_huggingface import HuggingFaceEmbeddings  # free, no API key

embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5")
splitter = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",   # "standard_deviation" or "gradient" also available
    breakpoint_threshold_amount=95,            # split at the 95th percentile of similarity drops
)
chunks = splitter.split_text(document_text)

The cost: every ingestion run calls your embedding model per sentence to detect breakpoints — roughly 10–30% more ingestion time. Worth it for multi-topic documents (research papers, legal briefs, long-form reports). Not worth it for short, homogeneous texts.

Which strategy to use?

Document type	Recommended strategy
Blog posts, documentation, emails	RecursiveCharacterTextSplitter
Python / JavaScript source files	from_language(Language.PYTHON) etc.
Markdown docs with headings	from_language(Language.MARKDOWN)
Research papers, multi-topic reports	SemanticChunker
CSV rows, JSON, structured exports	CharacterTextSplitter

Start with RecursiveCharacterTextSplitter at 512 tokens / 64 overlap. Run retrieval evals on a held-out question set. Only upgrade to SemanticChunker if precision is failing and the added ingestion cost is acceptable.

Sources: Chunking Strategies for LLM Applications — Pinecone · Text Splitters — LangChain Docs · RAG Tutorial #8: Text Chunking Strategies for Better RAG Performance — YouTube · SemanticChunker — LangChain Experimental