Before the Splitter and the Vector Store, There's the Loader: LangChain Document Loaders for Any Source

TL;DR: Three LangChain loaders — PDF, web, and directory — cover most RAG ingestion cases in under 15 lines each, and they all hand you the same Document(page_content, metadata) interface your chunker already expects.

What you'll be able to do after this: load PDFs, web pages, and entire folders into a LangChain RAG pipeline using the right loader for each source, and know which limits to plan around before they bite you in production.

Three things to know before you start:

• Every loader returns the same Document(page_content=…, metadata={…}) type — swap loaders without touching your splitter or embedder
• .lazy_load() streams documents one at a time; use it for large corpora where .load() would hold everything in memory at once
• Metadata (metadata["source"], metadata["page"]) is added automatically — these fields are what SelfQueryRetriever and RecordManager use for filtering and deduplication later

Install:

pip install langchain-community pypdf

The three loaders you'll use most:

# PDF: one Document per page
from langchain_community.document_loaders import PyPDFLoader
docs = PyPDFLoader("technical-spec.pdf").load()

# Web page: raw text from a public URL (uses BeautifulSoup under the hood)
from langchain_community.document_loaders import WebBaseLoader
docs = WebBaseLoader("https://docs.example.com/guide").load()

# Whole directory of Markdown or text files
from langchain_community.document_loaders import DirectoryLoader, TextLoader
docs = DirectoryLoader("./knowledge-base", glob="**/*.md", loader_cls=TextLoader).load()

All three outputs plug straight into RecursiveCharacterTextSplitter.split_documents(docs) — the chunker from a previous post.

The limits that catch people:

• PyPDFLoader loses table structure. Scanned (image-only) PDFs come back as empty or garbled — use UnstructuredPDFLoader for layout-sensitive docs, though it requires extra dependencies and is slower.
• WebBaseLoader runs BeautifulSoup on raw HTML. JavaScript-rendered pages (React/Vue apps, SPAs) return navigation-only content — swap in AsyncChromiumLoader when you need JS execution.
• DirectoryLoader processes files sequentially. For corpora with thousands of files, call .alazy_load() with asyncio to load in parallel without blocking.

For sources beyond these three — Notion, Google Drive, Slack, GitHub — LangChain ships integrations at python.langchain.com/docs/integrations/document_loaders.

Sources: Document loaders how-to — LangChain Python docs · LangChain Document Loaders: A Practical Guide (2026) — fast.io · Loading PDFs, web pages, and directories with code examples — Latenode