Every Answer Backed by Evidence: Inline RAG Citations With LlamaIndex's CitationQueryEngine

TL;DR: Wrap any LlamaIndex index with CitationQueryEngine and every answer arrives with [1], [2] inline citations mapped to the exact source chunk — one line of code, no pipeline rebuild.

What you'll be able to do after this:

• Return numbered inline citations in every RAG answer so users can verify every claim
• Tune citation granularity with citation_chunk_size — 256 chars for sentence-level precision, 1024 for paragraph-level
• Walk response.source_nodes to render clickable "View source" links or build a citations sidebar

RAG pipelines that return unattributed answers are a liability: users can't verify claims, debug hallucinations, or know which document to read next. LlamaIndex's CitationQueryEngine adds citation numbering as a single wrapper around your existing index — the retrieval logic, embeddings, and vector store all stay the same.

How it works

1. Retrieves the top-k nodes from your index as normal
2. Splits each node into sub-chunks and labels them Source 1, Source 2, ...
3. Sends those labeled chunks to the LLM with an instruction to cite by [n] inline
4. Returns the annotated answer plus response.source_nodes for programmatic display

Runnable walkthrough

pip install llama-index llama-index-llms-anthropic llama-index-embeddings-huggingface

import os
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.query_engine import CitationQueryEngine

# 1. Index your documents (swap SimpleDirectoryReader for any loader)
documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)

# 2. Wrap with CitationQueryEngine — one line
query_engine = CitationQueryEngine.from_args(
    index,
    similarity_top_k=5,       # retrieve 5 candidate chunks
    citation_chunk_size=512,   # sub-chunk each into 512-char citation units
)

# 3. Query — response now has inline citations
response = query_engine.query("What are the rate limits for Claude Sonnet 5?")
print(response)
# "The default rate limit is 1,000 requests per minute [1].
#  Tier 4 accounts can request higher limits via the console [2]."

# 4. Access the source nodes for display
for i, node in enumerate(response.source_nodes, 1):
    src = node.node.metadata.get("file_name", "unknown")
    print(f"[{i}] {src} — score {node.score:.3f}")
    print(node.node.text[:200], "\n")

Add source URLs for clickable citations

Store a source_url in your document metadata when you load documents:

from llama_index.core import Document

doc = Document(
    text="The default rate limit is 1,000 requests per minute...",
    metadata={
        "source_url": "https://docs.anthropic.com/en/api/rate-limits",
        "title": "Rate Limits — Anthropic",
    },
)

Now node.node.metadata["source_url"] gives you a direct link to render as [1](url) in Markdown or an <a> tag in HTML.

Stack with a cross-encoder reranker

Retrieve wide, rerank tight, then cite — the answer only references the 3 most relevant chunks:

from llama_index.core.postprocessor import SentenceTransformerRerank

reranker = SentenceTransformerRerank(
    top_n=3,
    model="cross-encoder/ms-marco-MiniLM-L-6-v2",
)
query_engine = CitationQueryEngine.from_args(
    index,
    similarity_top_k=10,
    citation_chunk_size=512,
    node_postprocessors=[reranker],
)

See the cross-encoder reranking post for the full reranker setup.

Sources: Build RAG with in-line citations — LlamaIndex Developer Docs · Citation API Reference — LlamaIndex · Citation-Aware RAG — Tensorlake