Your RAG Pipeline Returns Answers. Here's How to Tell If They're True: Ragas in 10 Lines

TL;DR: Ragas adds four LLM-as-judge metrics to any RAG pipeline — faithfulness, answer relevancy, context precision, and context recall — so you can measure where it hallucinates and which stage to fix.

What you'll be able to do after this: score a RAG pipeline's accuracy, relevance, and retrieval quality programmatically, so you know whether a change helped or hurt before it reaches users.

Three things to take away:

• Faithfulness catches hallucinations. Ragas extracts claims from the answer and checks each against the retrieved chunks. If an answer claims something the chunks don't support, faithfulness drops below 1.0.
• Context precision catches retrieval ordering failures. It scores whether the chunks that actually helped generate the answer were ranked near the top of what was retrieved — poor ranking wastes your context window on irrelevant content.
• LLM-as-judge costs real tokens. Each metric makes several LLM calls per sample. Batch your evaluation set and run it on a random sample of production queries, not a hand-picked "good" set, or the scores tell you nothing.

Install and run a basic evaluation:

pip install ragas openai datasets

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

data = {
    "question": ["What is chunking in RAG?"],
    "answer": ["Chunking splits documents into segments that match the embedding model's token limit."],
    "contexts": [["Document chunking divides text into segments sized to fit the embedding model."]],
    # "ground_truth": ["..."]  # only needed for context_recall
}
dataset = Dataset.from_dict(data)

result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])
print(result)
# {'faithfulness': 1.0, 'answer_relevancy': 0.91, 'context_precision': 0.88}

How to read the scores. Low faithfulness (< 0.8): your model is adding facts the retrieved chunks don't contain — tighten your prompt or fix chunking. Low context precision: useful chunks are buried in retrieval results — check your embedding model or similarity threshold. Low answer relevancy: the answer is drifting from the question — revisit your generation prompt.

Best starting video: RAG Evaluation Matrix — RAGAS Metrics: Faithfulness, Context Precision, Recall walks through all four metrics with code (April 2026).

Real limits. Faithfulness only checks consistency with retrieved context, not world-truth: a pipeline can score 1.0 faithfulness while still being factually wrong if the retrieved chunks themselves contain errors. context_recall requires ground-truth reference answers — without labeled data, skip it. LLM-as-judge scores carry the judge model's biases toward longer, more fluent outputs and toward its own family's style. Ragas is a floor, not a ceiling: 1.0 faithfulness means you passed, not that the answers are correct.

Sources: Ragas documentation — docs.ragas.io · Evaluation of RAG pipelines with Ragas — Langfuse · Evaluate RAG with Ragas + watsonx — IBM