Measure Before You Improve: Evaluate Any RAG Pipeline with RAGAS

Chris Harper

3 min read

Aug 8, 2026 · 04:07 UTC

Tutorial

RAG

Best Practices

ragas gives you four numbers that tell you exactly where your RAG pipeline breaks — before you touch the code, run a benchmark, or guess at what to fix.

What you'll be able to do after this:

Run automated RAG evaluation with one function call and get per-metric scores for every query in your test set
Distinguish retrieval failures (low context precision or recall) from generation failures (low faithfulness or answer relevancy) and know which layer to fix
Drop evaluation into CI so regressions surface before deployment

Install

pip install ragas langchain_openai

Set OPENAI_API_KEY — RAGAS uses an LLM as judge. Any OpenAI-compatible endpoint works.

Build an evaluation dataset

You need three things per example: the user question, the retrieved context chunks (as a list of strings), and the model's answer. Start with 10–30 representative queries from your actual use case.

from ragas import EvaluationDataset, SingleTurnSample

samples = [
    SingleTurnSample(
        user_input="What is the return policy?",
        retrieved_contexts=[
            "Items can be returned within 30 days of purchase with a receipt.",
            "Sale items are final and cannot be returned.",
        ],
        response="You can return items within 30 days with a receipt. Sale items are non-refundable.",
    ),
    # ... add more samples
]

dataset = EvaluationDataset(samples=samples)

Run the evaluation

from ragas import evaluate
from ragas.metrics import Faithfulness, AnswerRelevancy, ContextPrecision, ContextRecall
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

results = evaluate(
    dataset=dataset,
    metrics=[Faithfulness(), AnswerRelevancy(), ContextPrecision(), ContextRecall()],
    llm=evaluator_llm,
)

print(results.to_pandas()[["faithfulness","answer_relevancy","context_precision","context_recall"]])

Reading the scores

Metric	What it measures	Low score → fix
Faithfulness	Answer grounded in retrieved context?	Prompt: add citation enforcement; reduce hallucination
Context Precision	Right chunks ranked first?	Reranker, better embeddings, metadata filters
Context Recall	Did retrieval find all relevant chunks?	Increase `k`, better chunking, hybrid search
Answer Relevancy	Does the answer address the question?	Prompt engineering, filter off-topic retrieval

A score below ~0.7 on any metric usually points to a specific, fixable issue. Fix precision before recall — surfacing the right chunk first is cheaper than retrieving more and hoping a reranker saves you.

Add a CI gate

# In a CI script or conftest.py
results = evaluate(dataset, metrics=[Faithfulness(), ContextPrecision()], llm=evaluator_llm)
df = results.to_pandas()
assert df["faithfulness"].mean() > 0.75, f"Faithfulness regression: {df['faithfulness'].mean():.2f}"
assert df["context_precision"].mean() > 0.70, f"Precision regression: {df['context_precision'].mean():.2f}"

RAGAS also accepts Hugging Face datasets as input, so you can version your eval set alongside your code and track scores across commits.