Your RAG Pipeline is Probably Wrong: Catch Hallucinations and Bad Retrieval With RAGAS in 10 Lines

Chris Harper

4 min read

Jul 11, 2026 · 04:05 UTC

Tutorial

RAG

Best Practices

TL;DR: RAGAS scores faithfulness, context precision, answer relevancy, and context recall — four metrics that split your RAG's retrieval failures from its generation failures, so you know exactly which layer to fix.

You built a RAG pipeline, tuned your chunking, and it looks reasonable in demos. But manual spot-checks don't scale, and "it looks good" is how hallucinations ship to production. RAGAS (Retrieval Augmented Generation Assessment) is the standard open-source framework for automated, LLM-judged evaluation of RAG systems. It covers both the retriever and the generator with four independent metrics.

What you'll be able to do after this:

Install RAGAS and score any RAG pipeline output in minutes with a single evaluate() call
Read the four metrics to know whether your failure is bad retrieval (wrong chunks) or bad generation (hallucination)
Build a repeatable CI eval loop to catch regressions when you change chunking, embedding model, or system prompt

The four metrics

Metric	What it catches	Layer
Faithfulness	LLM claims facts not supported by retrieved context	Generator
Answer Relevancy	Answer doesn't address the question	Generator
Context Precision	Relevant chunks are buried behind irrelevant ones	Retriever
Context Recall	Retrieved context is missing key facts	Retriever

A faithfulness drop → your LLM is hallucinating. A context precision drop → your retriever is ranking noise above signal. Different diagnosis, different fix.

Install and run (10 lines)

pip install ragas langchain_openai

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

# Your RAG pipeline produces these four fields for each query
data = {
    "question":     ["What is RAG?", "How does FAISS work?"],
    "answer":       ["RAG retrieves documents and uses them to generate answers.",
                     "FAISS uses ANN search on dense vector embeddings."],
    "contexts":     [["Retrieval Augmented Generation combines a retriever with a generator..."],
                     ["Facebook AI Similarity Search indexes dense vectors for fast ANN lookup..."]],
    "ground_truth": ["RAG retrieves relevant documents and generates answers grounded in them.",
                     "FAISS performs approximate nearest neighbor search on dense embeddings."]
}

dataset = Dataset.from_dict(data)
results = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision, context_recall])
print(results)
# {'faithfulness': 0.97, 'answer_relevancy': 0.85, 'context_precision': 0.62, 'context_recall': 0.78}

Reading the scores

Score	What to do
faithfulness < 0.85	Add a citation requirement to your system prompt; use the Claude Citations API
answer_relevancy < 0.80	Simplify your system prompt; check that the retrieved context isn't swamping the question
context_precision < 0.70	Add reranking (cross-encoder or Cohere Rerank) or switch to hybrid search
context_recall < 0.70	Increase `top_k`; review your chunking strategy; check for missed document types

A CI-ready eval harness

from ragas import evaluate
from ragas.metrics import faithfulness, context_precision
from datasets import Dataset

THRESHOLDS = {"faithfulness": 0.85, "context_precision": 0.70}

def assert_rag_quality(samples: dict):
    """Run as part of CI — raises if any metric falls below threshold."""
    scores = evaluate(Dataset.from_dict(samples),
                      metrics=[faithfulness, context_precision])
    failures = {k: round(v, 3) for k, v in scores.items()
                if v < THRESHOLDS.get(k, 0)}
    if failures:
        raise AssertionError(f"RAG eval failed: {failures}")
    return scores

Run this whenever you change chunking strategy, embedding model, reranker, or system prompt. Treat a faithfulness regression the same as a failing unit test — it means your pipeline is making things up.

Using Claude as the judge

By default RAGAS uses OpenAI as the judge LLM. Switch to Claude with:

from ragas.llms import LangchainLLMWrapper
from langchain_anthropic import ChatAnthropic

judge = LangchainLLMWrapper(ChatAnthropic(model="claude-sonnet-4-6"))
results = evaluate(dataset, metrics=[faithfulness], llm=judge)