Your Agent Is Failing Silently: Design Rubric-Based Evals Like Anthropic Does

Your agent can pass a unit test and still silently fail every real task — rubric-based evals with an LLM judge are the fix.

What you'll be able to do after this:

• Design a three-dimension rubric that turns vague "did it work?" into a repeatable, measurable signal
• Build a zero-dependency LLM judge in 15 lines that scores agent outputs against explicit criteria
• Wire that judge into pytest so regressions surface before they ship

When an agent can call tools, read files, and run shell commands, comparing its final text to a gold answer does not work. Standard unit tests cover deterministic behavior; agents are not deterministic. A single passing run tells you almost nothing. Anthropic's engineering team faced this building their own product agents and published a detailed breakdown of what evaluation strategies actually work in practice.

The three-question rubric

Every agent eval rubric should answer three questions in order: (1) did the agent avoid breaking anything (side effects, corrupted state, security violations)? (2) did it address what was actually asked (intent alignment)? (3) is the output itself good (quality, format, completeness)? Write each dimension as explicit, independently gradeable criteria — not vague adjectives. "The output CSV has no null values in the price column" is a criterion; "the data looks right" is not. The bar: two domain experts should reach the same verdict on any sample output.

Trajectory vs. outcome evals

Outcome evals check the final artifact — did the file get written, does the test pass? Cheap to run, easy to automate, good regression guards.

Trajectory evals inspect intermediate steps — did the agent call the right tools, did it hallucinate a non-existent API endpoint, did it loop unnecessarily? They catch wrong-path solutions that accidentally produce a passing final state.

Anthropic recommends both: start with outcome evals on 20–50 cases drawn from real failures, then add trajectory checks for the failure modes you care most about.

Build an LLM judge in 15 lines

import anthropic, json

RUBRIC = """Grade this agent output 0-10 on:
1. Task completion — does it fully address the ask?
2. No hallucinations — only references files/APIs that actually exist
3. Code quality — runnable as-is, no placeholder values
Return JSON: {"score": int, "pass": bool, "reason": "..."}
"""

def judge(task: str, output: str) -> dict:
    client = anthropic.Anthropic()
    msg = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=256,
        temperature=0,
        system=RUBRIC,
        messages=[{"role": "user", "content": f"Task:\n{task}\n\nOutput:\n{output}"}],
    )
    return json.loads(msg.content[0].text)

Key bias guards: temperature=0, full task context in the prompt, judge model separate from generating model. For pairwise comparisons, run twice with A/B swapped and average the scores.

Wire it into CI

# test_agent_evals.py
import pytest
from myagent import run_task
from evals import judge

CASES = [
    ("List /tmp files sorted by size", "expected_pattern"),
    # 20-50 real failure cases
]

@pytest.mark.parametrize("task,_", CASES)
def test_agent_quality(task, _):
    output = run_task(task)
    result = judge(task, output)
    assert result["pass"], f"Score {result['score']}: {result['reason']}"

The full Anthropic guide also covers pass@k and pass^k metrics for non-deterministic agents, reading transcripts to catch grade inflation, and watching for eval saturation — when a suite consistently hits 100%, it has stopped measuring improvement and needs harder cases.

Sources: Demystifying evals for AI agents — Anthropic Engineering