See Inside Every Agent Step: LLM Observability with Arize Phoenix

Arize Phoenix is a free, open-source observability platform — three lines of code give you a local dashboard showing every LLM call, tool span, and eval score from your agent, with no signup or data sent off-device.

What you'll be able to do after this:

• Start Phoenix locally with one command and see structured traces of every LLM call, tool use, and retrieval step — no account or API key required
• Attach built-in LLM-judge evaluators (hallucination, relevance, faithfulness) to live traces to find exactly where your agent produces bad outputs
• Auto-instrument a Claude Agent SDK session in three lines and watch spans appear live in the Phoenix UI as your agent runs

You can score outputs with promptfoo (already covered) or build assertions into your CI pipeline. Observability is the complementary layer that answers a different question: "what actually happened during that run?" When an agent goes off the rails, tracing shows you the exact token sequence, tool call chain, and retrieval result that led there — without adding debug logs everywhere.

Install:

pip install arize-phoenix arize-phoenix-otel openinference-instrumentation-anthropic

Start Phoenix and instrument your Anthropic client in three lines:

import phoenix as px
from phoenix.otel import register

# Start the local Phoenix server — opens a UI at http://localhost:6006
px.launch_app()

# Auto-instrument all Anthropic API calls in this process
register(project_name="my-agent", auto_instrument=True)

# From here, every call you make appears as a trace
import anthropic
client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Summarize the quarterly financials."}],
)

Open http://localhost:6006 and you'll see one trace: an LLM span with the prompt, response, token counts, and latency. Add a retrieval step and it nests as a child span. Add a tool call and it appears as a sibling. Every agent run builds the trace tree automatically.

Run built-in evals against your traces:

from phoenix.evals import (
    HallucinationEvaluator,
    QACorrectnessEvaluator,
    run_evals,
)
from phoenix.session.evaluation import get_qa_with_reference

# Pull live traces from the running Phoenix session
df = get_qa_with_reference(px.active_session())

# Score them — Phoenix uses an LLM judge internally
results = run_evals(
    dataframe=df,
    evaluators=[HallucinationEvaluator(), QACorrectnessEvaluator()],
    provide_explanation=True,
)

Scores attach to each trace span in the UI. Filter by hallucination == True and you immediately see the exact inputs that caused it — the retrieval context, the user query, and the model response, all in one view.

Three deployment modes:

Mode	Command	When to use
Ephemeral (in-process)	px.launch_app()	Notebooks, quick debug runs
Persistent server	phoenix serve	Local dev — data survives restarts
Self-hosted (Docker)	docker run -p 6006:6006 arizephoenix/phoenix	Team-shared trace store

All three modes expose the same UI and accept the same trace data — switch deployment modes without changing your instrumentation code.

Sources: Arize Phoenix docs · Claude Agent SDK integration · Quickstart: Tracing (Python) · YouTube tutorial: Arize Phoenix Overview