A chatbot is graded on its answer. An agent takes actions, so you grade the whole trajectory. Replay three example runs of the same agent, score them with code checks and a rubric judge, then see why one lucky pass says nothing about reliability.
Two ways to grade an agent
Outcome evals (code-based)
Check the final state with code: does the answer match ground truth, was the required tool called, did the run stay in budget. Objective, fast, and cheap to run thousands of times, but blind to how the agent got there.
Process evals (rubric judge)
Grade the trajectory itself against a rubric: did it fetch the policy instead of assuming it, is every figure grounded in an observation, did it avoid redundant calls. Catches lucky passes and pinpoints where a run broke.
Anthropic's 2026 guide "Demystifying evals for AI agents" boils this down to habits you will use on this page: read full transcripts before trusting any metric, turn real failures into eval tasks, and grade with code where you can, with an LLM judge where you must.
The transcript lab: three runs, one task
The three transcripts below are authored examples, written for this guide so you can watch the two graders disagree: the same agent attempts the same refund question three times. Step through each run, then watch the graders score it. Every verdict below is computed by running the graders on the transcript you just read, and the ground-truth refund is recomputed from the order data.
Looks everything up, verifies, answers correctly.
TASK: Customer Chen asks: how much refund am I owed for my returned items this month? TOOLS: search_orders(customer) get_refund_policy() calculator(expression)
Transcript replay
step 1 of 9Chen wants this month's refund total. First I need the returned orders.
Scoreboard
| transcript | outcome grader | rubric judge | agree? |
|---|---|---|---|
| A: The careful run | step through this run to grade it | ||
| B: The lucky guesser | step through this run to grade it | ||
| C: The typo | step through this run to grade it | ||
One pass is not reliability: pass@k vs pass^k
Transcript B passed once by luck. Now imagine running that agent k times on the same task. pass@k measures whether it succeeds at least once (can it do the task at all), pass^k measures whether it succeeds every single time (does it do the task reliably). Drag p and k and watch the two curves pull apart as k grows: the pass@k curve climbs toward 100% while pass^k decays toward zero.
How often a single run of the agent succeeds on this task.
pass@k asks: did at least one of k runs succeed? pass^k asks: did all k?
pass@k = 1 - (1 - p)^k "can it do the task at least once in k tries" pass^k = p^k "does it do the task in every one of k tries"
Simulated points come from 200 task sets of 12 attempts each, drawn with a seeded PRNG (reseeded from p), so they are reproducible. Lines are the exact formulas.
pass@1 (at least one success)
95.0%
simulated: 94.0%
pass^1 (all 1 succeed)
95.0%
simulated: 94.0%
reliability gap
0.0%
pass@1 minus pass^1
Mini challenge: prove the demo-to-production gap
Find a setting where pass@k is at least 90% (a demo will almost surely work at least once) while pass^k is at most 50% (whether all k runs succeed is worse than a coin flip). That is the same agent, measured two ways.
Current setting: pass@1 = 95.0%, pass^1 = 95.0%. Not there yet, try raising k or lowering p.