An eval set is a measurement instrument built from your own failures. Assemble one task by task, watch the exact confidence interval narrow, then stress it with skewed sampling and noisy labels.
Sign in to save progress
Start from real failures, not invented test cases
The best eval tasks are the ones your system already got wrong in production: they are realistic by construction, and they encode exactly the behavior you care about fixing. Anthropic's evals guidance and Chip Huyen's AI Engineering (chapter 4) agree on the starting move: begin small, around 20 to 50 tasks pulled from real failures and real traffic, and grow the set as you learn where it is blind.
Below is a simulated pool of 400 production tasks for a support assistant, grouped into four slices. Each task carries a hidden true label: whether the current model handles it correctly. Sampling a task into your eval set reveals its label (you run the model and grade the output). The question the rest of this guide answers: how many tasks, sampled how, graded how reliably, before you can trust the number that comes out?
Billing and refunds
Routine policy lookups. The model is strong here.
40% of traffic · 160 tasks in pool
e.g. "Customer asks why they were charged twice for one order (case 1)"
Tone and formatting
Style rewrites and structured output requests.
30% of traffic · 120 tasks in pool
e.g. "Rewrite a blunt outage notice as an apologetic status update (case 1)"
Multi-step reasoning
Chained calculations and eligibility logic.
15% of traffic · 60 tasks in pool
e.g. "Compute the refund owed across two overlapping discount codes (case 1)"
Long-context retrieval
Needle-in-a-haystack lookups over long documents.
15% of traffic · 60 tasks in pool
e.g. "Find the cancellation clause buried in a 40-page contract (case 1)"
Size: watch the error bars narrow
A pass rate measured on n tasks is an estimate, not a fact. The interval below is the exact 95% Clopper-Pearson binomial interval, computed from your set's pass count with the incomplete beta function. Add tasks and watch it tighten; notice how expensive each extra point of precision becomes.
Sample randomly from production
Draws follow the live traffic mix, so every slice shows up in proportion to how often users actually hit it.
Convenience sampling (skewed on purpose)
Pull tasks from one slice only, the way an eval set drifts when you keep adding whatever bug report landed this week.
Tasks in set
0
Model passes
?
Measured pass rate
?
95% CI width
?
Measured pass rate with exact 95% CI (Clopper-Pearson)
Add tasks to measure anything. With zero tasks the pass rate could be anywhere from 0% to 100%.
CI width (points) vs tasks in your set
Precision buys itself slower and slower: halving the CI width costs roughly four times the tasks. The gate needs at least 30 tasks AND a width of 28 points or less.
Stratification: what are you actually measuring?
A narrow interval around the wrong quantity is worse than a wide one around the right quantity. If your set over-represents a slice, the raw average measures your collection habits, not your traffic. Stratification fixes it: estimate each slice separately, then weight by the slice's real production share.
Billing and refunds
0 sampledslice pass rate in your set: no data
Tone and formatting
0 sampledslice pass rate in your set: no data
Multi-step reasoning
0 sampledslice pass rate in your set: no data
Long-context retrieval
0 sampledslice pass rate in your set: no data
Raw average of your set
?
Treats every sampled task equally, whatever slice it came from.
Production-weighted (stratified)
n/a
Needs at least one task per slice. Missing: Billing and refunds, Tone and formatting, Multi-step reasoning, Long-context retrieval.
Simulation truth
65.0%
True production pass rate over all 400 tasks. Hidden in real life; shown here because this is a simulation.
Once every slice has at least one task, the two estimators appear side by side. Then skew your set: add 25 tasks from a single category above and watch them split apart.
Label noise: the ceiling on what you can measure
Every grade in an eval set comes from a judge: a human annotator or an LLM grader, and judges make mistakes. Inject label noise into your set and watch the measured score decouple from true quality. The relationship is exact and computed live, not simulated.
Assemble at least 20 tasks in the sampling lab first, so the noise experiment has a real set to corrupt.
Each grader verdict flips with this probability, independently per task (symmetric noise). Whenever two honest annotators can disagree on a task, your labels carry some of this. Push it to 10% or beyond to complete the experiment.
measured = true * (1 - err) + (1 - true) * err
at err = 0%: ceiling = 100% (no model can measure higher)
a +10 pt true improvement reads as +10.0 ptsTrue pass rate (your set)
?
Expected measured
?
One noise draw
?
Measurement ceiling
100%
The expected value is exact math; each draw is one seeded realization of it.