A leaderboard number is not a capability measurement. Leak test items into a simulated model's training set, watch its public score inflate while a private held-out set stays put, and learn the three ways a benchmark lies.
What a benchmark score actually is
A fraction: items answered correctly over items asked, on one dataset, at one moment, under someone's reporting choices. Everything on this page is simulated so the ground truth is visible: we define each model's true skill, generate item by item outcomes with a seeded random draw, and compute every displayed number from those outcomes. Three effects can pull the measured number away from the true skill.
Saturation
When items are too easy for the current model generation, every model scores near the ceiling. The remaining differences are noise, not capability.
Contamination
Test items that leak into training data get memorized. The model repeats answers it has seen, and the score measures recall of the test set, not skill.
Selection effects
Best run of many, friendliest benchmark of many, most flattering prompt of many. Every selection step adds points that vanish on a neutral re-test.
The contamination lab
This model is fully synthetic: you set its true skill, and the simulation generates every item outcome with a seeded random draw. The contamination slider leaks a share of the 120 public test items into its training set. Leaked items become memorized: the model repeats the answer it saw, so they score correct regardless of skill. The 120 private held-out items can never leak.
The ground truth. Both scores should track this number.
0 of 120 public items are now in the training set.
At 0% contamination (0 of 120 items leaked): public score 78.3%, private score 74.2%, gap 4.1 points.
True skill
68%
Public score
78.3%
Private score
74.2%
Public minus private
+4.1 pts
Even at zero leakage the two sets disagree by a couple of points. That is sampling noise on 120 items, and it is the floor below which small leaderboard differences mean nothing.
What the literature reports
Published surveys of data contamination in LLM benchmarks report that leaked test data can inflate measured scores by roughly 5 to 15 percentage points (arXiv:2502.17521, arXiv:2404.00699). That range is the survey finding, quoted for context. The gap you create above is computed live from this page's simulation, not taken from those papers.
Audit the leaderboard: public vs held-out
Four synthetic models, ranked by their public leaderboard score. All four are simulated: we defined each model's true skill and how much of the public test set leaked into its training data, then computed both scores from the item outcomes. The leaderboard only shows the public number. Audit a model to run it on the private held-out set.
Simulated public leaderboard
0 of 4 audited| rank | model | public score | private score | gap | leaked share | audit |
|---|---|---|---|---|---|---|
| #1 | Petrel-Xsimulated | 94.2% | ? | ? | ? | |
| #2 | Halcyon-3simulated | 83.3% | ? | ? | ? | |
| #3 | Corvid-1simulated | 81.7% | ? | ? | ? | |
| #4 | Lyra-Minisimulated | 77.5% | ? | ? | ? |
Saturation: when a benchmark stops measuring
Same four models, zero contamination, two different item mixes. The models span 16 points of true skill. Whether the benchmark can show that depends entirely on how hard its items are.
Item difficulties sit far below every model's skill. Everyone answers almost everything: the benchmark is saturated.
Clean scores on the legacy mix: top minus bottom is 5.8 points.
Spread on this mix
5.8 pts
across 16 points of true skill difference
On a saturated benchmark the whole field piles up near the ceiling, and the tiny differences that remain are the same size as sampling noise. A model that tops a saturated leaderboard has proven very little.
Selection effects: free points from noise
Below are eight clean evaluations of one simulated model with identical true skill (70%), differing only in the random draw. Report the best run instead of the mean and you gain points without the model changing at all. The same selection logic applies to picking the friendliest benchmark, the best prompt template, or the best checkpoint.
Honest number (mean of 8)
72.7%
Press-release number (best of 8)
75.8%
Free points from selection
+3.1 pts