Grade a few answers yourself, then run a rubric judge over the same corpus and measure how well it agrees with a human panel. Then inject position, verbosity, and self-preference bias and watch the kappa collapse.
Sign in to save progress
The setup: a judge you can x-ray
Why use a model as judge
Humans grading thousands of model outputs is slow and expensive. So teams hand a rubric to a strong model and let it grade instead: score this answer, or pick the better of these two. It scales beautifully, which is exactly why its failure modes matter.
Why calibrate it first
A judge is a measurement instrument, and instruments drift. Before trusting one, measure its agreement with human labels on a calibration set. Judges carry systematic biases that inflate or deflate scores while looking perfectly confident.
The corpus below has 12 questions, each with two answers of genuinely different quality and a gold label from an expert panel. The judge is Nomix-1, a deterministic rubric scorer standing in for an LLM judge, so every score, agreement rate, and kappa on this page is exactly computed from the data you can see. The other model in the corpus is Quorra-7B, and it matters that some answers were written by Nomix-1 itself: that is what the last bias feeds on.
First, you be the judge
Before trusting an automated judge, do its job yourself. For each question below, pick the answer you would grade higher. Your pick locks in, then you see what an expert panel decided and why. This is the human baseline every judge on this page is measured against.
Question 1 of 3
What does HTTP status code 404 mean?
Question 2 of 3
Is it safe to store passwords with MD5?
Question 3 of 3
What is overfitting in machine learning?
The rubric judge
Now the judge takes over on all 12 questions. Ours is a deterministic rubric scorer standing in for an LLM judge: it reads each answer's rubric features, weights them, and picks the higher score, so every number below is exactly computable. Strictness reallocates weight between facts and polish, the same way a judge prompt saying "prioritize factual accuracy" changes an LLM judge's behavior.
quality(answer) = (wAcc*accuracy/4 + wCov*coverage/3 + wCla*clarity/3) / (wAcc + wCov + wCla) verdict = the higher-scoring answer (an exact tie goes to Answer 1) po (observed) = items where judge matches the panel / total items pe (chance) = pJ1*pH1 + pJ2*pH2 from each rater's marginal pick rates Cohen's kappa = (po - pe) / (1 - pe)
Low strictness rewards polish over facts; high strictness cares almost only about accuracy. Watch which verdicts change at the extremes.
Agreement with panel
11/12
92% raw agreement
Cohen's kappa
0.80
corrects out pe = 0.58 chance agreement
Reading the kappa
Substantial
per the Landis and Koch (1977) convention
Why kappa and not just agreement? Raw agreement can be bought without skill. This panel prefers Answer 1 on 8 of the 12 items, so a broken judge that always picks Answer 1 scores 8/12 raw agreement while reading nothing, and its kappa is exactly 0. Kappa estimates the agreement the two raters would reach by chance from their marginal pick rates (the pe above, which shifts live as the sliders change the judge's behavior) and only credits agreement earned beyond that. Two settings with the same raw agreement can carry different kappas.
All 12 verdicts, live
unbiased| # | question | score A1 | score A2 | judge | panel | match |
|---|---|---|---|---|---|---|
| 1 | HTTP 404 | 0.90 | 0.51 | A1 | A1 | ✓ match |
| 2 | km to miles | 1.00 | 0.59 | A1 | A1 | ✓ match |
| 3 | Moon phases | 1.00 | 0.48 | A1 | A1 | ✓ match |
| 4 | MD5 passwords | 0.67 | 0.90 | A2 | A2 | ✓ match |
| 5 | Capital of Australia | 0.92 | 1.00 | A2 | A2 | ✓ match |
| 6 | Undo git commit | 1.00 | 0.59 | A1 | A1 | ✓ match |
| 7 | Seasons | 1.00 | 0.59 | A1 | A1 | ✓ match |
| 8 | List vs tuple | 1.00 | 0.51 | A1 | A1 | ✓ match |
| 9 | Correlation | 0.90 | 0.48 | A1 | A1 | ✓ match |
| 10 | Overfitting | 0.30 | 1.00 | A2 | A2 | ✓ match |
| 11 | Boiling an egg | 1.00 | 0.61 | A1 | A1 | ✓ match |
| 12 | Slow Python | 0.92 | 0.89 | A1 | A2 | ✗ miss |
11 of 12 verdicts match the panel, kappa 0.80 (substantial). Pink bonus numbers in the score columns are bias points added on top of the rubric score.
Dial the biases, watch the kappa
Real LLM judges carry documented, systematic biases. Each slider injects one into the scoring function as a bonus on the 0 to 1 quality scale, so you can watch exactly how much distortion it takes to corrupt the verdicts above. The kappa comparison below always shows your biased judge against the unbiased judge at the same strictness. Keep one eye on pe: position bias herds the judge toward Answer 1, the side this panel already favors on 8 of 12 items, so chance agreement rises and kappa falls faster than the raw agreement rate does. That gap is the whole argument for reporting kappa instead of agreement.
Adds up to +0.30 to whichever answer is shown first. Real judges favor one position, which is why careful evals grade every pair twice with the order swapped.
Adds up to +0.50 to the longer answer, scaled by how much longer it is. Padding starts beating substance.
Adds up to +0.30 to answers written by Nomix-1, the judge's own model. Also called preference leakage when judge and contestant share training lineage.
Kappa, biased vs unbiased
agreement 11/12 · pe 0.58 · Substantial
agreement 11/12 · pe 0.58 · Substantial
Kappa change: +0.00 · agreement change: +0 · 0 verdicts flipped
How eval teams fight these biases