NeuroNomixer
  • Home
  • Blog
  • Visual Guides
  • Authors
  • Contact
Sign InSign Up
HomeBlogAuthorsContactPrivacy Policy

© 2026 NeuroNomixer — Built with Next.js & Tailwind CSS

Visual Guides/Model Calibration
Machine Learning

Model Calibration

A calibrated model's confidence matches reality: among its 80 percent predictions, about 80 percent should come true. Break a model's calibration with one slider, then repair it with temperature scaling.

Create a miscalibrated model
Fix it with temperature
Read when it matters

Sign in to save progress

Confidence is not accuracy

Accuracy asks how often the model is right. Calibration asks whether the model knows how often it is right. A weather model that says "70 percent chance of rain" is calibrated if it rains on about 70 percent of the days it says that, regardless of how sharp its forecasts are. The probabilities here come from a sigmoid over logits, and the distortion you will create bends that sigmoid without touching the underlying ranking.

How this playground stays honest: 1,000 (logit, label) pairs are generated once with a seeded generator (seed 31), and each label is drawn from the known true probability sigmoid(logit). The model's reported score is sigmoid(logit / trueT). Because the label process is known, every chart and every number below is computed live from these 1,000 predictions. Nothing is faked.

Reliability Diagram

raw model scores · 10 bins · n = 1,000
0.000.000.250.250.500.500.750.751.001.00Bin 0.0 to 0.1: 125 predictions, mean confidence 5.3%, observed accuracy 5.6%Bin 0.1 to 0.2: 112 predictions, mean confidence 14.2%, observed accuracy 13.4%Bin 0.2 to 0.3: 106 predictions, mean confidence 24.8%, observed accuracy 23.6%Bin 0.3 to 0.4: 92 predictions, mean confidence 35.1%, observed accuracy 35.9%Bin 0.4 to 0.5: 91 predictions, mean confidence 44.5%, observed accuracy 42.9%Bin 0.5 to 0.6: 74 predictions, mean confidence 55.3%, observed accuracy 55.4%Bin 0.6 to 0.7: 92 predictions, mean confidence 64.9%, observed accuracy 65.2%Bin 0.7 to 0.8: 101 predictions, mean confidence 74.9%, observed accuracy 69.3%Bin 0.8 to 0.9: 95 predictions, mean confidence 85.2%, observed accuracy 90.5%Bin 0.9 to 1.0: 112 predictions, mean confidence 94.9%, observed accuracy 95.5%Mean predicted probability (confidence)Observed frequency (accuracy)
Observed accuracy per binCalibration gap (|accuracy - confidence|)Perfect calibration (y = x)

Confidence Histogram

raw model scores
Bin 0.0 to 0.1: 125 predictions1250.0Bin 0.1 to 0.2: 112 predictions1120.1Bin 0.2 to 0.3: 106 predictions1060.2Bin 0.3 to 0.4: 92 predictions920.3Bin 0.4 to 0.5: 91 predictions910.4Bin 0.5 to 0.6: 74 predictions740.5Bin 0.6 to 0.7: 92 predictions920.6Bin 0.7 to 0.8: 101 predictions1010.7Bin 0.8 to 0.9: 95 predictions950.8Bin 0.9 to 1.0: 112 predictions1120.9Predicted probabilityCount

Your Mission

  1. Drag trueT away from 1.00 until raw ECE reaches at least 0.080
  2. Drag T until the scaled ECE drops to 0.031 or below
  3. Open the "when calibration matters" panel below
1.00
0.30 = overconfident1.00 = calibrated3.00 = underconfident

The model reports sigmoid(logit / trueT) while labels follow sigmoid(logit). Below 1 the logits are inflated, so the model claims more certainty than it has.

1.00
0.30 = sharpen1.00 = no change3.50 = soften

Temperature scaling divides the model's logits by T before the sigmoid. Find the T that flattens the reliability gap and minimizes ECE.

Charts show

Raw model ECE

0.016

goal: reach 0.080

ECE after your T

0.016

target: 0.031 or below

Noise floor

0.016

ECE of perfectly calibrated scores on this finite sample; target = floor + 0.015

Best T found

?

move T while the model is miscalibrated

Why the best T recovers the known distortion

labels follow     P(y = 1 | z) = sigmoid(z)
model reports     p = sigmoid(z / trueT)          <- the distortion you dialed in
your fix          q = sigmoid((z / trueT) / T)    <- temperature scaling
exact inversion   trueT * T = 1  =>  q = sigmoid(z)  (perfectly calibrated)

Temperature scaling exactly inverts this family of miscalibration. Dividing the logits by trueT and then by T composes into a single scale factor trueT times T. When that product hits 1, the distortion cancels and the reported probability equals the true label rate. That is why the ECE-minimizing T sits at 1 / trueT (up to the sampling noise of 1,000 labels), so 1 / T recovers the distortion you dialed in.

Your T × trueT

1.00

target: 1.00

trueT (the knob)

1.00

known distortion

1 / (best T)

?

recovered distortion

Recovery check

?

1 / (best T) vs trueT, within 10 percent

Going Deeper

← All GuidesNext Guide →