A model can score 99% accuracy while missing every case you care about. Crank the imbalance and watch which metrics stay honest and which ones flatter you.
Step 1: Set the Imbalance
The dataset always holds about 400 points. The ratio decides how many belong to each class: 1:1 means 1 negative for every positive. Sweep it up to 1:50 or beyond, then bring it back.
The do-nothing baseline
A "model" that predicts negative for every single point scores accuracy = 200 / 400 = 50.0% on the current data, while catching 0 of the 200 positives. Your actual classifier at the current threshold scores 79.0%, so it beats the baseline. But notice how thin the gap gets as imbalance grows.
Step 2: Same Scores, Different Story
Both curves below are computed live from the exact points shown in the strip plot. Sweep the imbalance and compare how each one reacts.
Both class score distributions are fixed Gaussians (positive mean 0.62, negative mean 0.38, both sd 0.15, deterministic stratified samples). The imbalance slider changes only how many points each class contributes: currently 200 positive vs 200 negative.
ROC curve
AUC-ROC 0.871Dashed diagonal: random ranking. Gold dot: current threshold.
Precision-Recall curve
AP 0.871Dashed line: random-classifier precision = prevalence = 0.500. Teal dot: current threshold.
What the numbers say right now
At a balanced 1:1 split these same score distributions give AUC-ROC 0.871 and average precision 0.871. At the current 1:1 split, AUC-ROC is 0.871 (change +0.000) while average precision is 0.871 (change +0.000). ROC axes are per-class rates, so they ignore the class mix. Precision divides true positives by ALL predicted positives, and under heavy imbalance even a small false positive RATE from the huge negative pool produces enough false positive COUNTS to swamp the few true positives.
Step 3: Pick Your Operating Point
A curve is every threshold at once; production needs exactly one. See how the "best" threshold depends on which metric you optimize.
Decision Threshold
Every score at or above the threshold is predicted positive. Move it and watch the confusion matrix and metrics recompute.
Confusion matrix
| Predicted + | Predicted - | |
|---|---|---|
| Actual + | TP 158 | FN 42 |
| Actual - | FP 42 | TN 158 |
200 actual positives, 200 actual negatives. New to these four cells? Start with the confusion matrix guide linked below.
Accuracy
79.0%
(TP + TN) / all
Precision
79.0%
TP / (TP + FP)
Recall
79.0%
TP / (TP + FN)
F1
0.790
harmonic mean of P and R
Accuracy-optimal threshold
t = 0.49 (accuracy 79.0%)
F1-optimal threshold
t = 0.45 (F1 0.796)
The two optima are computed by sweeping 201 candidate thresholds over the current data. Right now they sit 0.05 apart. They sit close together on balanced data, but under imbalance they usually split: accuracy is happiest pushing the threshold up and ignoring the minority class, while F1 has to keep finding it. Try 1:10 and 1:50 and compare the two accuracy values against the do-nothing baseline above.
Step 4: What To Do About It
Four short reads. The third one is required to finish the guide.