Leakage is information from outside the training fold sneaking into training. The symptom: a validation score that looks fantastic, then a model that collapses in production. Break it three ways below, with every number computed live.
What it is
Any path by which knowledge of the validation or test data, or of the future, or of the label itself, reaches the model during training. Unlike overfitting, which honest validation catches, leakage corrupts the validation score itself. Your safety net is the thing that breaks.
The symptom
Validation scores too good to be true, followed by production collapse. A classic example: fitting a seasonal decomposition on the full series before splitting off the holdout, so the holdout's own values shape the trend it is later scored against. If you work with time series, see the time series guide for how forecasting pipelines are built.
Demo 1 of 3
Two identical 1-nearest-neighbor models on 20 training and 20 validation points. One thing differs: whether the z-scaler is fitted before the split (leaky) or on the training half only (honest). The validation half carries a batch offset on one feature, the kind of distribution shift real deployments hit constantly.
At zero shift both pipelines agree. Increase it and watch the leaky score stay rosy while the honest score collapses.
1-NN validation accuracy (computed live)
The leaky scaler saw the validation rows, so the batch offset between the halves got baked into the standard deviation it divides by. That inflated std shrinks every standardized distance along the shifted feature, muting the offset enough for nearest neighbors to keep matching across the halves. The honest pipeline reveals the truth: this 1-nearest-neighbor model breaks the moment a new batch arrives with a different offset, exactly what production would do to it.
Demo 2 of 3
A logistic regression predicts customer churn from 40 synthetic customers. Toggle in days_until_churn_call, a column that is generated as a noisy copy of the churn label, and watch validation accuracy jump toward perfect.
Learned weight magnitude per feature (standardized inputs)
days_until_churn_call is generated as the label plus this much noise. At zero it is a perfect copy of the answer. Mild-to-moderate corruption still inflates the score; push the slider to the max and the noise drowns the signal, so the inflation disappears.
Validation accuracy (computed live)
Training accuracy 83%. Always predicting the majority class would score 56%.
Training accuracy 100%.
Why does the leaky score never survive production? days_until_churn_call is written by the retention team after a customer announces they are leaving. When you score a live customer, that column does not exist yet. The offline validation number is real math on a feature you will never have at prediction time.
Demo 3 of 3
A trending series of 40 points, and a model that predicts each held-out point with the value at the nearest training timestamp. A random split scatters held-out points between training neighbors, so the model quietly interpolates the future from both sides. A time-based split holds out the end of the series, the only setup that matches how forecasting is used in reality.
With no trend the two splits score closest. The stronger the trend, the more the random split flatters the model.
Mean absolute error, 1-NN on the time index (computed live)
Right now the random split reports an error 4.4 times smaller than the time-based split. Lower error, but the wrong question: production only ever asks about the future, and the time-based number is the one that predicts it.
The fourth flavor: group leakage and duplicates
When several rows belong to the same entity (one patient with many scans, one user with many sessions), a random split puts siblings of the same entity on both sides. The model recognizes the entity instead of learning the pattern, and the validation score inflates just like in the demos above.
Near-duplicates do the same thing: if a row appears in both folds, the model is graded on questions it memorized. The fixes are structural: deduplicate before splitting, and split by group so an entity lives entirely in one fold.
The leakage checklist