Drag the split boundaries of a real dataset, fit polynomials on the train block only, and watch what each split is allowed to decide. Then peek at the test set and measure exactly what that shortcut costs you.
Three jobs, three datasets
Every decision in a modeling pipeline consumes data. Fitting parameters consumes the train set. Comparing candidate models consumes the validation set: whichever model wins has, in a small way, fit the validation data too. That is why a third set exists. The test set answers exactly one question, once, at the very end: how well does the chosen model generalize?
| split | fits parameters | chooses the model | final report |
|---|---|---|---|
| Train | yes | no | no |
| Validation | no | yes | no |
| Test | no | no | once, at the end |
Two close relatives are worth knowing. When a single validation split feels too noisy, k-fold cross-validation rotates the validation role so every point gets held out once. And when information from validation or test leaks into training through preprocessing or feature engineering, you get data leakage, the silent version of the peeking you will do on purpose below.
Step 1: carve up the dataset
Below are 60 samples drawn from y = sin(2 pi x) plus Gaussian noise (sd 0.3), shown in already-shuffled order, one cell per sample. Shuffling before splitting matters: if the data were sorted, each block would see a different slice of the world. Drag the two handles to reshape the split and watch every number on this page recompute.
Train n=36 (60.0%)
fits the parameters
Validation n=12 (20.0%)
chooses between models
Test n=12 (20.0%)
one honest read at the end
Drag a handle, or focus it and use the arrow keys (PageUp / PageDown move 5 samples). Minimum sizes: 12 train, 6 validation, 6 test.
Step 2: fit on train, only on train
Each degree is a real least-squares fit computed on the 36 train points and nothing else. Sweep the degree: low degrees underfit everything, high degrees chase the train noise, and the gap between the train bar and the other two bars is the overfit. Try squeezing the train block small with the strip above and then pushing the degree up: with almost as many parameters as points, the fit goes through the noise and the validation error explodes.
RMSE at degree 3
Train RMSE
0.328
what the fit minimizes
Validation RMSE
0.266
what you select on
Test RMSE
0.308
look, do not touch
All three recompute live from the current split and degree. The fit only ever sees the train points.
Step 3: choose a model, then break the rule
The proper workflow: pick the degree with the lowest validation RMSE (marked in gold), then report its test RMSE as your final number. The tempting shortcut: peek at the test errors and pick whatever scores best there. The toggle below lets you do exactly that, and then a fresh sample from the same generator, never used for fitting or choosing, shows what your reported number was hiding. This is selection overfitting: the minimum of many noisy scores is a biased estimate of the winner's true error.