Overfit models buy their perfect training fit with huge weights. Add a penalty on weight size, tune one knob λ, and watch a wild degree-8 polynomial calm down, or thin itself out.
Why overfitting means large weights
A degree-8 polynomial has enough freedom to pass close to every training point. To wiggle between nearby points it needs steep slopes, and steep slopes require coefficients with large magnitudes that nearly cancel each other. That cancellation act is exactly what fails on new data. You saw the symptom in Overfitting vs Underfitting; regularization attacks the cause by charging the model for weight size, trading a little bias for a lot less variance (the Bias-Variance Tradeoff in action).
Unregularized max |w|
9.94
Train MSE at λ = 0
0.0065
Holdout MSE at λ = 0
0.0219
Computed live from the least-squares fit to the 25-point dataset below (13 train, 12 holdout). Weights are standardized coefficients.
The λ playground: degree-8 polynomial regression
Pick a penalty type, then drag λ from 0 to 10 (log scale). Explore the low, mid, and high regions in BOTH modes, and find a λ where the holdout error drops below the λ = 0 baseline. Features are standardized before fitting so the penalty treats every power of x equally; that is the same reason scaling matters in Feature Scaling.
Fitted curve at λ = 0
The fitted curve is clipped to the plot range; at low λ it can swing far outside it between points.
Train MSE
0.0065
Holdout MSE
0.0219
this is the λ = 0 baseline
Zero coefficients
0 / 8
Max |w|
9.94
Coefficients w1..w8
Weights are in standardized feature space (bar scale adapts to the largest weight). Circled 0 marks a coefficient that is exactly zero.
The math running in your browser
objective minimize (1/2n) ||yc - Zw||^2 + (lambda/2) ||w||^2 solution w = (Z^T Z / n + lambda I)^-1 (Z^T yc / n)
Ridge has a closed form: adding λ to the diagonal of Z^T Z / n makes the system well conditioned, then one pass of Gaussian elimination (with partial pivoting) solves the 8x8 system exactly. Every number above comes from that solve.
The U-shape: train vs holdout error across all λ
Both curves below are computed by refitting the L2 Ridge model at every slider position on the real train/holdout split. Training error only gets worse as λ grows, but holdout error dips first: a little regularization removes variance faster than it adds bias.
Holdout MSE at λ = 0
0.0219
Current holdout MSE
0.0219
Best holdout on this curve (L2)
0.0174
Why L1 creates zeros and L2 does not
Penalized regression is equivalent to minimizing the loss inside a constraint region around the origin. The penalty type sets the region's shape, and the shape decides where the loss contours first touch it.
L1 (Lasso): diamond constraint
L2 (Ridge): circular constraint
Schematic illustration: this is the standard geometric picture of penalized regression as constrained optimization, drawn for two weights. Shapes and positions are illustrative, not computed from the dataset above. The ellipses are contours of the squared error loss; the first contour to touch the constraint region gives the regularized solution. A diamond gets touched at its corners (which sit on the axes) far more often than on its flat sides, which is why L1 zeros out weights while L2 only shrinks them.
Early stopping is implicit regularization
Gradient descent usually starts from small random weights and grows them over training. Halting when validation loss stops improving caps how large the weights ever get, so the number of training steps acts like a regularization knob. For linear models with squared loss, stopping gradient descent early has an effect closely related to ridge's shrinkage: directions the data supports weakly are the slowest to grow and get cut off first. You get much of L2's benefit without ever writing a penalty term.
Weight decay in deep networks
Deep learning frameworks expose L2-style regularization as weight decay: every update multiplies weights by a factor slightly below one. For plain SGD, weight decay and the L2 penalty produce identical updates. For adaptive optimizers like Adam they quietly diverge, which is why AdamW decouples the decay from the gradient step. In practice weight decay is combined with other regularizers such as Dropout and early stopping rather than used alone.