The step size is a curve over time, not a constant, and its shape decides convergence. Train a real 129-parameter net in your browser under four schedules, push the peak until training explodes, then watch warmup walk the same peak back from the cliff.
One knob, but it is a knob you turn over time
Every SGD step moves the weights by learning rate times gradient. Treating that rate as a single constant forces one compromise on the whole run. A schedule lets the rate be large while the model is far from a solution and small while it settles, and the same peak value can be fatal or fine depending purely on the curve around it. You will produce both outcomes below, from real runs.
Too low: the budget runs out
Small steps are safe but slow. With a fixed budget of 1200 steps, a tiny rate leaves the loss stranded far above what the same net reaches with a bolder curve. Try peak 0.02 below and compare.
Too high: the loss explodes
Past a stability threshold each step overshoots the minimum by more than the last, and the loss grows exponentially instead of shrinking. This is divergence, and you can trigger it on demand with the cliff preset.
The shape is the fix
Warmup keeps early steps small while the weights are in their most fragile state; decay shrinks late steps so minibatch noise stops rattling the loss. Same peak, different curve, different fate.
One dimension at a time: this lab trains with plain SGD on purpose. Momentum, RMSProp, and Adam change the update rule, and that dimension has its own guide, the Optimizers Race. Here the only thing that differs between runs is lr(t).
Step 1: pick a curve, then train for real
The net is a 2 → 32 (tanh) → 1 MLP with 129 parameters, trained by real minibatch SGD (batch 16) to regress the 16x16-point target surface shown at the bottom of the page. Every run starts from the exact same weights (seed 20260710) and sees the exact same batch order, so curves are directly comparable and re-running a config reproduces it to the last digit. The dashed gold line previews the schedule you selected; solid lines are finished runs.
Step 2: break it, then rescue it
At peak 0.4 this net sits past its stability threshold: a constant schedule feeds the full rate to the freshly initialized weights and the loss blows up within a handful of steps. The rescue run uses the identical peak but climbs to it linearly over 240 steps and glides back down with cosine, spending only a brief window near the danger zone. Same peak, same net, same data, opposite outcome. Both are real runs; check the table above afterwards.
What the net actually learned
Proof the training is real: the right panel is the net's own output surface after your most recent finished run, recomputed from its final weights. A converged run reproduces the hill and the pit; a diverged run leaves saturated garbage.
Target surface (what the net must learn)
f(x1, x2) = exp(-4 * ((x1 - 0.4)^2 + (x2 - 0.4)^2)) - exp(-4 * ((x1 + 0.4)^2 + (x2 + 0.4)^2))
Finish a training run to see what the net learned
Learned surface: no finished run yet
Teal = positive output, pink = negative, dark = near zero