Watch SGD, Momentum, RMSProp, and Adam descend the same loss landscape simultaneously. See why adaptive optimizers converge faster on non-uniform loss surfaces, and tune learning rates to explore their trade-offs.
Sign in to save progress
Adam is the default choice for most deep learning today, but it can generalize slightly worse than SGD+Momentum on some tasks. Many practitioners train with Adam then fine-tune with SGD for better final performance.
Change a learning rate and the trajectories recompute instantly. Watch how each optimizer responds differently.
The landscape is L(x,y) = 0.1x² + 2y², an elongated bowl. At the default learning rate (0.1), SGD settles the steep y-direction fast (y shrinks 40% per step) but crawls along the shallow x-axis (x shrinks only 2% per step): that slow x-progress is what loses the race. Push SGD's learning rate above 0.25 and the steep y-axis starts to oscillate; above 0.5 it diverges. Adaptive optimizers rescale each axis, so they make even progress in both directions.