See how batch normalization stabilizes activation distributions and speeds up training. Watch two neural networks race, with and without BatchNorm.
Sign in to save your progress
Without normalization, activation distributions shift layer by layer as earlier layers update. The original 2015 paper (Ioffe & Szegedy) called this "internal covariate shift" and proposed BatchNorm as the cure. Why BatchNorm actually helps is still debated: later work (Santurkar et al., 2018) found it speeds up training even when covariate shift is artificially re-injected, and attributes the benefit mainly to a smoother optimization landscape. The shifting distributions below illustrate the original motivation.
Each layer develops a different distribution (layer 4 here has μ=2.50, σ=1.76), forcing subsequent layers to constantly re-adapt.
BatchNorm operates on a mini-batch of activations. Step through each computation to see exactly how normalization is applied.
Training vs. inference: the four steps above use the current mini-batch's own mean and variance. At inference time there may be no batch at all, so BatchNorm instead uses running averages of the mean and variance accumulated during training, while γ and β stay fixed at their learned values. Forgetting to switch modes (e.g. model.eval() in PyTorch) is a classic source of inference bugs.
The stylized curves below illustrate what typically happens when two otherwise identical networks train with and without Batch Normalization. They are constructed for teaching, not recorded from a live run.
Stylized loss curves illustrating the typical effect of BatchNorm; both start at the same loss. The badge above is computed from these plotted curves.
BatchNorm normalizes across the BATCH dimension. For transformers & NLP, Layer Normalization normalizes across the FEATURE dimension instead: it works for single samples and variable-length sequences.
In the original 2015 paper by Ioffe & Szegedy, BatchNorm normalizes the pre-activation values: the order is Linear → BatchNorm → Activation. Some practitioners instead place it after the activation (Linear → Activation → BatchNorm); both orderings appear in production code. Separately, ResNet-v2's "pre-activation" blocks (BN → ReLU → weight layer) showed that normalizing before the weight layer improves gradient flow in very deep residual networks.