Where weights start determines whether signal and gradients survive depth. Initialize a real 6-layer network five different ways, watch its activations and gradients collapse or explode layer by layer, then race all five schemes in a live training run.
The first answer to the vanishing gradient problem
The vanishing and exploding gradients guide shows the disease: stack enough layers and the learning signal either fades to nothing or blows up on its way back. This guide is the first cure, and historically it came first too: pick the starting scale of the weights so that each layer preserves the variance of what flows through it. The second cure, changing the architecture so gradients get a shortcut around the layers, is covered in residual connections.
The network below is real: 6 hidden layers of 16 units plus a linear readout, classifying 192 two-dimensional points arranged in two rings. Every histogram and every loss curve on this page comes from actual forward passes, backward passes, and gradient updates running in your browser.
Step 1: watch signal and gradient flow at initialization
Pick an init scheme and an activation. The network (2 input features, 6 hidden layers of 16 units) runs one real forward pass on the full batch, then one real backward pass, and histograms every activation and every gradient per layer. Compare tiny random against Xavier, then switch the activation to ReLU and see why He initialization exists.
Init scheme
Hidden activation
Tiny random
W ~ N(0, 0.01²)
For this network that means weight std 0.010 on the first layer (2 to 16) and 0.010 on the 16 to 16 hidden layers. Biases start at 0 (the standard convention).
Activations per layer (forward pass)
Histogram of all 3072 post-activation values per layer, shared x-range [-1.000, 1.000]. Watch the distribution walk layer by layer.
Layer 1
std 6.0e-3
0% saturated
Layer 2
std 2.4e-4
0% saturated
Layer 3
std 8.0e-6
0% saturated
Layer 4
std 3.6e-7
0% saturated
Layer 5
std 1.4e-8
0% saturated
Layer 6
std 5.9e-10
0% saturated
Gradients per layer (backward pass)
Histogram of dL/d(pre-activation) from a real backward pass of the per-example loss, shared symmetric x-range [-0.013, 0.013]. Backprop flows right to left: layer 6 first, layer 1 last.
Layer 1
std 1.2e-9
Layer 2
std 2.2e-8
Layer 3
std 4.7e-7
Layer 4
std 1.2e-5
Layer 5
std 3.0e-4
Layer 6
std 5.8e-3
Gradient std leaves layer 6 at 5.8e-3 and arrives at layer 1 at 1.2e-9: a backward amplification factor of 2.2e-7 across 6 layers. That is a vanishing gradient: the early layers barely learn.
Saturated = |activation| above 0.95; there tanh is nearly flat, so its derivative kills the gradient.
Step 2: race the five schemes on real training
Statistics at initialization are a prediction; training is the verdict. This races five identical networks under the currently selected activation (tanh), differing only in their starting weights. Try it under both activations: the ranking changes, and that change is the whole point.
Five copies of the same 6-layer, 16-unit tanh network, identical except for how the weights start, each train on the same 192 ring points with full-batch gradient descent (learning rate 0.25, momentum 0.9, 240 steps). Every loss value is computed live from a real forward and backward pass; nothing is scripted. If a run overflows to a non-finite loss it is marked diverged and stops, which is itself a real outcome of bad initialization.
Symmetry breaking
With all-zero (or all-equal) weights, every unit in a layer computes the same output and receives the same gradient, so they can never differentiate. Random initialization is not optional noise; it is what makes units distinct.
Variance bookkeeping
Xavier (Glorot & Bengio 2010) sets Var(W) = 2/(fan_in + fan_out) to keep both activations and gradients at a stable scale under tanh-like units. He et al. 2015 doubled the fan-in term to 2/fan_in because ReLU zeroes half its inputs, halving the variance at every layer.
When init is not enough
Good initialization only controls the starting point. At extreme depth, training needs structural help too: residual connections give gradients a shortcut path, and batch normalization re-standardizes activations at every layer, every step.