The chain rule turns the gradient at layer 1 into a product of one factor per layer. Keep each factor near 1, or watch the training signal die or blow up exponentially with depth.
Sign in to save progress
The mechanism: one multiplication per layer
Backpropagation starts at the loss (the output layer) and applies the chain rule backwards, layer by layer. Every step multiplies the error signal by one more factor, a weight times an activation derivative. By the time it reaches layer 1 on the input side, the gradient is a product of one factor per layer. Products of numbers below 1 shrink exponentially; products of numbers above 1 grow exponentially. If the chain rule feels rusty, walk through the backpropagation guide first.
dL/dw_1 = delta_L * [ w_L * f'(z_(L-1)) ] * ... * [ w_2 * f'(z_1) ] * x delta_L = (a_L - y) * f'(z_L) <- error signal at the loss end each bracket = one layer's backward multiplier
Error signal |delta_L|
0.083
Mean backward multiplier
0.235
Multiplier^(L-1), L = 4
0.013
|dL/dw_1| (layer 1)
7.6e-4
All four numbers are computed live from the network below: change the controls and watch the geometric mean of the multipliers, raised to the number of backward steps, drag the layer 1 gradient up or down with it.
Gradient lab: a real MLP, one unit per layer
This is not a canned animation. Every bar comes from a real forward pass on the fixed input x = 0.7, then a real backward pass from the squared loss against the target y = 0.3. Weights are seeded deterministically in [0.8, 1.2] times your chosen scale, so the same settings always reproduce the same bars.
Each extra layer multiplies one more factor into the gradient product.
Activation function
f'(z) = f(z)(1 - f(z)), peaks at 0.25
Multiplies every seeded weight. Near 1.0 keeps each backward factor close to 1, which is exactly what He and Xavier initialization aim for.
Per-layer gradient magnitude |dL/dw| (log scale)
Healthy: |dL/dw_1| = 7.6e-4, between 1e-6 and 1e3 at layer 1Gradient at layer 1 (input side)
7.6e-4
Gradient at layer 4 (output side, next to the loss)
0.0564
Network output a_L (loss = 0.0737)
0.684
Notice the direction: the gradient is largest near the output, where backprop starts at the loss, and it shrinks (or grows) as the product accumulates backwards toward layer 1 on the input side. Recipes to try: sigmoid at depth 30 vanishes, ReLU at 3.0x weight scale explodes, and ReLU near 1.0x is healthy at any depth here.
Why sigmoid vanishes: the derivative never reaches 1
Sigmoid's derivative peaks at 0.25 and decays toward 0 in both tails. Even in the best case, every sigmoid layer multiplies the backward signal by at most 0.25, so ten layers cost you at least a factor of 0.25^10. Large weights make it worse, not better: they push pre-activations into the saturated tails where the derivative is nearly 0. Tanh peaks at 1 but still shrinks everywhere except exactly at z = 0, while ReLU holds a derivative of exactly 1 across its entire active region. Compare the shapes in the activation functions guide.
The same product problem appears across time steps in recurrent networks, where the sequence length plays the role of depth: see the RNNs and LSTMs guide.
The fixes: keep every factor near 1
Every practical remedy attacks the same root cause. Two of them are wired straight into the lab above: apply either one while the network is vanishing or exploding and watch the status badge turn healthy. Weight scale range here: 0.5x to 3.0x.
Fix 1: Switch to ReLU
ReLU's derivative is exactly 1 everywhere the unit is active, so the activation contributes no shrinkage to the backward product. Rerun the exact same network with ReLU and watch the bars flatten toward a horizontal line.
Fix 2: He / Xavier initialization
The weight scale slider above IS this fix. Xavier and He initialization choose the weight variance (1/n and 2/n for a layer with n inputs) precisely so each backward factor sits near 1. In this one unit per layer network, that corresponds to a scale near 1.0: drag the slider back there and the product stops running away.
Batch normalization
Re-centers and re-scales each layer's activations every step, so pre-activations stay out of the saturated flat zones where derivatives collapse. See the batch normalization guide for the full mechanics.
Residual connections
A skip connection computes a + F(a), so the local derivative is 1 + F'(a). The gradient gets an identity highway: even if the transform branch shrinks it, the skip path carries it backwards unattenuated. This is what makes 100+ layer ResNets trainable.
Gradient clipping
The blunt but effective fix for the exploding side: if the gradient norm exceeds a threshold, rescale it back down before the update.
g ← g · min(1, θ / ‖g‖)