Skip connections let each layer learn a small correction instead of a whole new representation, and give gradients an identity highway back to layer one. Measure real backward passes at any depth, then train two networks that differ only in their wiring.
The problem: depth multiplies, and multiplying kills
Backpropagation sends the loss signal through every layer in reverse, multiplying by each layer's local derivative on the way. When most of those factors sit below one, the product shrinks geometrically and the layers near the input stop learning. The vanishing and exploding gradients guide walks through that multiplication in detail. There are two classic answers: pick the multiplier carefully (weight initialization), or change the architecture so the signal never has to survive the full product. Residual connections are the architectural answer.
A residual block computes h + f(h) instead of f(h): the input flows through unchanged on an identity path, and the block only adds a correction. In the backward pass that identity path contributes a clean, unattenuated copy of the gradient, whatever the block's own derivative looks like. Introduced by He, Zhang, Ren and Sun in Deep Residual Learning for Image Recognition (2015, arXiv:1512.03385), this idea took image networks from around 20 trainable layers to over 150 and won the ILSVRC 2015 classification challenge.
Gradient Flow Lab: one real backward pass
A network of 16 blocks (width 12, block f(h) = W₂ tanh(W₁h + b₁) + b₂) is built fresh at every setting, then runs one full forward and backward pass on a fixed 32-point batch. Each bar is the measured norm of the loss gradient reaching that layer, exactly what the first SGD step would use. Both wirings share identical weights; only the skip connections differ. Block weights are Xavier-initialized then scaled by 0.9, slightly inside the stable regime, so each block genuinely attenuates the signal it passes back (the regime the vanishing-gradients guide describes).
Skip connections
Gradient reaching layer 0 (plain)
1.8e-3
vs 0.059 at the top layer
Plain net: top-to-bottom attenuation
3.2e+1x
weaker by the time it reaches layer 0
Skips deliver to layer 0
1.6e+3x
more gradient signal than the plain wiring
With skips the signal even grows toward the input, because every block adds its branch on top of the identity path. Real residual networks pair skips with normalization layers to keep that growth in check; He et al. 2015 used batch normalization inside every block.
Training Race: same weights, only the wiring differs
Two 16-block networks (width 12) start from byte-identical weights and train right here in your browser on the same task: fit y = sin(3x) on 48 points with full-batch gradient descent (learning rate 0.005, 2,000 steps each). The only difference is the skip connections. Seeds are fixed, so running again reproduces the same numbers exactly.
Learn the residual, not the map
If the ideal transformation is close to identity, learning f(h) = 0 is far easier than learning f(h) = h from scratch. Skips give every block that easier target: start from what you have, add a nudge.
Gradient highways
d(h + f(h))/dh = I + df/dh. The identity term guarantees the backward signal reaches every layer intact, no matter how small the block derivative is. Depth stops being a multiplicative death sentence.
Everywhere in modern nets
Every transformer block wraps both its attention and its feed-forward sublayers in residual connections, usually paired with normalization. Without skips, hundred-layer language models would not train.