Residual connections and gradient flow
In 2015 the ResNet authors reported something odd. A plain 56-layer network had higher training error than a 20-layer one on CIFAR-10. This is not overfitting: the deeper model had strictly more capacity, and could in principle copy the shallow one and use identity maps for the extra layers. It simply did not find that solution. Optimization, not representation, was the bottleneck.
The fix is a change of variables
Instead of asking a block to learn the desired map $H(x)$, give it a shortcut and ask it to learn the difference:
$$y = x + F(x), \qquad F(x) = H(x) - x$$
If the best thing a block can do is nothing, it only has to push $F$ toward zero, which is far easier than assembling an identity out of nonlinear layers. Skip connections add no parameters and almost no compute.
What it does to gradients
Differentiate $y = x + F(x)$:
$$\frac{\partial y}{\partial x} = I + \frac{\partial F}{\partial x}$$
Chain this over blocks $l, \dots, L$. If you unroll the additions, $x_L = x_l + \sum_{i=l}^{L-1} F_i(x_i)$, and so
$$\frac{\partial L}{\partial x_l} = \frac{\partial L}{\partial x_L}\left(I + \frac{\partial}{\partial x_l}\sum_{i=l}^{L-1} F_i(x_i)\right)$$
The identity term means the gradient at layer $L$ reaches layer $l$ unchanged, through the skip path alone, no matter how deep the stack is. A plain network instead has a product of Jacobians $\prod_i J_i$. If each has norm $0.9$, fifty layers give $0.9^{50} \approx 0.005$. If each has norm $1.1$, they give $117$. A product of fifty numbers is stable only if every one of them is almost exactly 1.
Numbers
Here is a fifty-layer ReLU network, width 64, with the first-layer gradient norm measured for a plain and a residual version. The branch weights are scaled by a factor $s$ relative to He initialization.
import numpy as np
rng = np.random.default_rng(0)
def first_layer_grad_norm(depth, residual, s, width=64):
ws = [rng.standard_normal((width, width)) * np.sqrt(2 / width) * s for _ in range(depth)]
h = rng.standard_normal((1, width))
saved = []
for w in ws:
saved.append(h)
z = np.maximum(h @ w, 0.0)
h = h + z if residual else z
g = np.ones_like(h)
for w, a in zip(reversed(ws), reversed(saved)):
branch = (g * ((a @ w) > 0)) @ w.T
g = g + branch if residual else branch
return np.linalg.norm(g)
for s in (0.1, 0.3, 1.0):
print(s, first_layer_grad_norm(50, False, s), first_layer_grad_norm(50, True, s))
One run printed this (the input gradient is a vector of norm 8 at the output, since the output has 64 entries of 1):
| Branch scale $s$ | Plain, first-layer gradient norm | Residual, first-layer gradient norm |
|---|---|---|
| 0.1 | $2 \times 10^{-49}$ | 94 |
| 0.3 | $6 \times 10^{-25}$ | $3 \times 10^{4}$ |
| 1.0 | 16 | $3 \times 10^{12}$ |
Two lessons. The plain network is a bet that every layer has gain almost exactly 1: it vanishes like $s^{50}$ the moment the scale is off, and only the perfectly calibrated $s = 1$ row survives. The residual network never vanishes, because the identity path always delivers at least the gradient it was given. But with no normalization and full-size branches it explodes, since every block adds a second moment of its own. That is why real ResNets pair skips with normalization and small or zero-initialized branches.
Practical details
- Zero-init the last norm layer of each branch. Setting the final batch norm $\gamma$ to 0 makes every block start as exactly the identity, so a 100-layer network starts as a shallow one.
- Shape changes need a projection. When channels or resolution change, the shortcut uses a $1 \times 1$ convolution so the sum is well defined.
- Pre-activation vs post-activation. The original block applies ReLU after the addition. The follow-up paper moves normalization and activation inside the branch, which leaves the shortcut a pure identity and trains deeper nets more cleanly.
- The same trick appears everywhere. A transformer block is $x + \mathrm{Attn}(\mathrm{LN}(x))$ followed by $x + \mathrm{MLP}(\mathrm{LN}(x))$. The LSTM cell state in the next note is an additive path for the same reason.
Common mistakes
- Adding tensors of different shapes. Broadcasting can hide a missing projection and give a model that runs and is wrong.
- Putting ReLU on the skip. It clips the identity path and removes the whole point.
- Explaining skips only as "fixing vanishing gradients". They also change the loss landscape, which is smoother with skips than without in the visualizations of Li et al.
Check yourself
- Why can a 56-layer plain network have higher training error than a 20-layer one, when it could copy the shallow net?
- Derive $\partial y / \partial x$ for $y = x + F(x)$ and say what the identity term guarantees.
- What does zero-initializing the last norm layer in each branch do at step 0?
- Which operation in a transformer block plays the role of the skip connection?