Why initialization matters: the variance math
At initialization a network is a random function. Training can only start from that function, so the first question is whether signals survive the trip through the layers. If activations explode or shrink to nothing, gradients do the same on the way back, and the first hundred steps are spent repairing a bad starting point instead of learning.
The variance argument
Take one linear unit $y = \sum_{i=1}^{n} w_i x_i$ with independent, zero-mean weights and inputs. Each product has variance $\mathrm{Var}(w),\mathbb{E}[x^2]$ and there are $n$ of them:
$$\mathrm{Var}(y) = n,\mathrm{Var}(w),\mathbb{E}[x^2]$$
Stack $L$ layers and the variance is multiplied by $n,\mathrm{Var}(w)$ at every layer. The factor to the power $L$ decides everything:
- Above 1: activations grow exponentially with depth.
- Below 1: they shrink exponentially.
- Exactly 1: the signal keeps its scale.
So the fix is to choose $\mathrm{Var}(w) = 1/n$ with $n$ the layer's fan-in. That is LeCun initialization. Glorot (Xavier) uses $2/(n_{in} + n_{out})$ to balance the forward and backward passes.
ReLU changes the constant
ReLU zeroes half of a symmetric distribution, so $\mathbb{E}[\mathrm{relu}(y)^2] = \tfrac12 \mathrm{Var}(y)$. Each layer now multiplies the second moment by $n,\mathrm{Var}(w)/2$. To hold it at 1:
$$\mathrm{Var}(w) = \frac{2}{n}$$
That is He (Kaiming) initialization. For tanh the standard gain is $5/3$, which compensates for tanh squashing its input; PyTorch exposes it as calculate_gain("tanh").
Worked example: ten layers, width 512
Width $n = 512$, ReLU, ten layers, inputs from a standard normal. The per-layer factor is $512 \cdot \mathrm{Var}(w)/2$ on the second moment, which is $\sqrt{512/2} = 16$ on the standard deviation when $\mathrm{std}(w) = 1$.
| Weight std | Per-layer factor on the std | Predicted after 10 layers | Measured |
|---|---|---|---|
| 1.0 | 16 | $16^{10} \approx 1 \times 10^{12}$ | about $9 \times 10^{11}$ |
| 0.02 | 0.32 | $0.32^{10} \approx 1 \times 10^{-5}$ | about $9 \times 10^{-6}$ |
| $1/\sqrt{n}$ | 0.71 | $0.71^{10} \approx 0.03$ | about 0.02 |
| $\sqrt{2/n}$ | 1.0 | about 1 | about 0.8 |
The 0.02 row is the classic "small random numbers" init from older tutorials. At width 512 it is already wrong by five orders of magnitude after ten layers. The measured values come from the script below with one seed; yours will differ in the second digit but not in the order of magnitude.
import numpy as np
rng = np.random.default_rng(0)
def forward_std(scale, layers=10, width=512, batch=256):
x = rng.standard_normal((batch, width))
for _ in range(layers):
w = rng.standard_normal((width, width)) * scale
x = np.maximum(x @ w, 0.0)
return x.std()
n = 512
for name, s in [("std 1.0", 1.0), ("std 0.02", 0.02), ("1/sqrt(n)", n ** -0.5), ("sqrt(2/n)", (2 / n) ** 0.5)]:
print(name, forward_std(s))
The loss at init is a free test
If the output layer is calibrated, a $C$-class classifier starts out predicting roughly uniform probabilities, so the cross-entropy at step 0 should be about $\ln C$: $2.303$ for 10 classes and $3.296$ for 27. If you see 25 or 100, the logits are huge and the network is confidently wrong. The usual repair is to scale the last layer's weights by something like 0.01 and zero its bias. Check this number before every training run. It costs one line and catches a surprising number of bugs.
Normalization changes the question
Batch norm stops asking you to pick the right scale and enforces it instead. Per feature, over the batch:
$$\hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \qquad y = \gamma,\hat{x} + \beta$$
The learned $\gamma$ and $\beta$ let the network undo the normalization if it wants to. At eval time the batch statistics are replaced by running averages kept during training, which is the single most common source of "works in training, broken in eval". Layer norm normalizes over the features of one example instead, so it has no batch dependence and no train/eval gap. That is why transformers use it.
Common mistakes
- Initializing with a fixed std copied from a tutorial. The right number depends on fan-in.
- Judging init by the loss curve. Look at per-layer activation std and the fraction of saturated tanh or dead ReLU units at step 0.
- Forgetting eval mode with batch norm. Running statistics are only used when you ask for them.
- Using batch norm with batch size 2. The statistics are too noisy to mean anything.
Check yourself
- Derive $\mathrm{Var}(y) = n,\mathrm{Var}(w),\mathbb{E}[x^2]$ and say which independence assumptions you used.
- Why does ReLU turn $1/n$ into $2/n$?
- A 27-class model starts at loss 12. What is wrong and what is the one-line fix?
- Why does layer norm remove the train/eval mismatch that batch norm has?