The bias-variance decomposition with real numbers
Every model makes errors for three different reasons, and only two of them are yours to fix. Bias is the error from a model too simple to express the truth. Variance is the error from a model so flexible that it follows the noise in whatever sample it was given. Noise is the error no model can remove. The tradeoff is not a metaphor. It is an identity, and you can measure each term.
The decomposition
Suppose the truth is $y = f(x) + \varepsilon$ with $\mathbb{E}[\varepsilon] = 0$ and $\mathrm{Var}(\varepsilon) = \sigma^2$. You train a model $\hat f$ on a random training set $D$ and test it at a fixed point $x_0$ against a fresh label. Then
$$\mathbb{E}\big[(y - \hat f(x_0))^2\big] = \underbrace{\sigma^2}{\text{noise}} + \underbrace{\big(\mathbb{E}[\hat f(x_0)] - f(x_0)\big)^2}{\text{bias}^2} + \underbrace{\mathbb{E}\big[(\hat f(x_0) - \mathbb{E}[\hat f(x_0)])^2\big]}_{\text{variance}}$$
The expectation is over training sets, not over test points. Bias asks: if I could average my model over infinitely many training sets, how far from the truth would the average be? Variance asks: how much does one fitted model wobble around that average?
The derivation is two moves. Add and subtract $\mathbb{E}[\hat f]$ inside the square, and observe that the cross terms vanish because $\varepsilon$ is independent of $D$ and the deviation $\hat f - \mathbb{E}[\hat f]$ has mean zero.
Measure it with real numbers
We can do exactly this experiment, because here we know the truth. Let $f(x) = \sin(2\pi x)$, noise $\sigma = 0.3$ (so $\sigma^2 = 0.09$), 30 inputs on an even grid, and fit polynomials of increasing degree to 2,000 independent noise draws.
import numpy as np
rng = np.random.default_rng(0)
f = lambda x: np.sin(2 * np.pi * x)
x = np.linspace(0, 1, 30)
sigma, trials = 0.3, 2000
for deg in [1, 3, 5, 9, 12]:
preds = np.empty((trials, x.size))
for t in range(trials):
y = f(x) + rng.normal(0, sigma, x.size)
preds[t] = np.polynomial.Polynomial.fit(x, y, deg)(x)
bias2 = np.mean((preds.mean(0) - f(x)) ** 2)
var = np.mean(preds.var(0))
print(deg, round(bias2, 4), round(var, 4), round(bias2 + var + sigma**2, 4))
Here is what that printed on my machine (numpy default generator, seed 0), one line per degree:
- Degree 1: bias² 0.2197, variance 0.0060, total 0.3157
- Degree 3: bias² 0.0056, variance 0.0121, total 0.1076
- Degree 5: bias² 0.0000, variance 0.0182, total 0.1082
- Degree 9: bias² 0.0000, variance 0.0295, total 0.1195
- Degree 12: bias² 0.0000, variance 0.0390, total 0.1291
Total means bias² + variance + 0.09 of noise.
Read it left to right. A straight line cannot bend, so its bias² is 0.22 and swamps everything. Degree 3 almost captures one period of a sine, bias² drops to 0.006, and total error falls to 0.108, close to the floor of 0.09 that noise sets. After that, bias is gone and every extra coefficient only adds variance.
The variance column is not random. For a linear model with $d + 1$ coefficients fitted on $n$ points, the average variance of the fitted values is $\sigma^2 (d+1)/n$. For degree 9 that is $0.09 \times 10 / 30 = 0.03$, and we measured 0.0295. Each parameter costs about $\sigma^2/n = 0.003$ of variance, so flexibility has a fixed price you can compute in advance.
One honest caveat: this table uses a friendly fixed grid. In a run where I drew the 30 inputs at random instead, the degree-9 variance was hundreds of times larger, because gaps between inputs let the polynomial swing wildly. Variance depends on the data design as well as on the model.
What moves each term
- More flexible model (higher degree, deeper tree, smaller $k$ in k-NN): bias down, variance up.
- More training data: variance down. Bias does not move, since a line stays a line however many points you give it.
- Regularization (ridge, lasso, early stopping, pruning): variance down, bias up a little. You are paying bias to buy variance, and it is a good trade when variance is the bigger term.
- Averaging many high-variance models (bagging, random forests) cuts variance while leaving bias about the same.
- Better features can reduce bias and variance together, which is why they beat tuning.
Diagnosing a real model
You cannot measure bias and variance on real data, because you never know $f$. You can see their symptoms with a learning curve: plot training error and validation error against training set size.
- Both errors high and close together: high bias. More data will not help; change the model or the features.
- Training error low, validation error much higher: high variance. More data, simpler model or regularization will help.
Common mistakes
- Treating the tradeoff as a law that applies to every model. Very large neural networks can show validation error falling again past the interpolation point. For the classical models in this roadmap, the identity above is the whole story.
- Confusing noise with variance. Noise stays however much data you collect. Variance shrinks.
- Reading a train-test gap as proof of overfitting. Some gap is normal. Look at validation error as complexity rises: if it keeps falling, you are not overfitting yet.
- Picking lambda or depth on the test set. That moves the test set into training.
Check yourself
- In the table, why does the total error at degree 12 exceed the one at degree 3 even though bias is zero in both?
- Predict the variance column's value at degree 15 with $n = 30$ before running it.
- Which term does doubling the data change for a straight-line fit?
- For ridge with growing lambda, which term rises and which falls?