How gradient boosting fits residuals
A random forest builds many deep trees independently and averages them. Gradient boosting does the opposite: it builds many shallow trees one after another, and each tree is trained to fix what the ensemble so far still gets wrong. The phrase "fits residuals" is nearly the whole algorithm. The word "gradient" tells you why it generalizes beyond squared error.
The recipe
Start with a constant prediction $F_0$. Then for $m = 1, \dots, M$:
- Compute the residual of every training point: $r_i = y_i - F_{m-1}(x_i)$.
- Fit a small tree $h_m$ to predict $r_i$ from $x_i$.
- Update $F_m(x) = F_{m-1}(x) + \nu , h_m(x)$, with a learning rate $\nu$ such as 0.1.
The final model is the sum $F_0 + \nu\sum_m h_m$. Shrinking each step by $\nu$ is the regularizer that makes boosting work: many small corrections generalize better than a few large ones.
Why residuals are gradients
Write the loss for squared error as $L = \tfrac{1}{2}(y - F)^2$. The derivative with respect to the prediction is $\partial L / \partial F = F - y$, so the negative gradient is exactly $y - F$, the residual. Each round is a step of gradient descent, but taken in function space: instead of nudging parameters, we add a new function pointing downhill.
That reading makes the generalization automatic. For any differentiable loss, fit the next tree to the negative gradient of the loss at the current predictions:
- Squared error: $-\partial L/\partial F = y - F$.
- Log-loss with $F$ as the log-odds: $-\partial L/\partial F = y - p$, where $p = \sigma(F)$. (The same $p - y$ from week 4, with the sign flipped.)
- Absolute error: $-\partial L/\partial F = \mathrm{sign}(y - F)$.
XGBoost goes one step further. It also uses the second derivative $h_i$ of the loss and sets the value of a leaf to $w = -G/(H + \lambda)$, where $G$ and $H$ sum the gradients and second derivatives of the points in the leaf and $\lambda$ is an L2 penalty on leaf values. For squared error $g_i = F_i - y_i$ and $h_i = 1$, so with $\lambda = 0$ the leaf value is the plain average of the residuals, which is what we will compute below.
A worked example you can check
Six points, one feature. Use stumps (trees with one split) and $\nu = 0.5$.
- $x$: 1, 2, 3, 4, 5, 6
- $y$: 1.0, 1.4, 1.1, 3.0, 3.4, 2.9
Round 0. $F_0$ is the mean of $y$, which is $2.1333$. The mean squared error is $0.972$.
Round 1. Residuals: $-1.133, -0.733, -1.033, 0.867, 1.267, 0.767$. The best single split is at $x \le 3.5$. The left leaf is the average of its three residuals, $-0.967$, and the right leaf is $+0.967$. After the update $F_1 = 2.1333 + 0.5 \cdot (\mp 0.967)$, which gives $1.65$ on the left and $2.617$ on the right. MSE drops to $0.271$.
Round 2. New residuals: $-0.65, -0.25, -0.55, 0.383, 0.783, 0.283$. Same split, leaves $\mp 0.483$. Now $F_2 = 1.408$ on the left and $2.858$ on the right. MSE is $0.096$.
Round 3. Leaves $\mp 0.242$, giving $1.287$ and $2.979$, MSE $0.052$.
Notice what is happening: with $\nu = 0.5$ every round closes half of the remaining gap between the group average and the current prediction. The ensemble creeps toward the group means (1.167 on the left, 3.1 on the right) and never jumps. Later rounds will start finding splits inside the groups, which is where the small within-group differences get modelled and, if you run too long, where noise gets modelled too.
import numpy as np
from sklearn.tree import DecisionTreeRegressor
x = np.array([1, 2, 3, 4, 5, 6.0]).reshape(-1, 1)
y = np.array([1.0, 1.4, 1.1, 3.0, 3.4, 2.9])
lr = 0.5
F = np.full(len(y), y.mean())
print("round 0 mse", round(float(np.mean((y - F) ** 2)), 4))
for m in range(1, 4):
residual = y - F
stump = DecisionTreeRegressor(max_depth=1).fit(x, residual)
F = F + lr * stump.predict(x)
print("round", m, np.round(F, 3), "mse", round(float(np.mean((y - F) ** 2)), 4))
Knobs that matter
- Learning rate $\nu$ and number of rounds $M$ trade off. Halve $\nu$, roughly double $M$. Fix a small $\nu$ (0.05 to 0.1) and let early stopping on a validation set choose $M$.
- Tree depth controls interaction order. Depth 1 is an additive model, depth 3 allows three-way interactions. Boosting wants weak learners: 3 to 6 levels is typical.
- Subsampling rows and columns per round adds randomness and cuts variance.
- $\lambda$ and the minimum gain in XGBoost prune splits that do not pay for themselves.
Common mistakes
- Tuning the number of rounds on the test set. Use a validation set or cross-validation with early stopping, then evaluate once.
- Deep trees with a large learning rate. The ensemble memorizes the training set in a handful of rounds.
- Expecting boosting to extrapolate. Trees output averages of training targets, so predictions are capped inside the range of $y$.
- Comparing models at default settings. Boosting is sensitive to $\nu$, depth and $M$. A fair bake-off tunes every model the same way.
Check yourself
- Show that the negative gradient of $\tfrac{1}{2}(y-F)^2$ is the residual.
- With $\nu = 0.5$, how much of the group-mean gap remains after 3 rounds? Does the table above agree?
- Why does boosting use shallow trees while a random forest uses deep ones?
- What does XGBoost's leaf formula reduce to for squared error with $\lambda = 0$?