Home / Learn / Data & AI / ML foundations / Lesson 6.6
All lessons

Module 6: Ensembles: random forests and boosting · Lesson 6.6 · about 30 min

Gradient boosting

Start from the average, then add small trees one at a time, each fitted to the residuals: what the model still gets wrong. This is gradient descent in function space, the idea behind XGBoost and LightGBM. Boost by hand.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 6 · about 30 minutes · read first: 1.5, 5.4, 6.5. You’ll be able to run gradient boosting by hand for a few rounds, explain residuals as gradients, and set the learning rate, tree depth and number of rounds.

Gradient boosting starts with a rough guess, then adds small trees one at a time, each one a gentle correction aimed at whatever is still wrong.

1. Why this matters

Gradient-boosted trees (XGBoost, LightGBM, CatBoost) win most machine-learning competitions on table data and run in production at banks, shops and search engines. If you work with spreadsheets and databases, it is the model you will most likely end up shipping. Under the hood it is just regression trees (Lesson 5.4) and gradient descent (Lesson 1.5), put together.

2. Fitting the residuals

For a number to predict, with squared error:

  1. Start with a constant: F0(𝐱)=y‾, the average.
  2. Work out the residuals ri=yi−F0(𝐱i): what is still unexplained for each row.
  3. Fit a small regression tree h1 to the residuals (not to y).
  4. Update: F1=F0+ηh1, where η (eta) is the learning rate, say 0.1.
  5. Repeat from step 2 with the new residuals.

Each tree is a small correction. Adding only a fraction η of each one, rather than all of it, leaves room for later trees and overfits less. Think of a golfer: the first shot gets near the hole, each putt aims at what is left, and a careful golfer taps rather than slams.

Round 0: the average. Four houses priced 100, 120, 200 and 220 ($1,000s). Predict 160 for every one.

The residuals: −60, −40, +40, +60. Small houses are overpriced by the guess, big ones underpriced.

Round 1. A stump on the residuals splits at size 80: left leaf −50, right leaf +50. Take half a step (η = 0.5): predictions 135 and 185.

Round 2. New residuals −35, −15, +15, +35. The same split gives leaves −25 and +25; half a step: 122.5 and 197.5.

The error shrinks every round. Sum of squared residuals: 10,400, then 2,900, then 1,025. Later rounds start fixing smaller details, like the gap between the two small houses.

Try it: boost one tree at a time
trees
training error (RMSE)
test error (RMSE)
Top: the data and the model so far (it starts as a flat line at the average). Bottom: the residuals, y minus the model, which is all that is left to learn; the dashed steps are the newest tree, fitted to them. Each tree is a stump (one question) by default, too weak alone, but each fixes a little of what is left. A small learning rate needs more trees and usually generalises better; a large one learns fast and starts fitting noise sooner.

3. Why “gradient”?

For squared error L=12(y−F)2, the slope (derivative) with respect to the prediction F is −(y−F): the residual with a minus sign. So “fit a tree to the residuals and take a step” is the same as “fit a tree to the negative gradient and take a step”. It is gradient descent, where the thing being adjusted is the whole prediction function instead of a few weights.

Fm(x)
The model's prediction after m rounds: the starting average plus all the small corrections so far.
ri
The pseudo-residual of row i: which way, and how far, the loss wants that prediction to move. For squared error it is just y − F.
hm
The small tree fitted to those residuals in round m.
η
The learning rate: the fraction of each tree's correction actually used.
Physical meaning
Walking downhill in fog. Each tree is one step in the direction that looks steepest from where you stand; the learning rate is the length of your stride. Short strides take longer but rarely overshoot.

That view lets boosting use any loss with a slope. For classification, use log-loss (Lesson 4.3): the negative gradient is y−p, the label minus the predicted probability. For predictions that must resist outliers, use absolute or Huber loss.

4. The settings

  • Learning rate η. 0.01 to 0.3. Smaller is usually better, and slower.
  • Number of trees. Set by early stopping: watch the validation error each round and stop when it hasn’t improved for, say, 50 rounds.
  • Tree depth. Small trees: 2 to 8 levels. Depth 1 (stumps) models each column on its own; depth 2 allows two-column interactions.
  • Subsampling. Train each tree on a random 50 to 80 % of rows (and columns), as in a forest: faster, and usually better.

“Training error keeps falling, so more rounds are always better.” Training error always falls with more rounds, because each tree fixes a bit more of the training data, noise included. The validation error falls, flattens, then rises once the trees start chasing noise. Early stopping watches the validation error and stops at its lowest point.

Maths and principle

Principle: minimise a loss over functions by repeatedly adding a small tree that points downhill.

Round m: compute the negative gradients (“pseudo-residuals”)

ri=−∂L(yi,F)∂F|F=Fm−1(𝐱i)

fit a regression tree hm to the pairs (𝐱i,ri), and update

Fm(𝐱)=Fm−1(𝐱)+ηhm(𝐱)

For squared error L=12(y−F)2, ri=yi−Fm−1(𝐱i), the ordinary residual.

XGBoost’s extras: it also uses the second derivative (a Newton step), and adds penalties on the number of leaves and the size of leaf values: regularisation (Lesson 3.8) for trees.

Try it: a row has y=10, and the current prediction is 6. The new tree predicts +3.5 for it and η=0.5. New prediction? New residual? (6 + 0.5 × 3.5 = 7.75; residual 2.25.)

5. Worked example

Made-up numbers. Four houses, one input (size), prices in $1,000s: 100, 120, 200, 220 for sizes 50, 60, 100, 110. Learning rate 0.5, stumps.

  • Round 0: F0 = mean = 160. Residuals: −60, −40, 40, 60.
  • Round 1: the best stump on the residuals splits at size 80: left mean −50, right mean +50. Update: F1 = 160 + 0.5 × (−50) = 135 for the small houses, 160 + 25 = 185 for the big ones. Residuals: −35, −15, 15, 35.
  • Round 2: again the best split is at 80, with leaf means −25 and +25. F2 = 122.5 and 197.5. Residuals: −22.5, −2.5, 2.5, 22.5.
  • Round 3: the gap between the two groups is now small, so the best stump changes: splitting off the smallest house (size ≤ 55: leaf values −22.5 and +7.5) removes more error than another split at 80.

The training error falls every round: sum of squared residuals 10,400, then 2,900, then 1,025. With a learning rate of 1 the first stump would have jumped straight to 110 and 210; the smaller steps take longer but, on real noisy data, generalise better.

6. Practice

  1. In the simulation, add 100 trees at learning rate 1 and at 0.1. Which has the lower test error?
  2. Why are boosting’s trees kept small?
  3. What is early stopping, and why is it needed for boosting but not for a random forest?
Answers
  1. Usually 0.1: the big steps start chasing noise early, and the test error climbs while the training error keeps falling.
  2. Each tree only needs to fix a little; small trees are weak learners that keep each step gentle and stop the model overfitting quickly.
  3. Stopping at the round where validation error is lowest. Boosting keeps fitting the training data more closely with every tree and eventually fits noise; a forest’s average just settles.
  • Gradient boosting starts from the average and adds small trees fitted to the residuals.
  • Each tree is added at a fraction η (the learning rate) of its full size; smaller η needs more trees and usually generalises better.
  • Residuals are the negative gradient of squared error, so boosting is gradient descent on the prediction function; other losses give other pseudo-residuals.
  • Choose the number of rounds by early stopping on validation data.

Try it on your own data: the regression workbench fits a linear model to your table; its residual plot shows exactly what a first boosting tree would try to fix.

New words: gradient boosting, residual, pseudo-residual, learning rate (shrinkage), early stopping, XGBoost, LightGBM.

Check yourself

1In gradient boosting with squared error, each new tree is fitted to:
2A smaller learning rate (shrinkage) usually needs:
3The number of boosting rounds should be chosen by:

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.