Home / Learn / Data & AI / ML foundations / Lesson 6.5
All lessons

Module 6: Ensembles: random forests and boosting · Lesson 6.5 · about 25 min

Boosting: each model fixes the last one's mistakes

Bagging trains models side by side; boosting trains them one after another, each paying most attention to what the others got wrong. Follow AdaBoost through one round by hand and compare the two ideas.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 6 · about 25 minutes · read first: 6.2. You’ll be able to explain how boosting differs from bagging, follow one round of AdaBoost by hand, and say why boosting can overfit if run too long.

Boosting builds a strong model from a chain of weak ones, where each new model studies hardest the cases the earlier ones got wrong.

1. Why this matters

Boosting turned a theory question (“can many weak learners make a strong one?”, Kearns and Valiant, 1988) into some of the most accurate models in use. AdaBoost (Freund and Schapire, 1995) powered the first real-time face detectors in cameras; its descendant, gradient boosting (Lesson 6.6), runs much of the world’s credit scoring, ranking and demand forecasting.

2. A chain of weak learners

A weak learner is a model only a little better than guessing, such as a stump: a tree with a single question. Boosting builds a strong model from many of them:

  1. Train a weak learner on the data.
  2. Find where it is wrong.
  3. Train the next weak learner to do better exactly there.
  4. Repeat for hundreds of rounds, and combine them all in a weighted vote (or sum).
Picture: bagging versus boosting Top, bagging: the data feeds three trees in parallel, each on its own random sample, and their predictions are averaged. Bottom, boosting: the data feeds tree 1; its leftover errors feed tree 2; those leftovers feed tree 3; the final prediction is the sum of small steps from every tree. Bagging (random forest): side by side data tree 1 tree 2 tree 3 average Own random sample each; averaging cuts variance. Boosting: one after another data tree 1 fits y tree 2 fits errors tree 3 fits errors prediction = start + η·tree 1 + η·tree 2 + … Small trees in a chain; adding them cuts bias. fibub.com
Bagging trains many strong, deep trees independently (they can even run at the same time) and averages away their noise. Boosting trains many weak, shallow trees in a chain; each one learns from the mistakes left by all the trees before it, and their small steps add up to a strong model.

Bagging and boosting attack different errors. Bagging averages strong, overfitting models side by side to cut variance (wobble). Boosting adds up weak, underfitting models one after another to cut bias (being too simple). Adding models to bagging can’t hurt much; boosting, run too long, starts fitting noise, so the number of rounds must be chosen with validation data.

3. AdaBoost

AdaBoost (“adaptive boosting”) gives every training row a weight, starting equal. A weight is how much that row counts when the next stump is scored.

  1. Train a stump that makes the weighted error ε (epsilon) as small as possible: rows with big weights count more.
  2. Give the stump a say α=12ln⁡1−εε (alpha): big when its error is small, zero at 50 %.
  3. Multiply the weights of the rows it got wrong by eα and of those it got right by e−α, then rescale so the weights add to 1.
  4. Repeat. The final classifier takes the sign of the weighted vote ∑tαtht(𝐱), with each stump answering +1 or −1.

Ten rows, equal weights of 0.1 each. Stump 1 gets 3 of them wrong, so its weighted error ε = 0.3.

Its say. α = ½ ln(0.7 ÷ 0.3) = 0.424. A decent stump earns a decent vote.

Reweight. The 3 mistakes: 0.1 × e0.424 = 0.153 each. The 7 right ones: 0.1 × e−0.424 = 0.065 each.

Rescale to add up to 1. The total is 0.917, so the mistakes become 0.167 each and the others 0.071. The 3 hard rows now hold half of all the weight.

Next round. Stump 2 is trained on these weights, so getting those three rows right is now worth as much as all the other seven together.

wi
The weight of row i: how much it counts when the next stump is scored.
ε
The stump's weighted error: the total weight of the rows it gets wrong.
α = ½ ln((1 − ε)/ε)
The stump's say in the final vote: 0 for a coin-flip stump, large for an accurate one.
F(x) = Σ αt ht(x)
The final model: a weighted sum of all the stumps' ±1 answers. Its sign is the prediction.
Physical meaning
A student revising for an exam who keeps a pile of the questions they got wrong and practises those more each day. Each day's practice is narrow, but over weeks the gaps close.

Rows that keep being misclassified get heavier and heavier, so later stumps concentrate on them. That is also AdaBoost’s weakness.

“Boosting focuses on hard cases, so it is great on messy data.” A row with a wrong label is “hard” for ever: no honest model can get it right, so its weight keeps growing and later stumps bend themselves to fit it. On noisy data AdaBoost can overfit badly. Gradient boosting with a gentler loss (Lesson 6.6), a small learning rate and early stopping copes much better.

Maths and principle

Principle: build an additive model one piece at a time, each piece chosen to reduce what is still wrong.

The model after M rounds:

FM(𝐱)=∑t=1Mαtht(𝐱),yˆ=sign⁡FM(𝐱)

Why those weights: AdaBoost is exactly what you get by adding one stump at a time to minimise the exponential loss ∑ie−yiF(𝐱i) with labels yi=±1 (Friedman, Hastie and Tibshirani, 2000). Choosing the next piece to cut a loss is the general idea that Lesson 6.6 turns into gradient boosting.

Try it: a stump has weighted error 0.2. What is its say α, and by what factor do the weights of its mistakes grow? (α=12ln⁡4=0.693; mistakes are multiplied by e0.693=2 and correct rows by 0.5, before rescaling.)

4. Worked example

Made-up numbers. Ten rows, each with weight 0.1. Stump 1 gets 3 wrong.

  • Weighted error ε=0.3. Say: α1=12ln⁡(0.7/0.3)=0.424.
  • Wrong rows: 0.1 × e0.424 = 0.1 × 1.528 = 0.153. Right rows: 0.1 × e−0.424 = 0.1 × 0.655 = 0.065.
  • Total: 3 × 0.153 + 7 × 0.065 = 0.458 + 0.458 = 0.917. Rescaled: each wrong row 0.167, each right row 0.071.

After rescaling, the three mistakes hold exactly half the total weight (3 × 0.167 = 0.5). That is no accident: AdaBoost always reweights so the last stump would score exactly 50 % on the new weights, forcing the next stump to find something new.

5. Practice

  1. Why can’t you train boosting’s rounds in parallel, as you can bagging’s trees?
  2. What is a stump’s say α if its weighted error is 0.5?
  3. Why does AdaBoost struggle with mislabelled data?
Answers
  1. Each round needs the previous rounds’ mistakes (the new weights) before it can start.
  2. 12ln⁡1=0: a coin-flip stump gets no vote.
  3. A wrong label is always “misclassified”, so its weight keeps growing and later stumps bend to fit it.
  • Boosting trains weak learners one after another, each focusing on the earlier ones’ mistakes.
  • Bagging mainly cuts variance; boosting mainly cuts bias.
  • AdaBoost raises the weights of misclassified rows and gives each stump a say α = ½ ln((1 − ε)/ε).
  • Boosting run too long fits noise, especially wrong labels; choose the number of rounds with validation data.

Try it on your own data: the small model trainer grows a depth-1 tree (a stump) if you set the maximum depth to 1: see how weak one stump is on your CSV.

New words: boosting, weak learner, stump, AdaBoost, sample weights, additive model, exponential loss.

Check yourself

1The main difference between bagging and boosting is:
2In AdaBoost, after a round, the weights of the rows the model got wrong:
3Boosting mainly reduces:

Sources

  • Freund and Schapire (1997), A decision-theoretic generalization of on-line learning and an application to boosting, Journal of Computer and System Sciences 55(1), 119-139 https://doi.org/10.1006/jcss.1997.1504
  • Friedman, Hastie and Tibshirani (2000), Additive logistic regression: a statistical view of boosting, The Annals of Statistics 28(2), 337-407 https://doi.org/10.1214/aos/1016218223
  • Hastie, Tibshirani and Friedman (2009), The Elements of Statistical Learning, 2nd edition, Springer, Chapter 10 (Boosting and additive trees)

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.