Module 6: Ensembles: random forests and boosting · Lesson 6.1 · about 20 min
Wisdom of crowds: why many models beat one
Many voters who are each only a little better than a coin can be right almost every time, if their mistakes are independent. That is the idea behind ensembles. Change the crowd and see when it works and when it fails.
Module 6 · about 20 minutes · read first: 1.6, 5.5. You’ll be able to explain why averaging many models works, compute a majority vote’s accuracy, and say what an ensemble needs.
1. Why this matters
An ensemble is a group of models whose answers are combined. Ensembles are behind most winning entries in data competitions and many systems in daily use: random forests and gradient boosting (this module), stacked models, even the habit of averaging several runs of a neural network. The reason they work fits in one simulation.
2. A crowd of weak voters
In 1906 the scientist Francis Galton watched 787 people at a country fair guess the weight of an ox. Single guesses were all over the place; the middle guess was within 1 % of the truth. Some people guessed too high and some too low, and the errors cancelled out.
The same happens with classifiers. Suppose each of models is right with probability , and their mistakes are independent (one model being wrong tells you nothing about whether another is). The majority vote is right whenever more than half are right, and for that becomes more and more likely as grows.
Three spam filters, each right 80 % of the time, with independent mistakes. They vote on each email.
All three right: 0.8 × 0.8 × 0.8 = 0.512.
Exactly two right: there are 3 ways to pick which one is wrong, each 0.8 × 0.8 × 0.2 = 0.128. Together 0.384.
The majority is right in both cases: 0.512 + 0.384 = 0.896. Three 80 % filters make one 90 % filter.
But if they are copies of one filter, they are wrong on the same emails, and the vote stays at 80 %. The gain came entirely from their differences.
3. The catch: independence
Real models trained on the same data make many of the same mistakes. In the simulation, “copying” is the chance that the whole crowd simply follows one shared vote. At 100 % copying, 101 voters are no better than one.
So an ensemble needs two things:
- Each model better than guessing ( for two classes). A crowd of bad voters is confidently wrong.
- Diversity: models that differ, so their mistakes overlap as little as possible.
- σ²
- The variance of one model's prediction: how much it wobbles around the truth.
- ρ
- The correlation between two models' errors: 0 if they wobble independently, 1 if they wobble together.
- n
- The number of models averaged.
- ρσ² + (1 − ρ)σ²/n
- The variance of the average: a shared part that never goes away, plus a part that shrinks as you add models.
- Physical meaning
- Asking for directions. Five strangers who each guess are more reliable together; five people who all read the same wrong sign are not. Only independent opinions average out.
The rest of this module is about manufacturing diversity: random samples of rows (bagging, Lesson 6.2), random choices of columns (random forests, 6.3), or models trained one after another on each other’s mistakes (boosting, 6.5).
“Add enough models and the ensemble becomes perfect.” Only the independent part of the error averages away. If the models share a blind spot (the same missing column, the same biased data), every one of them makes that mistake, and a million copies still make it. Averaging fixes wobble, not shared bias.
Maths and principle
Principle: averaging reduces variance, but only the part of it that is not shared.
Majority vote of (odd) independent voters each right with probability : the number right follows a binomial distribution, so
Here (“n choose k”) counts the ways to pick which voters are right.
Averaging numbers: predictions, each with variance and correlation between any two, have an average with variance
As grows, the second part vanishes, but stays. Lowering is the whole game.
Try it: five independent voters, each right 60 % of the time. Majority right? (.)
4. Worked example
Made-up numbers. Three spam filters, each 80 % accurate, are combined by majority vote.
- Independent mistakes: right if at least two are right: . About 90 %, from three 80 % models.
- Always the same mistakes (three copies of one filter): 80 %. No gain.
- Correlation 0.5 in their numeric scores, each with variance 1: the averaged score has variance instead of 1. Some gain, much less than the that full independence would give.
5. Practice
- In the simulation, with each voter right 60 %, how many independent voters do you need for the majority to be right 90 % of the time?
- Why would three copies of the same model add nothing?
- Why must the voters be better than a coin?
Answers
- About 41 (the curve passes 90 % in the low 40s).
- They make exactly the same mistakes, so the vote always equals any one of them.
- If each is more often wrong than right, the majority is wrong even more reliably.
- An ensemble combines many models by voting or averaging.
- With independent mistakes and each model better than a coin, the majority gets more accurate as the crowd grows.
- Correlated mistakes don’t cancel: the shared part ρσ² stays however many models you add.
- Ensemble methods are ways of making models that differ: random rows, random columns, or learning from each other’s errors.
Try it on your own data: the small model trainer trains several kinds of model on the same CSV; compare where each one goes wrong.
New words: ensemble, majority vote, independent errors, diversity, correlation between models.
Check yourself
Sources
- Galton (1907), Vox populi, Nature 75, 450-451 https://doi.org/10.1038/075450a0
- Hastie, Tibshirani and Friedman (2009), The Elements of Statistical Learning, 2nd edition, Springer, Section 15.2 (variance of an average of correlated trees)
- James, Witten, Hastie and Tibshirani (2021), An Introduction to Statistical Learning, 2nd edition, Springer, Section 8.2 (Bagging, random forests, boosting) https://www.statlearning.com/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.