Module 6: Ensembles: random forests and boosting · Lesson 6.3 · about 25 min
Random forests
A random forest grows many deep trees, each on a bootstrap sample and each split choosing from a random handful of columns, then lets them vote. Add trees one by one and watch the jagged boxes smooth into a good boundary.
Module 6 · about 25 minutes · read first: 6.2. You’ll be able to explain how a random forest is built and why the random columns matter, choose its main settings, and read its out-of-bag score.
1. Why this matters
The random forest (Leo Breiman, 2001) is the most dependable first model for table data. It needs almost no tuning, handles numbers and categories, copes with useless columns and odd scales, rarely overfits badly, and gives a built-in test score. Many data teams try a random forest before anything else, and keep it more often than you’d expect.
2. Two kinds of randomness
A random forest is bagged trees (Lesson 6.2) plus one more trick:
- Random rows: each tree is grown on its own bootstrap sample.
- Random columns: at every split, the tree may only choose among a random handful of columns (a fresh handful at each split).
Then all trees vote (classification) or are averaged (regression). The trees are grown deep, with no pruning: each one overfits in its own way, and the vote cancels it out.
Why the random columns? With bagging alone, if one column is very strong, every tree puts it at the top and the trees end up alike. Their mistakes are then correlated, and averaging helps less (Lesson 6.1). Hiding the strong column from some splits forces trees to find other routes, and the forest gets more varied. It is like a quiz team where each member is only allowed to study some of the topics: nobody can copy the best student, so they learn different things.
Tree 1 gets its own bootstrap sample of the customers. At its first split it may only look at 4 of the 16 columns, say tenure, region, age and plan.
It picks the best of those four (tenure), splits, and at the next split draws a new random 4. It keeps going until its leaves are pure.
Trees 2 to 300 do the same with their own samples and their own random columns. The strongest column, contract type, is missing from about three quarters of the first splits, so many trees start somewhere else.
A new customer arrives. 210 trees say "leaves", 90 say "stays". The forest predicts "leaves" with probability 210 ÷ 300 = 0.70.
- B
- The number of trees in the forest.
- m
- How many randomly chosen columns each split may look at (often √p for p columns).
- P̂(y = c | x)
- The forest's probability for class c: the average of the trees' answers, which is close to the share of trees voting for c.
- Physical meaning
- A panel of experts who each studied a different selection of case files and were each told to ignore some of the evidence. None is the best expert, but their vote beats any one of them, because their blind spots are in different places.
3. The settings that matter
- Number of trees. More is never worse, only slower. A few hundred is usual; stop when the out-of-bag score stops improving.
- Columns per split, . The main tuning dial. A common default is of the columns for classification. Smaller means more varied but individually weaker trees.
- Leaf size or depth. Usually left deep; a slightly larger smallest leaf can help on noisy data.
Things a forest does not do well: extrapolate beyond the training range (it is made of trees, Lesson 5.4), explain itself in one picture (a hundred trees is not a rule you can print), or handle very wide, sparse data such as raw text as well as linear models do.
“Too many trees will make a random forest overfit.” Adding trees doesn’t make any single tree more complex; it only makes the average steadier. Accuracy rises and then levels off. What can overfit is the individual tree settings (leaves that are too small on very noisy data), not the number of trees. Extra trees cost only time.
Maths and principle
Principle: average many low-bias, high-variance trees, and make them as uncorrelated as possible so the average removes most of the variance.
The forest’s prediction with trees : for regression the mean, for classification the average share for each class,
Why fewer columns help: from Lesson 6.1, the averaged prediction has variance . As grows the second part vanishes; what remains, , falls as the trees become less correlated, and random columns lower .
Try it: with 25 columns in a classification problem, how many does each split look at by default? (.)
4. Worked example
Made-up numbers. A telecom company predicts which customers will leave, with 16 columns and 20,000 rows.
| model | test accuracy | out-of-bag accuracy |
|---|---|---|
| one deep tree | 84.1 % | (none) |
| bagging, 300 trees, all 16 columns per split | 88.6 % | 88.4 % |
| random forest, 300 trees, 4 columns per split | 89.9 % | 89.7 % |
The random columns add 1.3 points over plain bagging. The out-of-bag score lands within 0.2 points of the test score, so for tuning the forest the team could have skipped the separate validation set.
5. Practice
- In the simulation, compare 1 tree with 60. What happens to the shading and the test accuracy?
- Why are the forest’s trees grown deep instead of pruned?
- Your forest has 500 trees and you raise it to 2,000. What do you expect?
Answers
- The hard boxes become a smooth gradient of vote shares, and test accuracy rises by several points before levelling off.
- Deep trees have low bias; their high variance is what the averaging removes. Pruned trees would add bias that averaging can’t fix.
- Almost the same accuracy, four times the training and prediction time.
- A random forest = bootstrap samples (random rows) + a random handful of columns at every split + a vote.
- The random columns make the trees less alike, so averaging removes more of their errors.
- More trees never hurt accuracy, only speed; the main dial is the number of columns per split.
- Each row is left out by about a third of the trees, which gives a free out-of-bag score.
Try it on your own data: the small model trainer grows a single tree on your CSV and reports its test accuracy: the baseline a random forest (in Python or R) should beat.
New words: random forest, feature subsampling, mtry (columns per split), out-of-bag score, decorrelation.
Check yourself
Sources
- Breiman (2001), Random forests, Machine Learning 45, 5-32 https://doi.org/10.1023/A:1010933404324
- Hastie, Tibshirani and Friedman (2009), The Elements of Statistical Learning, 2nd edition, Springer, Chapter 15 (Random forests)
- scikit-learn user guide: Ensembles, random forests https://scikit-learn.org/stable/modules/ensemble.html#random-forests-and-other-randomized-tree-ensembles
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.