Module 7: Judging models honestly · Lesson 7.1 · about 25 min
Bias and variance
Every model's test error splits into bias (wrong on average), variance (changes with the sample) and noise (nobody can remove). Train the same model on 20 different samples and see both, then read the dartboard.
Module 7 · about 25 minutes · read first: 3.7, 6.1. You’ll be able to name the three parts of test error, tell from training and validation scores whether a model is too simple or too jumpy, and pick the right fix for each.
1. Why this matters
When a model disappoints, ask first: is it too simple, or too jumpy? The two problems need opposite fixes. One needs more flexibility, the other needs less. Guess wrong and you can waste days. Bias and variance are the two words for this question. They also explain why random forests average many trees and why boosting adds trees up (Module 6).
2. Two ways to be wrong
Picture an archer. Some archers always hit low and to the left: their arrows land close together, but in the wrong place. Others aim at the centre on average, but their arrows fly all over the board. Models behave the same way.
Imagine training the same kind of model many times, each time on a fresh sample from the same source.
- Bias is how far the average of those models is from the truth. A model that is too simple misses the same way every time, like the archer who always hits low.
- Variance is how much the models differ from each other. A model that is too flexible follows the noise in each sample, so every sample gives a different fit, like the archer whose arrows scatter.
The simulation does this for real. It takes 20 samples of 12 points from one curve and fits 20 polynomials, so you can see how far the fits wander and where their average lands.
Watch it happen
Here is the whole idea with five small numbers. At one point the true value is 5.
Train five times. The same model is trained on five different samples. At our point it predicts 6, 7, 6, 5 and 6. The truth is 5.
Find the centre. The average prediction is (6 + 7 + 6 + 5 + 6) ÷ 5 = 6. The models miss by 1 on average, all in the same direction.
Bias². Square that miss: (6 − 5)² = 1. This is the part of the error that more data will not fix, because the model itself is off.
Variance. How far does each prediction sit from the centre of 6? Distances 0, 1, 0, −1, 0. Squares 0, 1, 0, 1, 0. Average: 2 ÷ 5 = 0.4.
Add the noise. Real data also wobble by themselves. Say that noise has variance 0.6. Expected squared error = 1 + 0.4 + 0.6 = 2.0. Each part has a different cure.
3. The trade-off and the cures
More flexibility lowers bias and raises variance. Less flexibility does the opposite. Test error is lowest somewhere in between: the U-shaped curve of Lesson 3.7.
You can’t watch bias and variance directly on real data, because you only have one sample. But training and validation error give the game away:
| training error | validation error | diagnosis | fixes | |
|---|---|---|---|---|
| A | high | high, close to training | high bias (underfitting) | more flexible model, better features, less regularisation, boosting |
| B | low | much higher | high variance (overfitting) | more data, simpler model, regularisation, averaging (bagging, forests), early stopping |
| C | low | low, close | good | ship it |
The third part, noise, is the randomness in the data itself: measurement error, or things your columns don’t capture. No model can remove it. Only better data can.
- y = f(x) + ε
- what we observe: a true pattern f plus random noise ε
- f̂ (f-hat)
- the model you get from one particular training sample
- Bias²
- how far the average model is from the truth, squared: the error built into the kind of model you chose
- Variance
- how much the model changes from one sample to the next: the error that comes from the luck of the sample
- σ² (noise)
- the wobble in the data that no model can predict
- Physical meaning
- A forecaster that always says "same as last week" is steady but stubbornly wrong (bias). One that rebuilds itself from scratch every Monday is clever but nervous (variance). Good models sit between, and part of tomorrow's sales will always be a surprise (noise).
Maths and principle
Principle: averaged over possible training sets, squared test error splits into three parts that can be measured separately.
The decomposition: data come from with noise variance . Let be the model trained on a random sample, and its average over samples ( means “average over many tries”). At a point :
where
The cross terms vanish because the noise in a new is independent of the training sample.
Analogy: the dartboard. Bias is where the cluster of darts is centred; variance is how widely they scatter.
Try it: at a point, the truth is 5. Five models trained on different samples predict 6, 7, 6, 5, 6. Bias² and variance? (Average 6, so bias² = 1. Variance = (0 + 1 + 0 + 1 + 0)/5 = 0.4.)
“More data always helps.” More data mostly cuts variance: each fit depends less on the luck of the sample. It does little for bias. A straight line fitted to a million points from a curve is still a straight line, missing the curve in the same places. If training and validation error are both high and close, collect better features or use a more flexible model, not just more rows.
4. Worked example
Made-up numbers. A shop forecasts daily sales. Errors are root mean squared (RMSE), in units sold.
| model | training RMSE | validation RMSE |
|---|---|---|
| average of last week | 41 | 42 |
| linear model on 6 columns | 30 | 31 |
| boosted trees, 2,000 rounds, depth 10 | 4 | 38 |
| boosted trees, early stopping (310 rounds), depth 4 | 21 | 24 |
The first two are close on training and validation: high bias. The third has a huge gap: high variance. The fourth is the best compromise. The noise floor is probably near 20: even perfect knowledge of the columns can’t predict a sudden bus strike.
5. Practice
- In the simulation, which degree gives the lowest bias² + variance?
- Training error 3 %, validation error 3.5 %, but the business needs 1 %. Bias or variance problem?
- Why does a random forest mostly reduce variance, and boosting mostly bias?
Answers
- Around 3 on most draws (degree 1 is a close second, because this curve is gentle).
- Bias: training and validation agree, so the model isn’t overfitting; it needs more flexibility or better features.
- A forest averages deep trees that already have low bias, cancelling their variance. Boosting adds up very simple, high-bias pieces until the sum is flexible enough.
Try it on your own data: the small model trainer shows training and test accuracy side by side; use the table above to diagnose your model.
New words: bias, variance, irreducible error (noise), bias-variance decomposition, learning curve.
- Test error = bias² + variance + noise.
- Bias: the model misses the same way every time (too simple). Variance: the model changes a lot with the sample (too flexible).
- High training and validation error, close together, means high bias. Low training error with much higher validation error means high variance.
- More data, simpler models and averaging cut variance; more flexibility and better features cut bias. Nothing removes noise except better data.
Check yourself
Sources
- James, Witten, Hastie, Tibshirani and Taylor (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Section 2.2.2 (the bias-variance trade-off) https://www.statlearning.com/
- Hastie, Tibshirani and Friedman (2009), The Elements of Statistical Learning, 2nd edition, Springer, Sections 7.2 and 7.3
- Geman, Bienenstock and Doursat (1992), Neural networks and the bias/variance dilemma, Neural Computation 4(1), 1-58 https://doi.org/10.1162/neco.1992.4.1.1
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.