Home / Learn / Data & AI / ML foundations / Lesson 5.4
All lessons

Module 5: Decision trees · Lesson 5.4 · about 20 min

Trees that predict numbers

A regression tree predicts a number: the average of the training rows in each leaf. Its splits lower the spread (variance) of the target. See the staircase it draws and work out a split by hand.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 5 · about 20 minutes · read first: 3.5, 5.3. You’ll be able to build a regression tree split by hand, read its staircase predictions, and explain why trees cannot extrapolate.

A regression tree sorts similar cases into groups and predicts each group's average, so its answers form a staircase.

1. Why this matters

Many business questions are numbers, not classes: next week’s sales, a delivery time, a house price. Regression trees answer them, and they are the building block of gradient boosting (Lesson 6.6), which wins most competitions on table data.

2. Same tree, different leaves

A regression tree asks the same kind of questions as a classification tree. Two things change:

  • The leaf’s answer is the average of the target over the training rows in that leaf.
  • Impurity is how spread out the target is in a box: its variance, the average squared distance from the box’s own average. A split is good if each side’s values sit close to their own average.

So the gain of a split is the drop in weighted variance (the same as the drop in total squared error, Lesson 3.2).

One guess for every day. Six days of ice-cream sales: 4, 5, 6, 14, 16, 18 (hundreds). With no questions, the best single guess is the mean, 10.5.

How wrong is that? The squared distances from 10.5 average to 31.9. That variance is the "impurity" of the whole group.

Ask "warmer than 21 °C?" Cool days {4, 5, 6} average 5, with variance 0.67. Warm days {14, 16, 18} average 16, with variance 2.67. Weighted: 1.67. A drop of 30.25: a superb question.

The staircase. The tree now says 500 on cool days and 1,600 on warm days. A 40 °C heatwave? Still 1,600, because the last step carries on flat.

c = ȳ
The leaf's prediction: the average of the training values that landed in it.
I = (1/n) Σ (yi − ȳ)²
The leaf's impurity: its variance, the average squared distance from that average.
gain
Variance before the split minus the weighted variance after it.
Physical meaning
Pricing flats by neighbourhood. Instead of a formula, group similar flats and quote each group's average price. Finer groups give finer quotes, but a group of one flat quotes that flat's odd price exactly.
Try it: a staircase prediction
leaves (steps)
training error (RMSE)
test error (RMSE)
Made-up data: daily ice-cream sales (in hundreds) against the day's temperature. Each leaf predicts the average sales of the training days that land in it, so the prediction is flat inside each box and jumps between boxes. A split is chosen to lower the spread (variance) of sales on each side the most. Deeper trees make finer steps; past depth 2 or 3 they start chasing single noisy days and the test error rises again.

3. A staircase

Because each leaf predicts one fixed number, the prediction is flat inside each box and jumps at each threshold: a staircase in one dimension, a set of flat-topped blocks in two. More depth means more, finer steps.

Two consequences:

  • No extrapolation. Beyond the training data the last step just carries on flat. A straight-line model would keep going up; a tree stays at the last leaf’s average. That is safe for some jobs and wrong for others (prices that keep rising).
  • No smoothness. Two days 0.1 °C apart can get very different predictions if a threshold falls between them. Averaging many trees (Module 6) smooths the steps out.

“A tree will follow the trend into new territory, like a line does.” It can’t. Every prediction is the average of some training values, so a tree never predicts above the highest training value or below the lowest. If the future is likely to go beyond your past data (growing sales, rising prices), use a model that can extrapolate, or predict the change rather than the level.

Maths and principle

Principle: within a box, the single number with the lowest squared error is the mean, so each leaf predicts its mean, and splits are chosen to lower the total squared error.

Leaf value and impurity for the rows yi in a box with n rows and mean y‾:

c=y‾=1n∑iyi,I=1n∑i(yi−y‾)2

Gain of a split: I(parent)−nLnI(L)−nRnI(R), the same formula as Lesson 5.3 with variance in place of Gini.

Try it: values 2, 4, 10, 12. Which split is best: {2} | {4, 10, 12} or {2, 4} | {10, 12}? (Second: each side has variance 1, total weighted 1. The first has a right side of mean 8.67 and variance 11.6, weighted 8.7.)

4. Worked example

Made-up numbers. Daily ice-cream sales (in hundreds) on six days:

temperature (°C) 12 15 18 24 27 30
sales 4 5 6 14 16 18

Mean 10.5; variance = (6.5² + 5.5² + 4.5² + 3.5² + 5.5² + 7.5²) / 6 = 191.5 / 6 = 31.92.

  • Split at 21 °C: left {4, 5, 6}, mean 5, variance (1 + 0 + 1)/3 = 0.667. Right {14, 16, 18}, mean 16, variance (4 + 0 + 4)/3 = 2.667. Weighted: 0.5 × 0.667 + 0.5 × 2.667 = 1.667. Gain = 31.92 − 1.67 = 30.25.
  • Split at 16.5 °C: left {4, 5}, variance 0.25; right {6, 14, 16, 18}, mean 13.5, variance 20.75. Weighted: 2/6 × 0.25 + 4/6 × 20.75 = 13.92. Gain 18.0.

The tree splits at 21 °C. Its prediction: 500 sales on cool days, 1,600 on warm days. For a 40 °C heatwave it still says 1,600: it cannot extrapolate.

5. Practice

  1. In the simulation, at depth 0 what does the tree predict, and what is that?
  2. Why does the test error rise again at large depths?
  3. A leaf holds one training row. What does it predict for every point that lands there?
Answers
  1. One flat line at the average sales of all training days: the simplest possible baseline.
  2. Deep trees make tiny leaves that follow the noise of single days (overfitting).
  3. That one row’s value, noise and all.
  • A regression tree predicts the average of the training values in each leaf.
  • Its splits are chosen to lower the weighted variance (squared error) of the target.
  • Predictions form a staircase: flat within a box, jumping at each threshold.
  • Trees cannot extrapolate beyond the range of the training values.

Try it on your own data: the regression workbench fits straight-line models to your table; compare its smooth predictions with a tree’s steps.

New words: regression tree, leaf mean, variance reduction, extrapolation, step function.

Check yourself

1A regression tree's leaf holds the values 10, 12 and 17. It predicts:
2What does a regression tree's split try to lower?
3A regression tree is trained on prices from $100k to $900k. For a huge house it predicts at most

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.