Home / Learn / Data & AI / ML foundations / Lesson 3.5
All lessons

Module 3: Linear regression: the first model · Lesson 3.5 · about 25 min

Measuring fit: residuals, RMSE and R²

How good is a regression model? RMSE and MAE give the typical error in real units, R² gives the share of variation explained, and a residual plot shows what the scores hide. Switch between four datasets and read the residuals.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 3 · about 25 minutes · read first: 3.2. You’ll be able to compute MAE, RMSE and R², choose the right one for a job, and diagnose a model from its residual plot.

Ask two questions of every model: how far off is it typically (MAE, RMSE, R²), and is it missing a pattern (the residual plot)?

1. Why this matters

A model always produces numbers, even a bad one, just as a broken clock always shows a time. Before anyone acts on its predictions they need two answers: how far off is it, typically, and is it missing something systematic? The scores answer the first. The residual plot answers the second, and catches problems the scores hide.

2. Three scores

With residuals ei=yi−yˆi on n test rows:

Score Formula Units Reads as
MAE (mean absolute error) 1n∑|ei| same as y “on average we are off by this much”
RMSE (root mean squared error) 1n∑ei2 same as y “a typical error”, with big misses counted more
R² (coefficient of determination) 1−∑ei2∑(yi−y‾)2 none “share of the variation the model explains”

Choose by what hurts: if one miss of 10 is much worse than five misses of 2 (stock levels, dosing), use RMSE. If every unit of error costs the same (delivery minutes), MAE is more honest. R² is good for comparing models on the same data, but it says nothing about whether the error is small enough for the job: an R² of 0.9 can still mean errors of $50,000 on house prices.

Always quote scores on test data. On training data they flatter.

Here are all three scores for one small test, step by step. True sandwich sales on four days were 3, 5, 7 and 9 (dozens); a model predicted 4, 5, 6 and 9:

The errors. y − ŷ = −1, 0, 1, 0.
MAE. Average the sizes: (1 + 0 + 1 + 0) ÷ 4 = 0.5 dozen. "We are typically off by half a dozen."
RMSE. Square, average, square root: √((1 + 0 + 1 + 0) ÷ 4) = √0.5 ≈ 0.71 dozen. A little bigger than MAE, as it always is unless every error is the same size.
The baseline. Always guessing the mean, 6, gives squared errors 9, 1, 1, 9: SST = 20. The model's squared errors add to SSE = 2.
R². 1 − 2 ÷ 20 = 0.9. The model removes 90 % of the baseline's squared error.

3. Reading residuals

Plot each residual against its prediction. A good model leaves residuals that look like random noise: centred on zero, the same spread everywhere, no shape.

Try it: read the residuals
RMSE (typical error)
MAE (average absolute error)
R²

Left: the data and the least-squares line. Right: each point's residual, the actual value minus the prediction. Good residuals look like random noise around zero. A curve means the model is missing a bend (Lesson 3.6); a funnel means its errors grow with size; one far point is an outlier pulling the fit. Scores alone can hide all three.
You see It means Try
Random cloud around 0 The model has caught the pattern Nothing; it is as good as this feature set allows
A curve (U or arch) The relationship bends Squared terms (Lesson 3.6), a log, or a tree model
A funnel (spread grows) Errors grow with the size of y Predict log(y), or report errors in percent
One point far out An outlier or a data error Check the row; consider capping or a robust loss
Groups or stripes A missing category Add the feature that separates the groups
Maths and principle

Principle: R² compares the model with the simplest baseline, always predicting the mean y‾.

Equations:

R2=1−SSESST=1−∑i(yi−yˆi)2∑i(yi−y‾)2

SSE is the model’s sum of squared errors; SST (“total”) is the baseline’s. On the training data of a least-squares line, R² equals the squared correlation r2 (Lesson 2.3). On test data R² can be negative: the model is worse than the mean.

Also, RMSE=SSE/n, so lowering one lowers the other, and MAE≤RMSE always.

Analogy: the baseline is a guess with no information. R² asks how much of the guess’s error the model removes.

Try it: y = 3, 5, 7, 9; predictions 4, 5, 6, 9. Find R². (y‾=6; SST = 9 + 1 + 1 + 9 = 20; SSE = 1 + 0 + 1 + 0 = 2; R² = 1 − 2/20 = 0.9.)

MAE
The average size of a miss, in the units of y. Every unit of error counts the same.
RMSE
The square root of the average squared miss. Big misses count extra.
SSE and SST
The model's total squared error, and the total squared error of always guessing the mean.
R² = 1 − SSE ÷ SST
The share of the baseline's error the model removes: 1 is perfect, 0 is no better than the mean, below 0 is worse.
Physical meaning
MAE is what an average customer feels: "it's usually about 9 minutes late". RMSE is what the complaints desk feels, because the one order that was 40 minutes late hurts far more than four that were 10 minutes late. R² is what the manager asks: "how much better is this than just guessing the usual?".

“An R² of 0.9 means the predictions are good enough.” R² compares with the mean, not with what the job needs. House prices that vary by $500,000 can give R² = 0.9 while the typical miss is still $100,000, too much for a buyer. Always quote an error in real units (MAE or RMSE) next to R².

4. Worked example

Made-up numbers. Two models predict delivery times (minutes) for the same 6 test orders, whose true times are 20, 25, 30, 35, 40, 90 (the last was a road closure).

errors MAE RMSE
Model A 3, −3, 3, −3, 3, −40 9.2 16.6
Model B 8, −8, 8, −8, 8, −20 10.0 11.0

Model A is closer on five of six orders and has the lower MAE; Model B has the lower RMSE because it misses the road closure by less. Which is better? If a 40-minute surprise loses a customer, the business should prefer B and RMSE. If every minute costs the same, A and MAE.

(The arithmetic for A: MAE = (3×5 + 40)/6 = 55/6 ≈ 9.2. RMSE = √((9×5 + 1600)/6) = √274.2 ≈ 16.6.)

5. Practice

  1. Errors of 1, −1, 1, −1. Find MAE and RMSE.
  2. A model’s test R² is −0.2. What does that tell you?
  3. A residual plot fans out to the right. Name one fix.
Answers
  1. Both are 1.
  2. It is worse than predicting the mean for everyone: back to the drawing board (often a sign of overfitting or a bug).
  3. Model log(y) instead of y, or judge errors in percent.

Try it on your own data: the regression workbench shows R² and a residual plot for any model you fit.

New words: MAE, RMSE, R², SSE and SST, residual plot, heteroscedasticity (a funnel: the error spread changes with the prediction).

  • MAE is the average size of the errors; RMSE counts big errors more; both are in the units of y, and MAE ≤ RMSE.
  • R² = 1 − SSE ÷ SST is the share of the mean-guess error the model removes; on test data it can be negative.
  • Choose the score by what hurts the business, and always quote it on test data.
  • A residual plot should look like random noise; curves, funnels and groups point to what the model is missing.

Check yourself

1Errors of 2, −2, 2, −2 and 10. Which is larger, MAE or RMSE?
2R² = 0 on the test set means:
3The residuals form a clear U shape. What should you try?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.