Module 3: Linear regression: the first model · Lesson 3.4 · about 30 min
Many inputs: multiple regression and matrices
With two inputs a linear model is a flat plane; with more it is a hyperplane. Stack the data in a matrix and one equation, the normal equation, gives every weight at once. Turn a 3-D plane and work a small example.
Module 3 · about 30 minutes · read first: 1.4, 3.2. You’ll be able to write multiple regression in matrix form, explain the normal equation, solve a two-input example, and spot multicollinearity.
1. Why this matters
Real predictions use many inputs: a price depends on size, age, location and more. Writing one formula per weight gets messy fast, like a shopping list with one line per apple instead of “6 apples”. Matrices let us write the model, the loss and the solution in one line each, whatever the number of inputs. They are also how computers (and graphics cards) actually do the sums, which is why the same notation carries straight into neural networks.
2. The picture: a plane
With two inputs, is a flat, tilted plane above the floor of . Least squares tilts and lifts it to sit as close as possible to all the points, measured straight up and down.
3. The model as a matrix product
Put the data in a table , one row per example, with a column of 1s at the front for the bias. Put the weights in a column , with the bias first:
Matrix times vector means: for each row, take the dot product of that row with (Lesson 1.4). So the whole model, for every example at once, is
4. The normal equation
The sum of squared errors is . Its gradient is , where (X transposed) is with rows and columns swapped. Setting it to zero gives the normal equation:
is a small square table (3 × 3 here, whatever is), so solving it is quick. For one input it reduces exactly to the slope and intercept formulas of Lesson 3.2.
In practice software solves the equation by elimination or a QR decomposition rather than inverting, which is faster and more accurate. For millions of inputs (text, images) even that is too slow, and gradient descent (Lesson 3.3) takes over.
Maths and principle
Principle: the least-squares weights make the residual vector perpendicular to every column of . That is what says: each column’s dot product with the residuals is zero.
Equations: ; .
Analogy: the shadow of a stick on the floor. The closest point on the floor to the stick’s tip is straight below it; the leftover (the residual) points straight up, at right angles to the floor. The predictions are the shadow of on the space the columns of can reach.
Try it: with only the column of 1s (no inputs), what does the normal equation give? ( and , so : the baseline that always predicts the mean.)
- X
- The design matrix: one row per example, a column of 1s for the bias, then one column per input.
- w
- The weights, bias first, as one column.
- Xw
- Every prediction at once: each row of X dotted with w.
- XᵀX w = Xᵀy
- The normal equation: a small square set of equations, one per weight, whose solution is the least-squares fit.
- Physical meaning
- The predictions are the shadow of the true values on the "floor" of everything the inputs can explain. The leftover, the residuals, stands straight up from that floor: no mix of the inputs can reduce it further. Each weight is the effect of one input with all the others held still, which is what lets the model untangle size from age.
5. Worked example
Made-up numbers. Four flats with size (in 10 m²), age (in 10 years) and price ($000):
| x₁ | x₂ | y |
|---|---|---|
| 5 | 1 | 250 |
| 6 | 3 | 250 |
| 8 | 2 | 340 |
| 9 | 4 | 340 |
Centring the columns first removes the bias from the sums (it is then minus the weights times the means). Means: , , . Centred values: : −2, −1, 1, 2; : −1.5, 0.5, −0.5, 1.5; : −45, −45, 45, 45.
The normal equation for the two weights uses sums of products of the centred columns:
- , ,
- ,
Subtracting: , so , then . Bias: .
The same solution as a sequence:
So price ≈ 36 × size(10 m²) − 18 × age(10 yr) + 88: each extra 10 m² adds $36,000, and each extra 10 years takes off $18,000. Check: the first flat gives 180 − 18 + 88 = 250, exactly its price (this tiny example fits perfectly). Notice that prices alone (two at 250, two at 340) hid the age effect: the bigger flats were also older.
6. Multicollinearity
When two inputs carry almost the same information (size in m² and in ft², or “years since built” and “year built”), is nearly impossible to invert. The model fits fine, but the split of credit between the twins is arbitrary: one weight may come out large and positive and the other large and negative, and both swing wildly with a few rows changed. Fixes: drop one twin, combine them, or add a ridge penalty (Lesson 3.8).
7. Practice
- A model has 5 inputs plus a bias. What size is ?
- In the worked example, predict a flat of 70 m² that is 30 years old.
- Why did the age weight only appear once size was in the model too?
Answers
- 6 × 6 (one row and column per weight, including the bias).
- 36 × 7 − 18 × 3 + 88 = 252 − 54 + 88 = 286, so about $286,000.
- Size and age were correlated (bigger flats were older), so on its own age’s effect was hidden behind size’s. Holding size fixed revealed it.
Try it on your own data: the regression workbench fits multiple regression to your CSV and gives each input’s VIF (variance inflation factor), which flags inputs that are too alike.
New words: multiple regression, design matrix, transpose, normal equation, hyperplane, multicollinearity.
- Multiple regression fits a flat plane (or hyperplane) through the data: ŷ = Xw, with a column of 1s for the bias.
- The least-squares weights solve the normal equation XᵀXw = Xᵀy, one equation per weight.
- Each weight is the effect of its input with the other inputs held still, which can reveal effects hidden in simple comparisons.
- When two inputs are near copies (multicollinearity), the fit is fine but their separate weights are unstable.
Check yourself
Sources
- James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Chapter 3 (linear regression) https://www.statlearning.com/
- Hastie, T., Tibshirani, R. and Friedman, J. (2009), The Elements of Statistical Learning, 2nd ed., Springer, Chapter 3 (linear methods for regression) https://hastie.su.domains/ElemStatLearn/
- Strang, G. (2016), Introduction to Linear Algebra, 5th ed., Wellesley-Cambridge Press, Chapter 4 (projections and least squares)
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.