Home / Learn / Data & AI / ML foundations / Lesson 3.4
All lessons

Module 3: Linear regression: the first model · Lesson 3.4 · about 30 min

Many inputs: multiple regression and matrices

With two inputs a linear model is a flat plane; with more it is a hyperplane. Stack the data in a matrix and one equation, the normal equation, gives every weight at once. Turn a 3-D plane and work a small example.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 3 · about 30 minutes · read first: 1.4, 3.2. You’ll be able to write multiple regression in matrix form, explain the normal equation, solve a two-input example, and spot multicollinearity.

With many inputs, the whole model is one matrix product, ŷ = Xw, and the best weights solve one small set of equations: the normal equation.

1. Why this matters

Real predictions use many inputs: a price depends on size, age, location and more. Writing one formula per weight gets messy fast, like a shopping list with one line per apple instead of “6 apples”. Matrices let us write the model, the loss and the solution in one line each, whatever the number of inputs. They are also how computers (and graphics cards) actually do the sums, which is why the same notation carries straight into neural networks.

2. The picture: a plane

With two inputs, yˆ=w1x1+w2x2+b is a flat, tilted plane above the floor of (x1,x2). Least squares tilts and lifts it to sit as close as possible to all the points, measured straight up and down.

Try it: turn the 3-D view
fitted model
Each dot is a flat: its position on the floor is its size and age, and its height is its price. The model price = w₁ × size + w₂ × age + b is a flat plane. Stems join each flat to the plane: those are the errors that least squares makes as small as possible. Turn the view until the plane is edge-on to see how well it fits. Made-up flats.

3. The model as a matrix product

Put the data in a table X, one row per example, with a column of 1s at the front for the bias. Put the weights in a column 𝐰, with the bias first:

(yˆ1yˆ2⋮yˆn)⏟𝐲ˆ=(1x11x121x21x22⋮⋮⋮1xn1xn2)⏟X(bw1w2)⏟𝐰

Matrix times vector means: for each row, take the dot product of that row with 𝐰 (Lesson 1.4). So the whole model, for every example at once, is

𝐲ˆ=X𝐰

4. The normal equation

The sum of squared errors is S(𝐰)=‖𝐲−X𝐰‖2. Its gradient is −2X𝖳(𝐲−X𝐰), where X𝖳 (X transposed) is X with rows and columns swapped. Setting it to zero gives the normal equation:

X𝖳X𝐰=X𝖳𝐲⇒𝐰=(X𝖳X)−1X𝖳𝐲

X𝖳X is a small square table (3 × 3 here, whatever n is), so solving it is quick. For one input it reduces exactly to the slope and intercept formulas of Lesson 3.2.

In practice software solves the equation by elimination or a QR decomposition rather than inverting, which is faster and more accurate. For millions of inputs (text, images) even that is too slow, and gradient descent (Lesson 3.3) takes over.

Maths and principle

Principle: the least-squares weights make the residual vector 𝐲−X𝐰 perpendicular to every column of X. That is what X𝖳(𝐲−X𝐰)=𝟎 says: each column’s dot product with the residuals is zero.

Equations: 𝐲ˆ=X𝐰; X𝖳X𝐰=X𝖳𝐲.

Analogy: the shadow of a stick on the floor. The closest point on the floor to the stick’s tip is straight below it; the leftover (the residual) points straight up, at right angles to the floor. The predictions are the shadow of 𝐲 on the space the columns of X can reach.

Try it: with only the column of 1s (no inputs), what does the normal equation give? (X𝖳X=n and X𝖳𝐲=∑yi, so b=y‾: the baseline that always predicts the mean.)

X
The design matrix: one row per example, a column of 1s for the bias, then one column per input.
w
The weights, bias first, as one column.
Xw
Every prediction at once: each row of X dotted with w.
XᵀX w = Xᵀy
The normal equation: a small square set of equations, one per weight, whose solution is the least-squares fit.
Physical meaning
The predictions are the shadow of the true values on the "floor" of everything the inputs can explain. The leftover, the residuals, stands straight up from that floor: no mix of the inputs can reduce it further. Each weight is the effect of one input with all the others held still, which is what lets the model untangle size from age.

5. Worked example

Made-up numbers. Four flats with size x1 (in 10 m²), age x2 (in 10 years) and price y ($000):

x₁ x₂ y
5 1 250
6 3 250
8 2 340
9 4 340

Centring the columns first removes the bias from the sums (it is then y‾ minus the weights times the means). Means: x‾1=7, x‾2=2.5, y‾=295. Centred values: x1: −2, −1, 1, 2; x2: −1.5, 0.5, −0.5, 1.5; y: −45, −45, 45, 45.

The normal equation for the two weights uses sums of products of the centred columns:

  • ∑x12=4+1+1+4=10, ∑x1x2=3−0.5−0.5+3=5, ∑x22=2.25+0.25+0.25+2.25=5
  • ∑x1y=90+45+45+90=270, ∑x2y=67.5−22.5−22.5+67.5=90
(10555)(w1w2)=(27090)that is10w1+5w2=270,5w1+5w2=90

Subtracting: 5w1=180, so w1=36, then w2=(90−180)/5=−18. Bias: b=295−36×7−(−18)×2.5=295−252+45=88.

The same solution as a sequence:

Centre the columns. Subtract the means (size 7, age 2.5, price 295), so the bias drops out and only two weights are left.
Build the small table. Sums of products of the centred columns: Σx₁² = 10, Σx₁x₂ = 5, Σx₂² = 5, Σx₁y = 270, Σx₂y = 90.
Two equations. 10w₁ + 5w₂ = 270 and 5w₁ + 5w₂ = 90. Subtract the second from the first: 5w₁ = 180, so w₁ = 36.
Back-substitute. 5 × 36 + 5w₂ = 90 gives w₂ = −18. Bias: 295 − 36 × 7 + 18 × 2.5 = 88.
Read it. +$36,000 per extra 10 m², −$18,000 per extra 10 years, holding the other still. Price alone hid the age effect, because the bigger flats were also older.

So price ≈ 36 × size(10 m²) − 18 × age(10 yr) + 88: each extra 10 m² adds $36,000, and each extra 10 years takes off $18,000. Check: the first flat gives 180 − 18 + 88 = 250, exactly its price (this tiny example fits perfectly). Notice that prices alone (two at 250, two at 340) hid the age effect: the bigger flats were also older.

6. Multicollinearity

When two inputs carry almost the same information (size in m² and in ft², or “years since built” and “year built”), X𝖳X is nearly impossible to invert. The model fits fine, but the split of credit between the twins is arbitrary: one weight may come out large and positive and the other large and negative, and both swing wildly with a few rows changed. Fixes: drop one twin, combine them, or add a ridge penalty (Lesson 3.8).

7. Practice

  1. A model has 5 inputs plus a bias. What size is X𝖳X?
  2. In the worked example, predict a flat of 70 m² that is 30 years old.
  3. Why did the age weight only appear once size was in the model too?
Answers
  1. 6 × 6 (one row and column per weight, including the bias).
  2. 36 × 7 − 18 × 3 + 88 = 252 − 54 + 88 = 286, so about $286,000.
  3. Size and age were correlated (bigger flats were older), so on its own age’s effect was hidden behind size’s. Holding size fixed revealed it.

Try it on your own data: the regression workbench fits multiple regression to your CSV and gives each input’s VIF (variance inflation factor), which flags inputs that are too alike.

New words: multiple regression, design matrix, transpose, normal equation, hyperplane, multicollinearity.

  • Multiple regression fits a flat plane (or hyperplane) through the data: ŷ = Xw, with a column of 1s for the bias.
  • The least-squares weights solve the normal equation XᵀXw = Xᵀy, one equation per weight.
  • Each weight is the effect of its input with the other inputs held still, which can reveal effects hidden in simple comparisons.
  • When two inputs are near copies (multicollinearity), the fit is fine but their separate weights are unstable.

Check yourself

1In ŷ = Xw, what is in each row of X?
2The normal equation is:
3Two inputs are almost copies of each other (size in m² and in ft²). What happens to their weights?

Sources

  • James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Chapter 3 (linear regression) https://www.statlearning.com/
  • Hastie, T., Tibshirani, R. and Friedman, J. (2009), The Elements of Statistical Learning, 2nd ed., Springer, Chapter 3 (linear methods for regression) https://hastie.su.domains/ElemStatLearn/
  • Strang, G. (2016), Introduction to Linear Algebra, 5th ed., Wellesley-Cambridge Press, Chapter 4 (projections and least squares)

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.