Module 3: Linear regression: the first model · Lesson 3.3 · about 30 min
Fitting by gradient descent
Instead of a formula, start with any line and keep stepping downhill on the error. Walk the error bowl of a straight line, see why the learning rate and scaling matter, and meet stochastic and mini-batch descent.
Module 3 · about 30 minutes · read first: 1.5, 3.2. You’ll be able to compute the gradient of the mean squared error, run gradient descent on a straight line, choose a learning rate, and explain why large models use mini-batches.
1. Why this matters
Least squares has a neat formula, but almost nothing else does. Think of finding the lowest point of a dark valley with only a torch that lights the ground at your feet: you can’t see the bottom, but you can always see which way is down. Logistic regression, neural networks and language models have no formula for their best weights. They are all trained the same way: start somewhere, measure the slope of the error, step downhill, repeat. Seeing this work on a straight line, where you can check the answer, is the best way to trust it on a billion-weight model.
2. The error as a landscape
For a line , every pair gives a mean squared error. Plot the error for every pair and you get a surface: a bowl, lowest at the least-squares line. Training is walking to the bottom of the bowl.
3. The gradient of the mean squared error
The loss is . By the chain rule (Lesson 1.5):
Read them in words: “average the errors (for ), or the errors weighted by their inputs (for )”. If predictions are too high on average, the bias gradient is positive and the step lowers . If they are too high mainly where is large, the slope gradient is positive and the step flattens the line.
The update for both, at the same time, with learning rate :
Here are the first steps on three points, (0, 1), (1, 3) and (2, 5), whose true line is y = 2x + 1, with learning rate 0.2:
4. Try it
Tap the error map to start from somewhere else.
Experiments:
- Scaled x, rate about 0.1: smooth steps into the centre of the round bowl in 20 to 40 steps.
- Scaled x, rate above 1: the steps overshoot further each time and the error blows up.
- Raw x: the bowl stretches into a narrow diagonal valley. Any rate big enough to move quickly along the valley throws you across it. Small rates crawl. This is why Lesson 2.4 said: scale your inputs before gradient descent.
5. Batches: how big models do it
The gradient above averages over all rows. With 10 rows that is instant; with 10 billion it is not. Three choices:
| Version | Rows per step | Steps | Notes |
|---|---|---|---|
| Batch gradient descent | all | smooth, exact | Slow per step on big data |
| Stochastic gradient descent (SGD) | 1 random row | very noisy | Cheap; the noise can help escape poor valleys |
| Mini-batch | e.g. 32 to 4,096 | a little noisy | What practically all neural networks use |
A pass through all the rows is called an epoch. (Think of a batch as tasting the whole pot of soup before adding salt, and a mini-batch as tasting a spoonful: a spoonful is quicker and nearly as informative.) Lesson 9.5 returns to batches, epochs and the improved step rules (momentum, Adam) used to train networks.
Maths and principle
Principle: repeat “compute the gradient of the loss, step a little against it” until the loss stops falling.
Equations: in vector form, with all weights ,
When it converges: for a bowl whose steepest curvature is , steps settle only if . For the mean squared error of a line with standardised x, the curvature in each direction is 2, so . Unscaled x spread from 1 to 9 has an average of about 30, so the steepest curvature is about 2 × 31 ≈ 62 and must stay below about 0.03. That rate crawls along the flattest direction, whose curvature is below 1.
Analogy: a skier who can only feel the slope underfoot. Short careful turns are safe but slow; long bold ones overshoot the bottom and climb the far side.
Try it: data (1, 3), (2, 5). Start at , , rate 0.1. (Predictions 0, 0; errors −3, −5. , . New , .)
- L(w, b)
- The loss: the mean squared error for one choice of weights. Every (w, b) pair has its own height.
- ∂L/∂w, ∂L/∂b
- How steeply the loss rises if you nudge w or b: the average error, weighted by x for w.
- η
- The learning rate: how far to move against the slope each time.
- Epoch
- One pass through all the training rows.
- Physical meaning
- A marble released on the side of a bowl rolls to the bottom without knowing where the bottom is; at each moment the tilt under it tells it which way to go. Gradient descent is that marble in slow motion, one measured step at a time, in a "bowl" that may have millions of directions.
“A bigger learning rate always trains faster.” Only up to a point. Past it, each step jumps over the bottom of the valley and lands higher up the far side, and the error grows instead of shrinking. With unscaled inputs the safe rate can be tiny. If the loss rises, the first fix is a smaller rate and scaled inputs.
6. Worked example
Made-up numbers. Three points (0, 1), (1, 3), (2, 5), so the true line is . Start at , , .
| step | w | b | predictions | errors ŷ − y | ∂L/∂w | ∂L/∂b |
|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0, 0, 0 | −1, −3, −5 | −8.67 | −6 |
| 1 | 1.73 | 1.2 | 1.2, 2.93, 4.67 | 0.2, −0.07, −0.33 | −0.49 | −0.13 |
| 2 | 1.83 | 1.23 | 1.23, 3.06, 4.89 | 0.23, 0.06, −0.11 | −0.11 | 0.12 |
The gradients for step 0: and . After a few dozen more steps, and settle at 2 and 1, the same answer as the least-squares formula.
7. Practice
- Your loss rises every step. What do you change?
- After 1,000 steps the loss is still falling slowly and steadily. What could help?
- Why might two runs of stochastic gradient descent end with slightly different weights?
Answers
- Lower the learning rate (and check the inputs are scaled).
- A larger learning rate, scaling the inputs, or a smarter step rule such as momentum or Adam.
- Each run picks rows in a different random order, so the paths differ. With a bowl-shaped loss they end near the same bottom; with bumpy losses (neural networks) they may end in different valleys.
New words: loss landscape, learning rate, batch, stochastic gradient descent, mini-batch, epoch, convergence.
- Gradient descent repeats: compute the slope of the loss for every weight, then step a little against it.
- For a line with mean squared error, the slopes are averages of the errors (for b) and of the errors times x (for w).
- Too large a learning rate overshoots and blows up; too small crawls. Scaling the inputs makes the bowl round and training easy.
- Big models use mini-batches: each step uses a small random handful of rows, and one pass through all rows is an epoch.
Check yourself
Sources
- Goodfellow, I., Bengio, Y. and Courville, A. (2016), Deep Learning, MIT Press, Sections 4.3 and 8.1 to 8.3 (gradient descent and SGD) https://www.deeplearningbook.org/
- Robbins, H. and Monro, S. (1951), A Stochastic Approximation Method, The Annals of Mathematical Statistics 22(3), 400-407 https://doi.org/10.1214/aoms/1177729586
- scikit-learn user guide: Stochastic Gradient Descent https://scikit-learn.org/stable/modules/sgd.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.