Module 9: Neural networks · Lesson 9.4 · about 30 min
Backpropagation: the chain rule, step by step
To train a network we need the slope of the loss for every weight. Backpropagation gets all of them in one backward pass, by the chain rule. Follow a tiny network forward and backward with real numbers.
Module 9 · about 30 minutes · read first: 1.5, 4.3, 9.3. You’ll be able to run a forward and a backward pass by hand on a small network, explain the chain rule as multiplying local slopes, and say why backpropagation is fast.
1. Why this matters
Backpropagation (popularised by Rumelhart, Hinton and Williams in 1986) is how every neural network learns, from a 17-weight toy to a language model with hundreds of billions of weights. Libraries do it automatically (“autograd”), but knowing what it computes explains vanishing gradients, why ReLU helps, why scaling inputs matters, and what “training” actually is.
2. The problem
Gradient descent (Lesson 3.3) needs for every weight . Read it as “how much the loss would change if were nudged a tiny bit”. Nudging each weight one at a time and rerunning the network would take one forward pass per weight: billions of passes for a large model. Backpropagation gets every gradient with one forward and one backward pass.
3. The chain rule
The loss depends on a weight through a chain of steps. Think of gears: turn the first gear, and the last one turns by the product of all the gear ratios in between. If depends on , which depends on , which depends on , then
Each factor is a local slope, easy to compute at its own step. Backpropagation works from the loss backwards, multiplying in one local slope per step, and reuses each partial product for every weight further back. Nothing is computed twice.
Watch it happen
The network in the picture: input x = 2, w₁ = 0.5, a ReLU, w₂ = 1.5, bias b = −0.5, a sigmoid, and the true answer y = 1.
Forward, part 1. z₁ = w₁x = 0.5 × 2 = 1. The ReLU passes it on: h = 1.
Forward, part 2. z₂ = w₂h + b = 1.5 − 0.5 = 1. p = sigmoid(1) = 0.731. The loss is −ln 0.731 = 0.313. Keep every one of these numbers: the backward pass needs them.
Error at the output. With a sigmoid and log-loss the error at z₂ is simply p − y = −0.269. So ∂L/∂w₂ = −0.269 × h = −0.269, and ∂L/∂b = −0.269.
Back through w₂. How much is h to blame? Its error is −0.269 × w₂ = −0.269 × 1.5 = −0.403.
Back through the ReLU to w₁. ReLU's slope at z₁ = 1 is 1, so z₁'s error is also −0.403. The weight w₁ multiplied x = 2, so ∂L/∂w₁ = −0.403 × 2 = −0.807.
Update. Step against each gradient with η = 0.5: w₁ = 0.903, w₂ = 1.634, b = −0.366. Run forward again: p = 0.930 and the loss falls from 0.313 to 0.073.
- ∂L/∂w
- the gradient: how much the loss changes per tiny nudge of weight w; its sign says which way to move
- δ (delta, error)
- the blame arriving at a neuron's weighted sum z
- p − y
- the blame at the output: how far the prediction is from the truth
- g′(z)
- the activation's slope, which the blame is multiplied by on its way back through each neuron
- a (input to the weight)
- a weight's gradient is the blame at its neuron times the input it multiplied
- Physical meaning
- A factory tracing a faulty product. Start at the end, ask each station how much its output added to the fault, and pass the blame back along the line in proportion. Every station then adjusts its machine a little, and the next product comes out better.
4. The recipe
- Forward: compute and keep every layer’s weighted sums and outputs .
- Output error: with a sigmoid and log-loss, .
- Backward, layer by layer: a hidden neuron’s error is the sum of the errors of the neurons it feeds, each times the connecting weight, times its own activation’s slope at its .
- Gradients: each weight’s gradient is the error at the neuron it feeds times the input it multiplied; each bias’s gradient is that neuron’s error.
- Update: for every weight.
“Backpropagation is how a network learns.” Backpropagation only measures: it works out every weight’s gradient. The learning, the actual change of weights, is done by gradient descent or a cousin such as Adam (Lesson 9.5). The two always work as a pair, but they are different jobs: one says “which way and how much is each weight to blame”, the other takes the step.
Maths and principle
Principle: the chain rule, applied from the output back, sharing work between weights.
Errors (deltas) for layer , with meaning multiply element by element:
Gradients:
Why : for log-loss and , and ; the product is .
Analogy: a factory finding who is responsible for a fault. Start at the faulty product, ask each station how much its output contributed, and pass the blame back along the line in proportion.
Try it: a neuron computes with , and the error arriving at is . Gradient for ? New with if ? (; .)
5. Worked example
The network in the picture: input ; with ; ; with , ; ; the true answer is .
Forward: , , , , .
Backward:
- .
- ; .
- .
- .
- .
Update with : , , . Forward again: , , , loss 0.073. One step cut the loss by more than three quarters.
6. Practice
- In the worked example, why is ’s gradient twice ’s error?
- If had been −1, what would ’s gradient be, and why?
- Why is one backward pass enough for all the weights?
Answers
- Because multiplied the input : its gradient is the error at times 2.
- 0: ReLU’s slope is 0 for negative inputs, so no signal reaches (a “dead” neuron for this example).
- Each step’s error is computed once and reused by all the weights behind it; the work is about the same as one forward pass.
Try it on your own data: the simulation in Lesson 9.3 runs exactly this recipe thousands of times a second; watch its log-loss fall.
New words: backpropagation, chain rule, forward pass, backward pass, error (delta), autograd, gradient.
- Gradient descent needs ∂L/∂w for every weight; backpropagation gets them all with one forward and one backward pass.
- The chain rule multiplies local slopes along the path from the loss back to each weight; shared parts are computed once.
- With a sigmoid output and log-loss the output error is p − y; a hidden neuron’s error is the weighted sum of the errors it feeds, times its activation’s slope.
- A weight’s gradient is the error at its neuron times the input it multiplied; gradient descent then steps each weight against its gradient.
Check yourself
Sources
- Rumelhart, Hinton and Williams (1986), Learning representations by back-propagating errors, Nature 323, 533-536 https://doi.org/10.1038/323533a0
- Goodfellow, Bengio and Courville (2016), Deep Learning, MIT Press, Section 6.5 (back-propagation) https://www.deeplearningbook.org/
- PyTorch documentation, A gentle introduction to torch.autograd https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.