Home / Learn / Data & AI / ML foundations / Lesson 9.4
All lessons

Module 9: Neural networks · Lesson 9.4 · about 30 min

Backpropagation: the chain rule, step by step

To train a network we need the slope of the loss for every weight. Backpropagation gets all of them in one backward pass, by the chain rule. Follow a tiny network forward and backward with real numbers.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 9 · about 30 minutes · read first: 1.5, 4.3, 9.3. You’ll be able to run a forward and a backward pass by hand on a small network, explain the chain rule as multiplying local slopes, and say why backpropagation is fast.

To learn, every weight needs to know how much it is to blame for the error; backpropagation passes the blame backwards through the network, one step at a time, in a single sweep.

1. Why this matters

Backpropagation (popularised by Rumelhart, Hinton and Williams in 1986) is how every neural network learns, from a 17-weight toy to a language model with hundreds of billions of weights. Libraries do it automatically (“autograd”), but knowing what it computes explains vanishing gradients, why ReLU helps, why scaling inputs matters, and what “training” actually is.

2. The problem

Gradient descent (Lesson 3.3) needs ∂L/∂w for every weight w. Read it as “how much the loss L would change if w were nudged a tiny bit”. Nudging each weight one at a time and rerunning the network would take one forward pass per weight: billions of passes for a large model. Backpropagation gets every gradient with one forward and one backward pass.

3. The chain rule

The loss depends on a weight through a chain of steps. Think of gears: turn the first gear, and the last one turns by the product of all the gear ratios in between. If L depends on p, which depends on z, which depends on w, then

∂L∂w=∂L∂p⋅∂p∂z⋅∂z∂w

Each factor is a local slope, easy to compute at its own step. Backpropagation works from the loss backwards, multiplying in one local slope per step, and reuses each partial product for every weight further back. Nothing is computed twice.

Picture: forward, then backward A chain of five boxes: input x = 2, z1 = w1 x = 1, h = ReLU(z1) = 1, z2 = w2 h + b = 1, p = sigmoid(z2) = 0.731, loss 0.313. Below, the gradients flowing back: dL/dz2 = −0.269, dL/dh = −0.403, dL/dz1 = −0.403, and the weight gradients dL/dw2 = −0.269, dL/db = −0.269, dL/dw1 = −0.807. Forward: compute values, left to right x 2 z₁ = w₁x 1 h = ReLU 1 z₂ = w₂h + b 1 p = σ(z₂) 0.731 loss L = −ln p = 0.313 (the true answer is y = 1) Backward: gradients, right to left −0.403 −0.403 −0.269 ∂L/∂z₁∂L/∂h∂L/∂z₂ = p − y Weight gradients (gradient × input) ∂L/∂w₂ = ∂L/∂z₂ × h = −0.269 ∂L/∂b = −0.269 ∂L/∂w₁ = ∂L/∂z₁ × x = −0.807 All negative: raising each weight would lower the loss. fibub.com
The forward pass works out each value from the one before. The backward pass starts from the loss and multiplies by one local slope per step (the chain rule): from z₂ to h it multiplies by w₂ = 1.5; through the ReLU it multiplies by 1, because z₁ is positive. Each weight's gradient is the gradient arriving at its box times the input it multiplied. One forward and one backward pass give every gradient at once, however many weights there are.

Watch it happen

The network in the picture: input x = 2, w₁ = 0.5, a ReLU, w₂ = 1.5, bias b = −0.5, a sigmoid, and the true answer y = 1.

Forward, part 1. z₁ = w₁x = 0.5 × 2 = 1. The ReLU passes it on: h = 1.

Forward, part 2. z₂ = w₂h + b = 1.5 − 0.5 = 1. p = sigmoid(1) = 0.731. The loss is −ln 0.731 = 0.313. Keep every one of these numbers: the backward pass needs them.

Error at the output. With a sigmoid and log-loss the error at z₂ is simply p − y = −0.269. So ∂L/∂w₂ = −0.269 × h = −0.269, and ∂L/∂b = −0.269.

Back through w₂. How much is h to blame? Its error is −0.269 × w₂ = −0.269 × 1.5 = −0.403.

Back through the ReLU to w₁. ReLU's slope at z₁ = 1 is 1, so z₁'s error is also −0.403. The weight w₁ multiplied x = 2, so ∂L/∂w₁ = −0.403 × 2 = −0.807.

Update. Step against each gradient with η = 0.5: w₁ = 0.903, w₂ = 1.634, b = −0.366. Run forward again: p = 0.930 and the loss falls from 0.313 to 0.073.

∂L/∂w
the gradient: how much the loss changes per tiny nudge of weight w; its sign says which way to move
δ (delta, error)
the blame arriving at a neuron's weighted sum z
p − y
the blame at the output: how far the prediction is from the truth
g′(z)
the activation's slope, which the blame is multiplied by on its way back through each neuron
a (input to the weight)
a weight's gradient is the blame at its neuron times the input it multiplied
Physical meaning
A factory tracing a faulty product. Start at the end, ask each station how much its output added to the fault, and pass the blame back along the line in proportion. Every station then adjusts its machine a little, and the next product comes out better.

4. The recipe

  1. Forward: compute and keep every layer’s weighted sums z and outputs a.
  2. Output error: with a sigmoid and log-loss, δout=p−y.
  3. Backward, layer by layer: a hidden neuron’s error is the sum of the errors of the neurons it feeds, each times the connecting weight, times its own activation’s slope at its z.
  4. Gradients: each weight’s gradient is the error at the neuron it feeds times the input it multiplied; each bias’s gradient is that neuron’s error.
  5. Update: w←w−η∂L/∂w for every weight.

“Backpropagation is how a network learns.” Backpropagation only measures: it works out every weight’s gradient. The learning, the actual change of weights, is done by gradient descent or a cousin such as Adam (Lesson 9.5). The two always work as a pair, but they are different jobs: one says “which way and how much is each weight to blame”, the other takes the step.

Maths and principle

Principle: the chain rule, applied from the output back, sharing work between weights.

Errors (deltas) for layer l, with ⊙ meaning multiply element by element:

𝜹(L)=𝐩−𝐲,𝜹(l)=(𝐖(l+1)⊤𝜹(l+1))⊙g′(𝐳(l))

Gradients:

∂L∂Wij(l)=δi(l)aj(l−1),∂L∂bi(l)=δi(l)

Why p−y: for log-loss L=−[yln⁡p+(1−y)ln⁡(1−p)] and p=σ(z), ∂L/∂p=(p−y)/(p(1−p)) and ∂p/∂z=p(1−p); the product is p−y.

Analogy: a factory finding who is responsible for a fault. Start at the faulty product, ask each station how much its output contributed, and pass the blame back along the line in proportion.

Try it: a neuron computes z=wa with a=3, and the error arriving at z is ∂L/∂z=−0.2. Gradient for w? New w with η=0.5 if w=1? (−0.6; 1+0.3=1.3.)

5. Worked example

The network in the picture: input x=2; z1=w1x with w1=0.5; h=ReLU⁡(z1); z2=w2h+b with w2=1.5, b=−0.5; p=σ(z2); the true answer is y=1.

Forward: z1=1, h=1, z2=1.5−0.5=1, p=σ(1)=0.731, L=−ln⁡0.731=0.313.

Backward:

  • ∂L/∂z2=p−y=−0.269.
  • ∂L/∂w2=−0.269×h=−0.269; ∂L/∂b=−0.269.
  • ∂L/∂h=−0.269×w2=−0.403.
  • ∂L/∂z1=−0.403×ReLU′⁡(1)=−0.403×1=−0.403.
  • ∂L/∂w1=−0.403×x=−0.807.

Update with η=0.5: w1=0.5+0.403=0.903, w2=1.5+0.134=1.634, b=−0.5+0.134=−0.366. Forward again: z1=1.807, z2=1.634×1.807−0.366=2.586, p=0.930, loss 0.073. One step cut the loss by more than three quarters.

6. Practice

  1. In the worked example, why is w1’s gradient twice h’s error?
  2. If z1 had been −1, what would w1’s gradient be, and why?
  3. Why is one backward pass enough for all the weights?
Answers
  1. Because w1 multiplied the input x=2: its gradient is the error at z1 times 2.
  2. 0: ReLU’s slope is 0 for negative inputs, so no signal reaches w1 (a “dead” neuron for this example).
  3. Each step’s error is computed once and reused by all the weights behind it; the work is about the same as one forward pass.

Try it on your own data: the simulation in Lesson 9.3 runs exactly this recipe thousands of times a second; watch its log-loss fall.

New words: backpropagation, chain rule, forward pass, backward pass, error (delta), autograd, gradient.

  • Gradient descent needs ∂L/∂w for every weight; backpropagation gets them all with one forward and one backward pass.
  • The chain rule multiplies local slopes along the path from the loss back to each weight; shared parts are computed once.
  • With a sigmoid output and log-loss the output error is p − y; a hidden neuron’s error is the weighted sum of the errors it feeds, times its activation’s slope.
  • A weight’s gradient is the error at its neuron times the input it multiplied; gradient descent then steps each weight against its gradient.

Check yourself

1Backpropagation computes:
2With a sigmoid output and log-loss, the error at the output's weighted sum is:
3A weight's gradient equals:

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.