Home / Learn / Data & AI / ML foundations / Lesson 1.5
All lessons

Module 1: Data, models and the maths you need · Lesson 1.5 · about 25 min

Derivatives and the gradient: which way is downhill

The derivative is the slope of a curve at one point. Step against it and you go downhill. That single rule, gradient descent, is how almost every model learns. Roll a ball down a curve and change the step size.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 1 · about 25 minutes · read first: 1.3. You’ll be able to explain what a derivative is, work out simple ones, say what a gradient is, and run gradient descent by hand.

The derivative tells you which way is uphill and how steep it is; step the other way, again and again, and you walk down to the smallest error.

1. Why this matters

Training a model means making its error as small as possible. The error depends on the parameters (the dials), so you can draw it as a curve or a landscape. Making the error small means finding the bottom of a valley. The derivative tells you which way is downhill from wherever you stand. Linear regression (Lesson 3.3), logistic regression, neural networks and language models are all trained by walking downhill this way.

2. The derivative is a slope at one point

A curve like f(x)=x2 has no single slope: it is gentle near 0 and steep further out. The derivative f′(x) (say “f prime of x”) is the slope of the curve at one point. Picture a ruler laid so it just touches the curve at that point without cutting it: that is the tangent, and its slope is the derivative. On a road, it is the steepness right under your wheels, not the average over the whole hill.

You can estimate it with two points very close together:

f′(x)≈f(x+h)−f(x)hfor a tiny h

This is just rise ÷ run (Lesson 1.3) over a tiny run. As h shrinks to zero, the estimate becomes exact.

3. Rules you’ll use

You don’t need to compute limits each time. A few rules cover this course:

Function Derivative Example
c (a constant) 0 f(x)=7 → 0
xn nxn−1 x2 → 2x; x3 → 3x2
ag(x) ag′(x) 5x2 → 10x
g(x)+h(x) g′(x)+h′(x) x2+3x → 2x+3
ex ex used in Lesson 4.2
ln⁡x 1/x used in Lesson 4.3
g(h(x)) (chain rule) g′(h(x))⋅h′(x) (x−2)2 → 2(x−2)

The chain rule in the last row says: for a function inside a function, multiply the outer slope by the inner slope. It is the heart of backpropagation in neural networks (Lesson 9.4).

4. Walking downhill: gradient descent

Here is the idea that trains models. Stand at some x. Look at the slope. If it is positive, the curve rises to the right, so step left. If negative, step right. Make the step proportional to the slope, so you take big steps where it is steep and small ones near the bottom:

xnew=x−ηf′(x)

η (the Greek letter eta) is the learning rate, the step size. Here is one small walk, three steps long, on the curve f(x)=(x−2)2+1, whose bottom is at x=2. Its slope is f′(x)=2(x−2), and we use rate 0.25:

Start at x = 0. The slope there is 2 × (0 − 2) = −4: the curve falls to the right. Step: x − 0.25 × (−4) = 0 + 1 = 1. The error drops from 5 to 2.
At x = 1. Slope 2 × (1 − 2) = −2, gentler now. Step: 1 + 0.25 × 2 = 1.5. Error 2 → 1.25.
At x = 1.5. Slope −1. Step: 1.5 + 0.25 = 1.75. Error 1.25 → 1.0625.
The pattern. Each step halves the gap to the bottom (2 → 1 → 0.5 → 0.25). Steps shrink by themselves, because the slope shrinks near the bottom. No one has to tell the ball to slow down.

Now try it with your own rate and curve:

Try it: roll downhill with the slope
x
slope f′(x)
height f(x)
steps

Tap or click the curve to drop the ball somewhere else.

The slope (derivative) says which way is uphill and how steep it is. Moving a small step against the slope always goes downhill: new x = x − rate × slope. This one rule, repeated, is how almost every model in this course learns. Try a rate of 1.0 on the bowl, and the bumpy curve from different starting points.

Experiments worth doing:

  • Rate 0.2 on the bowl: the ball slides smoothly to x = 2, taking smaller and smaller steps.
  • Rate 0.95: each step jumps across the valley and back, zig-zagging slowly in.
  • Rate 1.05: each jump lands higher than the last, and the ball flies off. Too large a learning rate makes training blow up.
  • Bumpy curve, start at the right: the ball settles in the right-hand valley, though the left one is deeper. Gradient descent finds a bottom, not always the lowest one.

“Gradient descent always finds the lowest point.” It only feels the ground under its feet, so it stops in the first valley it rolls into, a local minimum. For the bowl-shaped errors of linear and logistic regression there is only one valley, so that is fine. For neural networks there are many, and in practice the valleys it finds are usually good enough, but there is no guarantee of the very lowest.

5. Many parameters: the gradient

Real models have many parameters, so the error is a surface over many directions, not a curve. For each parameter you take the partial derivative: the slope in that one direction with all the others held still. Listed together they form the gradient, written ∇f (say “grad f”):

∇f(w1,w2)=(∂f∂w1, ∂f∂w2)

The gradient is a vector (Lesson 1.4) that points uphill, the steepest way up. Gradient descent steps the opposite way, for all parameters at once:

𝐰new=𝐰−η∇f(𝐰)

A language model does exactly this with billions of parameters at every training step.

f′(x)
The slope of the curve right at x: how fast the error changes if you nudge x a tiny bit.
∇f (gradient)
The list of slopes, one per parameter. Together they point straight uphill.
η (learning rate)
How big a step to take. Too small is slow; too big overshoots.
x − η f′(x)
The new setting: the old one, moved a little against the slope.
Physical meaning
A walker in thick fog on a hillside. They can't see the valley, but they can feel which way the ground tilts under their feet. They take a step downhill, feel again, and repeat. Steep ground gives long strides; nearly flat ground near the bottom gives tiny shuffles. That is all "training" is.
Maths and principle

Principle: the derivative is the slope of a function at a point. Moving a small step against it lowers the function.

Equation: f′(x)=limh→0⁡f(x+h)−f(x)h; gradient descent x←x−ηf′(x).

Why it works: near x, the curve is almost its tangent line, so f(x−ηf′(x))≈f(x)−ηf′(x)2. Since f′(x)2 is never negative, a small step always lowers f (unless the slope is already 0, at the bottom).

Analogy: walking down a hill in thick fog. You can’t see the valley, but you can feel which way the ground slopes under your feet, so you step that way and feel again.

Try it: f(x)=(x−2)2+1, start at x=0, rate 0.25. The slope is 2(x−2)=−4, so xnew=0−0.25×(−4)=1. Next: slope −2, x=1.5. Then 1.75, 1.875, … halving the gap to 2 each step.

6. Worked example

Made-up numbers. A café sets one price p (in dollars) for a sandwich. Its owner estimates that profit is roughly P(p)=−20(p−7)2+500 dollars a day: too cheap loses money on each sandwich, too dear loses customers. We want the top of the hill, so we walk up the slope (gradient ascent, or descent on −P).

The derivative is P′(p)=−40(p−7). Start at p=5 with rate 0.01:

  • P′(5)=−40×(−2)=80, so p=5+0.01×80=5.8.
  • P′(5.8)=48, so p=6.28.
  • P′(6.28)=28.8, so p=6.568. Each step closes 40 % of the remaining gap, heading for $7.

Setting the derivative to zero gets there in one go: −40(p−7)=0 gives p=7. For a simple curve, solving f′(x)=0 is quickest. For a model with a million parameters there is no such shortcut, and stepping downhill is the practical way.

7. Practice

  1. Find the derivative of f(x)=3x2−4x+1, and the x where the slope is zero.
  2. From x=4 on f(x)=x2, take one gradient descent step with rate 0.1.
  3. Use the chain rule on f(x)=(3x+1)2.
Answers
  1. f′(x)=6x−4; zero at x=2/3.
  2. Slope 2×4=8; x=4−0.1×8=3.2.
  3. Outer 2(3x+1) times inner 3: f′(x)=6(3x+1).

Try it yourself: plot f(x) and the derivative you worked out, such as x^2 and 2*x, together in the graphing calculator, and check that the second is zero exactly where the first bottoms out.

New words: derivative, tangent, chain rule, learning rate, gradient descent, partial derivative, gradient, local minimum (a valley that isn’t the lowest).

  • The derivative is the slope of a curve at one point: rise over a tiny run.
  • Gradient descent: new x = x − η × slope. Step against the slope and the error goes down.
  • The learning rate η sets the step size; too large and training bounces or blows up.
  • With many parameters the slopes form the gradient, a vector that points uphill; descent steps the opposite way.
  • Gradient descent finds a valley, not always the lowest one.

Check yourself

1For f(x) = x², what is the slope (derivative) at x = 3?
2At x = 3 the slope of a loss curve is +6. Which way should x move to lower the loss?
3What happens if the learning rate is far too large?

Sources

  • Goodfellow, I., Bengio, Y. and Courville, A. (2016), Deep Learning, MIT Press, Section 4.3 (gradient-based optimisation) https://www.deeplearningbook.org/
  • Strang, G. (2019), Linear Algebra and Learning from Data, Wellesley-Cambridge Press, Part VI (optimisation and gradient descent)
  • Cauchy, A.-L. (1847), Méthode générale pour la résolution des systèmes d'équations simultanées, Comptes rendus de l'Académie des sciences 25, 536-538 (the first description of gradient descent)

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.