Module 9: Neural networks · Lesson 9.6 · about 25 min
Images: convolution and filters
A picture is a grid of numbers. A convolution slides a small filter over it and scores how well each patch matches, wherever it is. Slide edge and blur filters over a pixel picture and see how CNNs see.
Module 9 · about 25 minutes · read first: 1.4, 9.3. You’ll be able to compute a convolution by hand, explain why filters are shared across the image, and describe how a CNN builds up from edges to objects.
1. Why this matters
Convolutional neural networks (CNNs) made computers see. They read cheques in the 1990s (LeCun’s LeNet), won the ImageNet contest in 2012 (AlexNet), and now check X-rays, sort parcels and help steer cars. The same sliding-filter idea is used for sound, sensor signals and text.
2. A picture is a grid of numbers
A black-and-white picture is a grid of brightness values: 0 for black, bigger for brighter. A colour picture is three grids (red, green, blue), called channels. A small 256 × 256 colour photo is 196,608 numbers. A fully connected layer of 1,000 neurons on it would need nearly 200 million weights, and would have to learn “an eye at the top left” and “an eye at the top right” separately.
3. Convolution
A filter (or kernel) is a small grid of weights, say 3 × 3. Slide it over the picture. At each position, multiply each of the 9 pixels under it by the matching weight and add them up. The results form a new grid, the feature map, which is bright where the patch looks like the filter’s pattern.
It’s like checking a page for the letter “e” with a cut-out shaped like an “e”: slide it along, and mark every spot where it lines up.
Watch it happen
A strip of 5 pixels, dark on the left and bright on the right: 0 0 0 9 9. The filter is [1, 0, −1]: “left minus right”.
Position 1. The filter covers 0, 0, 0. Sum: 0 × 1 + 0 × 0 + 0 × (−1) = 0. Flat: no edge here.
Position 2. It covers 0, 0, 9. Sum: 0 + 0 − 9 = −9. A strong response: brightness jumps under the filter.
Position 3. It covers 0, 9, 9. Sum: 0 + 0 − 9 = −9 again.
The feature map. 0, −9, −9: zero on the flat part, strong where dark meets bright. Flip the filter's signs and the responses turn positive. The same 3 weights did the whole job.
Tap the output (right) to see the sum for that pixel; its 3×3 window is outlined on the picture.
The trick is weight sharing: the same 9 weights are used at every position, so a pattern is found wherever it appears, with very few weights to learn.
- I(r, c)
- the image: the brightness of the pixel at row r, column c
- K(i, j)
- the filter: a small grid of weights, the pattern being looked for
- S(r, c)
- the feature map: how strongly the patch at (r, c) matches the pattern
- k
- the filter's width, such as 3 for a 3 × 3 filter
- Pooling
- shrinking a map by keeping, say, the biggest value in each 2 × 2 square
- Physical meaning
- Your eye does something similar: cells in the retina and visual cortex respond to small edges and spots in one patch of what you see, and later areas combine them into shapes and faces. A CNN learns its own version of those detectors from examples.
4. A CNN
A typical image network repeats:
- Convolution layer: many filters (say 32), each producing its own feature map, followed by ReLU.
- Pooling: shrink each map, for example by keeping the largest value in each 2 × 2 square. The network becomes less sensitive to exact positions, and the maps get smaller.
After several rounds, the small, deep stack of maps is flattened and fed to fully connected layers for the final answer. The filter weights are learned by backpropagation, not designed. Early layers end up as edge, colour and texture detectors; middle layers as parts (wheels, eyes, letters); late layers as whole objects.
“Engineers design the filters: one for edges, one for eyes, one for wheels.” In a CNN the filters start as random numbers. Backpropagation shapes them, picture after picture, into whatever detectors help the final answer. That they end up looking like edge detectors in the first layer is a discovery, not a design. Hand-made filters, like the ones in this lesson’s simulation, are for teaching.
Maths and principle
Principle: look for the same small pattern everywhere, with one shared set of weights.
Convolution (as used in deep learning, strictly a cross-correlation) of image with a filter :
Output size for an image with no padding and stride 1: . Padding the border with zeros keeps the size .
Weights in a layer: plus biases, however big the image.
Analogy: reading a page with a small magnifying glass shaped like the letter “e”, noting every spot where the glass lines up with an “e”.
Try it: apply the filter [[1, 0], [0, −1]] to the 3 × 3 image [[1, 2, 3], [4, 5, 6], [7, 8, 9]]. (Each output is top-left minus bottom-right: 1 − 5, 2 − 6, 4 − 8, 5 − 9, all −4. The image is a smooth ramp, so this diagonal-change detector sees the same change everywhere.)
5. Worked example
A 5-pixel-wide strip, dark on the left and bright on the right: 0 0 0 9 9. The 1-D edge filter [1, 0, −1] gives:
- position 1: 0 × 1 + 0 × 0 + 0 × (−1) = 0
- position 2: 0 + 0 − 9 = −9
- position 3: 0 + 0 − 9 = −9
Strong (negative) responses where the brightness jumps; zero on the flat part. Flip the filter’s sign and the response is positive. A first-layer filter in a CNN, trained on photos, typically learns something very like this, in two dimensions and at many angles.
6. Practice
- In the simulation, where does the vertical-edge filter respond strongly, and where is it zero?
- How many weights does a layer of 32 filters of size 3 × 3 on a colour image have?
- Why does pooling help a network recognise a shifted object?
Answers
- Along the walls and the door’s sides (left-right changes in brightness); zero on flat areas and along horizontal edges.
- 3 × 3 × 3 × 32 = 864 weights, plus 32 biases.
- Keeping only the strongest response in each small square means a pattern moved by a pixel or two gives almost the same pooled map.
Try it on your own data: the neural network toys lets you draw small black-and-white patterns and see how a network stores and recalls them.
New words: convolution, filter (kernel), feature map, weight sharing, stride, padding, pooling, CNN.
- A picture is a grid of numbers per channel; a filter is a small grid of weights slid over it, giving a feature map that is strong where the patch matches.
- Weight sharing means one filter finds its pattern anywhere in the image, with very few weights: k × k × c_in × c_out plus c_out biases.
- A CNN repeats convolution, ReLU and pooling, then uses fully connected layers for the answer.
- The filters are learned by backpropagation: early layers become edge and colour detectors, deeper ones detect parts and objects.
Check yourself
Sources
- LeCun, Bottou, Bengio and Haffner (1998), Gradient-based learning applied to document recognition, Proceedings of the IEEE 86(11), 2278-2324 https://doi.org/10.1109/5.726791
- Krizhevsky, Sutskever and Hinton (2012), ImageNet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems 25
- Goodfellow, Bengio and Courville (2016), Deep Learning, MIT Press, Chapter 9 (convolutional networks) https://www.deeplearningbook.org/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.