Module 9: Neural networks · Lesson 9.3 · about 25 min
Layers: why stacking neurons can draw any shape
A hidden layer turns the inputs into new features; the output layer is logistic regression on those features. Add neurons and layers and watch a network learn a ring, XOR and two moons live in your browser.
Module 9 · about 25 minutes · read first: 4.4, 9.2. You’ll be able to describe a fully connected network, count its parameters, and explain how hidden layers create the features that make curved boundaries possible.
1. Why this matters
In Lesson 4.4 we bent a straight boundary into a circle by adding squared features by hand. A neural network finds such features itself. That is the whole trick behind deep learning: networks that learn their own features from raw pixels, sounds or words, where nobody could design the features by hand.
2. Layers of neurons
A fully connected (dense) network is built in layers, like a relay team where each runner hands over to every runner of the next leg:
- The input layer holds the columns .
- Each hidden layer has several neurons. “Hidden” just means you never see its values in the data. Every neuron takes all the outputs of the layer before, makes a weighted sum, adds its bias, and applies an activation (Lesson 9.2).
- The output layer makes the prediction: one sigmoid neuron for yes/no, one per class with softmax for several classes, one plain neuron for a number.
Each hidden neuron draws one soft line across the space (where its weighted sum is 0) and reports which side, and how far, a point is. The next layer combines those reports, so a handful of lines can be folded into a ring, a corner, or a curve.
Watch it happen: two hidden neurons solve XOR
Neuron A asks “is at least one input on?” (x₁ + x₂ − 0.5 ≥ 0). Neuron B asks “are both on?” (x₁ + x₂ − 1.5 ≥ 0). The output fires when A − 2B − 0.5 ≥ 0.
(0, 0). A: −0.5, so 0. B: −1.5, so 0. Output sum: 0 − 0 − 0.5 = −0.5, so 0. Correct.
(0, 1). A: 0.5, so 1. B: −0.5, so 0. Output sum: 1 − 0 − 0.5 = 0.5, so 1. Correct.
(1, 0). The same as (0, 1): A = 1, B = 0, output 1. Correct.
(1, 1). A: 1. B: 0.5, so 1. Output sum: 1 − 2 − 0.5 = −1.5, so 0. Correct.
What happened. The hidden layer moved the four corners to (A, B) = (0, 0), (1, 0), (1, 0), (1, 1). In that new space one straight line separates them. That is "learning features".
3. Width and depth
- Wider (more neurons per layer): more lines to combine. One hidden layer with enough neurons can approximate any reasonable function: the universal approximation theorem (Cybenko, 1989).
- Deeper (more layers): features of features. Early layers find simple patterns; later ones combine them. For many real problems, deep and narrow beats shallow and wide with far fewer weights, which is why “deep learning” is deep.
More neurons means more weights, so more flexibility and more risk of overfitting (Lesson 3.7). Lesson 9.5 covers the defences.
- a⁽ˡ⁻¹⁾
- the outputs of the layer before: this layer's inputs
- W⁽ˡ⁾
- the weight matrix: one row of weights per neuron in this layer
- b⁽ˡ⁾
- one bias per neuron
- z⁽ˡ⁾ = Wa + b
- every neuron's weighted sum at once
- g
- the activation, applied to each sum: the bend
- Physical meaning
- A factory line. The first workers sort parts by simple features (edges, colours). Later workers assemble those into bigger pieces (wheels, windows). The last one names the product (a car). Nobody told the workers what to look for; training taught them.
“One hidden layer can learn anything, so that’s all you need.” The universal approximation theorem says a big enough network exists. It doesn’t say training will find it, how many neurons it would take (sometimes astronomically many), or that it would work on new data. In practice deeper networks reach the same accuracy with far fewer weights, which is why modern networks have many layers.
Maths and principle
Principle: each layer applies a linear map and then a bend; stacking them gives a flexible function that is still trained by gradient descent.
A layer: with the previous layer’s outputs , weight matrix and biases ,
Counting parameters: a layer from to neurons has weights and biases.
Why XOR needs only 2 hidden neurons: one neuron fires for “” (at least one on), the other for “” (both on); the output fires for “first and not second”.
Analogy: a factory line. The first workers sort parts by simple features; later workers assemble those into bigger pieces; the last one decides what the product is.
Try it: count the parameters of a network with layers 10, 32, 16, 1. (; ; . Total 897.)
4. Worked example
XOR with hand-picked weights and step activations. Inputs .
- Hidden neuron A: (OR).
- Hidden neuron B: (AND).
- Output: .
| x₁, x₂ | A | B | output sum | output |
|---|---|---|---|---|
| 0, 0 | 0 | 0 | −0.5 | 0 |
| 0, 1 | 1 | 0 | 0.5 | 1 |
| 1, 0 | 1 | 0 | 0.5 | 1 |
| 1, 1 | 1 | 1 | −1.5 | 0 |
The hidden layer moved the four corners to new positions, (0, 0), (1, 0), (1, 0) and (1, 1), where one straight line separates them. That is what “learning features” means.
5. Practice
- In the simulation, what is the fewest hidden neurons that learns the ring? XOR?
- Count the parameters of the 2-4-1 network in the quiz.
- Why does a network with one hidden neuron behave like logistic regression?
Answers
- Usually 3 for the ring (three lines make a triangle round it) and 2 for XOR, though training sometimes needs one more.
- 17 (12 in the hidden layer, 5 in the output).
- One hidden neuron gives one soft line; the output just rescales it, so the boundary is still a single straight line.
Try it on your own data: the neural network toys shows why one neuron fails on XOR; this lesson’s simulation shows the hidden layer that fixes it.
New words: fully connected layer, hidden layer, width, depth, feature learning, universal approximation, softmax.
- A fully connected network has an input layer, hidden layers of neurons (weighted sum, bias, activation) and an output layer suited to the task.
- Each hidden neuron draws one soft line; later layers combine them into curved boundaries: the network learns its own features.
- A layer from n_in to n_out neurons has n_in × n_out weights plus n_out biases.
- Wider layers add lines, deeper layers add features of features; more weights also mean more risk of overfitting.
Check yourself
Sources
- Goodfellow, Bengio and Courville (2016), Deep Learning, MIT Press, Chapter 6 (deep feedforward networks) https://www.deeplearningbook.org/
- Cybenko (1989), Approximation by superpositions of a sigmoidal function, Mathematics of Control, Signals and Systems 2(4), 303-314 https://doi.org/10.1007/BF02551274
- Bishop (2006), Pattern Recognition and Machine Learning, Springer, Chapter 5 (neural networks)
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.