Module 9: Neural networks · Lesson 9.1 · about 20 min
The perceptron: a neuron as a weighted vote
An artificial neuron adds up its weighted inputs and fires if the total passes a threshold. The perceptron learns those weights by nudging them after each mistake. Teach one AND and OR, and watch it fail at XOR.
Module 9 · about 20 minutes · read first: 1.4, 4.4. You’ll be able to compute a neuron’s output, run the perceptron learning rule by hand, and explain why one neuron is not enough.
1. Why this matters
Every neural network, including the large language models at the end of this course, is built from one simple unit repeated billions of times. The perceptron (Frank Rosenblatt, 1958) is that unit in its first form. Its story also explains a famous setback: when Minsky and Papert showed in 1969 that one perceptron can’t learn XOR, interest in neural networks collapsed for over a decade.
2. One neuron
Picture deciding whether to go to the beach. Sunny? That counts a lot for going. Windy? That counts against. Day off? Counts for. You add up the reasons, each with its own importance, and go if the total is high enough. A neuron does exactly this with numbers.
A neuron takes inputs , multiplies each by a weight (how much it counts), adds a bias (how keen it is to say yes anyway), and passes the total through an activation (the yes/no rule):
Each input pushes the total up or down by its weight, and the neuron “fires” (says 1) if the total reaches 0. The boundary between firing and not is the line (or flat plane) , as in Lesson 4.4.
- x₁ … x_d
- the inputs: the facts about one case
- w₁ … w_d (weights)
- how much each fact counts, and in which direction (negative weights vote against)
- b (bias)
- the neuron's starting lean: a negative bias means "it takes a lot to convince me"
- z
- the total score of the vote
- ŷ (y-hat)
- the decision: 1 if z ≥ 0, otherwise 0
- Physical meaning
- A committee where each member's vote is multiplied by their influence. The motion passes if the weighted total clears the bar. Learning means adjusting each member's influence after the committee gets a decision wrong.
3. Learning from mistakes
The perceptron rule: go through the examples. Whenever the neuron is wrong, nudge the weights towards the right answer. (eta, the learning rate) sets the size of each nudge.
- Should be 1, said 0: , .
- Should be 0, said 1: , .
- Right: change nothing.
Watch it happen
Teaching a neuron AND (fire only when both inputs are 1), starting from weights (0, 0), bias 0, η = 1.
Input (0, 0), answer 0. z = 0 + 0 + 0 = 0, so it fires: says 1. Wrong. Subtract: the weights stay (0, 0) because the input is zero, and the bias drops to −1.
Inputs (0, 1) and (1, 0), answer 0. z = −1 each time, so it says 0. Right: change nothing.
Input (1, 1), answer 1. z = 0 + 0 − 1 = −1, so it says 0. Wrong. Add the input: weights become (1, 1), bias 0.
Check the fix. For (1, 1) the new total is 1 + 1 + 0 = 2: it fires. Adding the input always raises that input's score, by ‖x‖² + 1 = 3 here.
Keep going. A few more passes nudge it to weights (2, 1) and bias −3. Now (1, 1) scores 0 and fires, and every other input scores below 0. AND learned.
One pass through all the examples is an epoch. If some straight line separates the two classes, the rule is guaranteed to find one in a finite number of steps (the perceptron convergence theorem). If no such line exists, it never stops.
4. The fix: more neurons
XOR (“one or the other, but not both”) needs two lines: “at least one input is on” and “not both”. Two neurons can each draw one line, and a third can combine their answers. That is a network with a hidden layer, and it can solve XOR. But the perceptron rule only trains a single layer: it needs to know each neuron’s correct answer, and hidden neurons have none. Training hidden layers needed two more ideas: smooth activations (Lesson 9.2) and backpropagation (Lesson 9.4).
“An artificial neuron works like a brain cell.” It borrowed the name and a rough idea: inputs, a threshold, an output. A real neuron is a living cell with thousands of connections, chemical signals, timing and constant change. An artificial neuron is a line of arithmetic: multiply, add, compare. Modern networks are powerful because of how many of these simple sums they stack and how they are trained, not because they copy biology.
Maths and principle
Principle: a weighted sum with a threshold, trained by correcting mistakes.
The update with label and prediction :
is +1, −1 or 0, giving the three cases above.
Why it moves the right way: after adding , the new score for the same input is , strictly bigger than before.
Neurons as logic gates: with , the bias gives AND and gives OR.
Analogy: a committee vote where each member’s vote counts by their weight, and members who keep backing the wrong side lose influence.
Try it: weights (2, 1), bias −3. Output for (1, 1)? For (1, 0)? (: 1. : 0. This neuron computes AND.)
5. Worked example
Learning AND (output 1 only for (1, 1)), starting from , , , visiting (0, 0), (0, 1), (1, 0), (1, 1) in order:
| epoch | input | z | said | should | new w, b |
|---|---|---|---|---|---|
| 1 | (0, 0) | 0 | 1 | 0 | (0, 0), −1 |
| 1 | (0, 1) | −1 | 0 | 0 | unchanged |
| 1 | (1, 0) | −1 | 0 | 0 | unchanged |
| 1 | (1, 1) | −1 | 0 | 1 | (1, 1), 0 |
| 2 | (0, 0) | 0 | 1 | 0 | (1, 1), −1 |
| 2 | (0, 1) | 0 | 1 | 0 | (1, 0), −2 |
| 2 | (1, 0) | −1 | 0 | 0 | unchanged |
| 2 | (1, 1) | −1 | 0 | 1 | (2, 1), −1 |
It carries on the same way; by epoch 6 it reaches , and makes no more mistakes: the AND neuron of the “Try it” above.
6. Practice
- In the simulation, how many epochs does OR take? And AND?
- Which bias makes the neuron with weights (1, 1) compute OR?
- Why does the rule never stop on XOR?
Answers
- OR settles in a few epochs; AND takes a few more (6 in the order used here).
- Any bias between −1 and 0, for example −0.5: then one input on is enough.
- Every line misclassifies at least one corner, so each epoch has a mistake, and each mistake changes the weights.
Try it on your own data: the neural network toys lets you click your own points and watch a perceptron’s line move epoch by epoch.
New words: artificial neuron, weight, bias, activation, perceptron, perceptron rule, epoch, linearly separable.
- A neuron computes a weighted sum z = w·x + b and, in a perceptron, outputs 1 if z ≥ 0.
- The perceptron rule adds η·x to the weights when the answer should have been 1, subtracts it when it should have been 0, and does nothing when right.
- If a straight line can separate the classes, the rule finds one; if not (XOR), it never settles.
- Problems like XOR need a hidden layer of neurons, which needs smooth activations and backpropagation to train.
Check yourself
Sources
- Rosenblatt (1958), The perceptron: a probabilistic model for information storage and organization in the brain, Psychological Review 65(6), 386-408 https://doi.org/10.1037/h0042519
- Minsky and Papert (1969), Perceptrons: An Introduction to Computational Geometry, MIT Press
- Goodfellow, Bengio and Courville (2016), Deep Learning, MIT Press, Chapters 1 and 6 https://www.deeplearningbook.org/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.