Module 10: From words to an LLM · Lesson 10.4 · about 30 min
Attention: each word looks at the others
In "the cat sat on the mat because it was tired", what is "it"? Attention lets each word score every other word, softmax the scores and mix in what it needs. Explore a live attention map and work the maths.
Module 10 · about 30 minutes · read first: 1.4, 10.2, 10.3. You’ll be able to explain queries, keys and values, work out an attention output by hand, and say why attention made large language models possible.
1. Why this matters
Attention is the idea behind the transformer (“Attention is all you need”, Vaswani and colleagues, Google, 2017), and so behind every modern LLM. It also powers models for images, speech and proteins.
It fixed the forgetting problem of recurrent networks (Lesson 9.7). Any word can reach any other word in one step, and all positions are worked out at the same time on a graphics card, which makes training on huge amounts of text practical.
2. The idea
In “the cat sat on the mat because it was tired”, the word “it” means nothing on its own. To understand it, the model must find the word it refers to. Attention lets every word ask a question of every other word, using three lists of numbers:
- Query : what this word is looking for (“an animal or person I might refer to”).
- Key : what each word offers, like a label on a box (“I am an animal”).
- Value : what is inside the box, passed on if chosen.
A word’s score for another word is the dot product of its query with that word’s key (multiply place by place and add). It is high when the two point the same way (Lesson 1.4). Softmax turns one word’s scores into weights that are all positive and add up to 1. The word’s new vector is the weighted average of the values.
Everyday picture: you walk into a party wanting someone to talk football with (your query). Everyone wears a name badge listing their interests (their keys). You spend most of your time with the best match, a bit with the next, and come away with a mix of what they told you (their values).
Queries, keys and values are made from each word’s embedding by three learned weight tables, so the model learns what to look for. Real models run several heads side by side, each with its own tables, so one head can track grammar while another tracks who “it” is.
3. One row, step by step
Here is one word’s attention worked out in full, with tiny numbers. Its query is (1, 0). Three words offer keys (2, 0), (0, 2) and (1, 1), with values (1, 0), (0, 1) and (1, 1). Each key has d = 2 numbers.
Score each word. Dot the query with each key: (1, 0)·(2, 0) = 2, (1, 0)·(0, 2) = 0, (1, 0)·(1, 1) = 1. Word 1 matches best.
Scale. Divide by √d = √2 = 1.414: scores 1.414, 0, 0.707. This stops long vectors from giving huge scores.
Make them positive. Take e to each score: 4.11, 1.00, 2.03. Their total is 7.14.
Turn into weights. Divide each by the total: 0.576, 0.140, 0.284. They add up to 1, like slices of a pie.
Blend the values. 0.576 × (1, 0) + 0.140 × (0, 1) + 0.284 × (1, 1) = (0.860, 0.424). The word takes mostly what word 1 offers, plus a little of the rest.
- qi
- word i's query: what it is looking for
- kj
- word j's key: the label saying what it offers
- vj
- word j's value: the information it hands over if chosen
- sij = qi·kj / √d
- how well word j matches what word i wants, scaled by the length d of the vectors
- aij
- the attention weight: softmax of word i's scores, all positive, adding up to 1
- Physical meaning
- Attention is each word deciding whom to listen to. "it" puts most of its weight on "cat", so after attention the vector for "it" carries the cat's information, and the model "knows" what "it" means in this sentence.
Maths and principle
Principle: score every pair, softmax the scores, average the values.
For word embeddings and learned matrices : , , . With numbers per key,
In matrix form, for all words at once: .
Causal mask: when writing text, set for , so and no word sees the future.
Cost: every word scores every word, so tokens take about scores. Doubling the context quadruples this part of the work.
Analogy: a library search. Your query is the question; each book’s key is its catalogue entry; the value is what’s inside. You read a blend of the best-matching books, most from the closest match.
Try it: scores 2, 0 and 0 for three words. Weights? (; total ; weights 0.79, 0.11, 0.11.)
4. Worked example
One query , three words with keys , , and values , , . Here , so divide by .
| word | q · k | score | e^score | weight |
|---|---|---|---|---|
| 1 | 2 | 1.414 | 4.11 | 0.576 |
| 2 | 0 | 0 | 1.00 | 0.140 |
| 3 | 1 | 0.707 | 2.03 | 0.284 |
Output = 0.576 × (1, 0) + 0.140 × (0, 1) + 0.284 × (1, 1) = (0.860, 0.424). The word takes mostly what word 1 offers, because word 1’s key matches its query best, but it still blends in some of the others.
The price of looking at everything. Each of tokens scores all tokens, so the work grows with . That is why long contexts are expensive.
“Attention means the model understands, like a person paying attention.” It is arithmetic: dot products, a softmax and a weighted average. The weights often line up with what a person would look at (“it” → “cat”), but they are learned patterns, not awareness, and they are not always a reliable explanation of why the model answered as it did.
5. Practice
- In the simulation, which word does “sat” look at most, and why might a verb care about it?
- Tick “Only look back”. Why does the row for “the” (the first word) become a single 1?
- A model’s context grows from 4,000 to 32,000 tokens. Roughly how much more attention work is that?
Answers
- “cat”: a verb needs its subject (who sat), and the verb’s query asks for an animal.
- The first word can see only itself, so all its weight goes there.
- 8 times the tokens, so about 64 times the scores (8²).
Try it on your own data: the embedding visualiser shows which of your items point the same way, which is exactly what a query and a key measure.
New words: attention, self-attention, query, key, value, attention weights, attention head, causal mask, transformer.
- Each word makes a query (what it wants), a key (what it offers) and a value (what it passes on).
- Score = query · key ÷ √d; softmax turns one word’s scores into weights that add up to 1.
- A word’s new vector is the weighted average of the values, so “it” can take on the meaning of “cat”.
- When writing, a causal mask stops each word from seeing words that come after it.
- Every token scores every token, so the work grows with the square of the context length.
Check yourself
Sources
- Vaswani, A. et al. (2017), Attention Is All You Need, Advances in Neural Information Processing Systems 30 https://arxiv.org/abs/1706.03762
- Bahdanau, D., Cho, K. and Bengio, Y. (2014), Neural Machine Translation by Jointly Learning to Align and Translate https://arxiv.org/abs/1409.0473
- Jurafsky, D. and Martin, J. H., Speech and Language Processing, 3rd edition draft, chapter on transformers and large language models https://web.stanford.edu/~jurafsky/slp3/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.