Home / Learn / Data & AI / ML foundations / Lesson 10.4
All lessons

Module 10: From words to an LLM · Lesson 10.4 · about 30 min

Attention: each word looks at the others

In "the cat sat on the mat because it was tired", what is "it"? Attention lets each word score every other word, softmax the scores and mix in what it needs. Explore a live attention map and work the maths.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 10 · about 30 minutes · read first: 1.4, 10.2, 10.3. You’ll be able to explain queries, keys and values, work out an attention output by hand, and say why attention made large language models possible.

Attention lets every word ask the other words "are you what I need?", then take a blend of the answers, most from the best match.

1. Why this matters

Attention is the idea behind the transformer (“Attention is all you need”, Vaswani and colleagues, Google, 2017), and so behind every modern LLM. It also powers models for images, speech and proteins.

It fixed the forgetting problem of recurrent networks (Lesson 9.7). Any word can reach any other word in one step, and all positions are worked out at the same time on a graphics card, which makes training on huge amounts of text practical.

2. The idea

In “the cat sat on the mat because it was tired”, the word “it” means nothing on its own. To understand it, the model must find the word it refers to. Attention lets every word ask a question of every other word, using three lists of numbers:

  • Query 𝐪: what this word is looking for (“an animal or person I might refer to”).
  • Key 𝐤: what each word offers, like a label on a box (“I am an animal”).
  • Value 𝐯: what is inside the box, passed on if chosen.

A word’s score for another word is the dot product of its query with that word’s key (multiply place by place and add). It is high when the two point the same way (Lesson 1.4). Softmax turns one word’s scores into weights that are all positive and add up to 1. The word’s new vector is the weighted average of the values.

Everyday picture: you walk into a party wanting someone to talk football with (your query). Everyone wears a name badge listing their interests (their keys). You spend most of your time with the best match, a bit with the next, and come away with a mix of what they told you (their values).

Try it: what does each word look at?
looks most at
its weights add up to
Each row is one word's attention: the softmax of its scores, so the row adds up to 1. "it" looks mostly at "cat", because its query asks for an animal and "cat"'s key offers one. The word's new meaning is the weighted mix of the words' values, so after attention "it" carries mostly "cat". Higher temperature flattens every row; lower sharpens it. With "only look back", a word cannot see the words after it, which is how a model writes text one word at a time. Hand-made vectors for teaching.

Queries, keys and values are made from each word’s embedding by three learned weight tables, so the model learns what to look for. Real models run several heads side by side, each with its own tables, so one head can track grammar while another tracks who “it” is.

3. One row, step by step

Here is one word’s attention worked out in full, with tiny numbers. Its query is (1, 0). Three words offer keys (2, 0), (0, 2) and (1, 1), with values (1, 0), (0, 1) and (1, 1). Each key has d = 2 numbers.

Score each word. Dot the query with each key: (1, 0)·(2, 0) = 2, (1, 0)·(0, 2) = 0, (1, 0)·(1, 1) = 1. Word 1 matches best.

Scale. Divide by √d = √2 = 1.414: scores 1.414, 0, 0.707. This stops long vectors from giving huge scores.

Make them positive. Take e to each score: 4.11, 1.00, 2.03. Their total is 7.14.

Turn into weights. Divide each by the total: 0.576, 0.140, 0.284. They add up to 1, like slices of a pie.

Blend the values. 0.576 × (1, 0) + 0.140 × (0, 1) + 0.284 × (1, 1) = (0.860, 0.424). The word takes mostly what word 1 offers, plus a little of the rest.

qi
word i's query: what it is looking for
kj
word j's key: the label saying what it offers
vj
word j's value: the information it hands over if chosen
sij = qi·kj / √d
how well word j matches what word i wants, scaled by the length d of the vectors
aij
the attention weight: softmax of word i's scores, all positive, adding up to 1
Physical meaning
Attention is each word deciding whom to listen to. "it" puts most of its weight on "cat", so after attention the vector for "it" carries the cat's information, and the model "knows" what "it" means in this sentence.
Maths and principle

Principle: score every pair, softmax the scores, average the values.

For word embeddings 𝐱j and learned matrices 𝐖Q,𝐖K,𝐖V: 𝐪i=𝐖Q𝐱i, 𝐤j=𝐖K𝐱j, 𝐯j=𝐖V𝐱j. With d numbers per key,

sij=𝐪i⋅𝐤jd
aij=esij∑mesim,𝐨𝐮𝐭i=∑jaij𝐯j

In matrix form, for all words at once: softmax⁡(𝐐𝐊⊤/d)𝐕.

Causal mask: when writing text, set sij=−∞ for j>i, so esij=0 and no word sees the future.

Cost: every word scores every word, so n tokens take about n2 scores. Doubling the context quadruples this part of the work.

Analogy: a library search. Your query is the question; each book’s key is its catalogue entry; the value is what’s inside. You read a blend of the best-matching books, most from the closest match.

Try it: scores 2, 0 and 0 for three words. Weights? (e2=7.39; total 7.39+1+1=9.39; weights 0.79, 0.11, 0.11.)

4. Worked example

One query 𝐪=(1,0), three words with keys 𝐤1=(2,0), 𝐤2=(0,2), 𝐤3=(1,1) and values 𝐯1=(1,0), 𝐯2=(0,1), 𝐯3=(1,1). Here d=2, so divide by 2=1.414.

word q · k score e^score weight
1 2 1.414 4.11 0.576
2 0 0 1.00 0.140
3 1 0.707 2.03 0.284

Output = 0.576 × (1, 0) + 0.140 × (0, 1) + 0.284 × (1, 1) = (0.860, 0.424). The word takes mostly what word 1 offers, because word 1’s key matches its query best, but it still blends in some of the others.

The price of looking at everything. Each of n tokens scores all n tokens, so the work grows with n2. That is why long contexts are expensive.

“Attention means the model understands, like a person paying attention.” It is arithmetic: dot products, a softmax and a weighted average. The weights often line up with what a person would look at (“it” → “cat”), but they are learned patterns, not awareness, and they are not always a reliable explanation of why the model answered as it did.

5. Practice

  1. In the simulation, which word does “sat” look at most, and why might a verb care about it?
  2. Tick “Only look back”. Why does the row for “the” (the first word) become a single 1?
  3. A model’s context grows from 4,000 to 32,000 tokens. Roughly how much more attention work is that?
Answers
  1. “cat”: a verb needs its subject (who sat), and the verb’s query asks for an animal.
  2. The first word can see only itself, so all its weight goes there.
  3. 8 times the tokens, so about 64 times the scores (8²).

Try it on your own data: the embedding visualiser shows which of your items point the same way, which is exactly what a query and a key measure.

New words: attention, self-attention, query, key, value, attention weights, attention head, causal mask, transformer.

  • Each word makes a query (what it wants), a key (what it offers) and a value (what it passes on).
  • Score = query · key ÷ √d; softmax turns one word’s scores into weights that add up to 1.
  • A word’s new vector is the weighted average of the values, so “it” can take on the meaning of “cat”.
  • When writing, a causal mask stops each word from seeing words that come after it.
  • Every token scores every token, so the work grows with the square of the context length.

Check yourself

1In attention, the weights in one word's row:
2Why divide the scores by √d?
3When a model writes text, a word may attend to:

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.