Module 10: From words to an LLM · Lesson 10.2 · about 25 min
Embeddings: meaning as distance
An embedding gives every word a list of numbers, learned so that words used in similar places sit close together. See king − man + woman land on queen, and how LLMs and search use the same idea.
Module 10 · about 25 minutes · read first: 1.4, 10.1. You’ll be able to explain what an embedding is, how it is learned, why similar words end up close, and how word arithmetic works.
1. Why this matters
TF-IDF (Lesson 10.1) sees “cheap” and “inexpensive” as two unrelated columns. Embeddings fix that. They are everywhere:
- the first layer of every language model;
- “semantic” search that finds documents by meaning, not just matching words;
- recommendations (products and films get embeddings too);
- chatbots that answer from your own files.
2. Words as points
An embedding gives each word a short list of numbers. You can think of the list as the word’s address: a point in a space of, say, 300 dimensions (300 numbers instead of the 2 on a paper map). Training puts words that appear in similar sentences close together.
Nobody decides what each number means. Even so, directions in the space end up meaning something, such as male to female, one to many, or country to capital.
Because relations are directions, you can do sums with words. Walk from “man” to “king” and you have taken a “royal” step. Take the same step from “woman” and you land near “queen”:
Words as points. On a made-up 2-number map: king (1, 2.8), man (1, 1.4), woman (2.2, 1.4), queen (2.2, 2.8). Fruit words sit far away in their own corner.
The royal step. king − man = (1 − 1, 2.8 − 1.4) = (0, 1.4): straight up by 1.4.
Take the same step from woman. woman + (0, 1.4) = (2.2, 1.4 + 1.4) = (2.2, 2.8).
Find the nearest word. The closest point to (2.2, 2.8) is queen. In real models the sum lands near queen, not exactly on it, so the nearest word is the answer.
Relations are directions. man → woman and king → queen are the same arrow. The map has learned a "male to female" direction without anyone telling it.
This was the famous result of word2vec (Mikolov and colleagues at Google, 2013). Real spaces are messier than the picture, and analogies work only some of the time.
3. How embeddings are learned
“You shall know a word by the company it keeps” (J. R. Firth, 1957). Word2vec trains a tiny network on billions of words of text with one job: given a word, guess the words around it (or the reverse).
- Start every word at a random point.
- For each word in the text, nudge it towards the words that really appear near it, and away from random other words.
- Repeat over the whole text. Words that share neighbours (“cheap” and “inexpensive” both sit near “price”, “very”, “quite”) end up close.
Everyday picture: seat people at a wedding by who they chat with. After a few rounds, the cousins are at one table and the work friends at another, though nobody wrote a label on anyone.
The table of points is simply the network’s first weight matrix. An LLM learns its own embedding table the same way, as part of guessing the next word (Lesson 10.3). Then it adjusts each word’s numbers to the sentence around it, so “bank” by a river and “bank” with money get different vectors.
- V
- how many words (or tokens) are in the vocabulary
- d
- how many numbers each word gets, such as 300
- E
- the embedding table: V rows by d columns, one row per word
- oi
- a one-hot row: all zeros except a 1 in word i's place
- ei = oiE
- word i's embedding: the one-hot row simply picks out row i of the table
- Physical meaning
- An embedding is an address for meaning. Two reviews that say the same thing in different words get nearby addresses, so a search for "battery lasts long" finds "goes two days on one charge". Distance on the map stands for difference in meaning.
Maths and principle
Principle: meaning is position; similar use gives similar position.
Lookup: with a vocabulary of words and numbers per word, the embedding matrix is . Word as a one-hot row (all 0 except a 1 at place ) has embedding
which is row of . So an embedding layer is a linear layer with one-hot input, done as a fast lookup.
Similarity: cosine similarity (Lesson 10.1), .
Analogy: a city map where similar shops cluster: bakeries near bakeries. Walking from “bakery” to “café” is the same short step anywhere in town.
Try it: and . Cosine? (; both lengths are 3; , very similar.)
4. Worked example
The made-up map in the simulation: king (1, 2.8), man (1, 1.4), woman (2.2, 1.4), queen (2.2, 2.8).
- king − man = (0, 1.4): the “royal” step.
- (0, 1.4) + woman = (2.2, 2.8) = queen exactly.
Search works the same way. To answer “phones with a long-lasting battery”, embed the query and every review, then return the reviews whose vectors have the highest cosine with the query. A review saying “lasts two days on one charge” scores high, though it shares almost no words with the query: TF-IDF would miss it.
The table itself takes memory. A vocabulary of 50,000 tokens with 4,096 numbers each is about 205 million numbers, before any of the model’s other layers.
“Each number in an embedding stands for one human idea, like ‘royalty’ or ‘colour’.” Usually not. The meaning is spread across many numbers at once, and only some directions line up with ideas we can name. That is why we look at distances and directions, not at single numbers.
5. Practice
- In the simulation, what does Paris − France + Italy give, and why?
- Why is an embedding of 300 numbers better than a one-hot row of 50,000?
- Why does “bank” need context to get the right meaning, and which part of an LLM supplies it?
Answers
- Rome: France to Paris is the same “capital of” arrow as Italy to Rome.
- It is far shorter, and similar words share numbers, so what a model learns about one word carries over to its neighbours. One-hot rows are all equally far apart.
- One fixed point can’t be near both “river” and “money”. Attention (Lesson 10.4) mixes in the surrounding words, giving each use of “bank” its own vector.
Try it on your own data: the embedding visualiser plots high-dimensional rows in 2D, so you can see which of your items sit together.
New words: embedding, one-hot, embedding matrix, word2vec, context, semantic search, contextual embedding.
- An embedding gives each word a list of numbers: a point in space where similar words sit close together.
- It is learned by guessing a word’s neighbours in lots of text; words with the same company end up near each other.
- Relations become directions, so king − man + woman lands near queen.
- Looking up an embedding is a one-hot row times the table: it just picks out that word’s row.
- Search by meaning embeds the query and the documents and ranks them by cosine similarity.
Check yourself
Sources
- Mikolov, T., Chen, K., Corrado, G. and Dean, J. (2013), Efficient Estimation of Word Representations in Vector Space https://arxiv.org/abs/1301.3781
- Jurafsky, D. and Martin, J. H., Speech and Language Processing, 3rd edition draft, chapter 6 (word2vec, embeddings) https://web.stanford.edu/~jurafsky/slp3/
- Goodfellow, I., Bengio, Y. and Courville, A. (2016), Deep Learning, MIT Press, section 12.4 (natural language processing) https://www.deeplearningbook.org/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.