Home / Learn / Data & AI / ML foundations / Lesson 10.3
All lessons

Module 10: From words to an LLM · Lesson 10.3 · about 25 min

Predicting the next word

A language model gives a probability to every possible next word. Build one by counting word pairs, write sentences with it, see what temperature does, and learn the loss every LLM is trained on.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 10 · about 25 minutes · read first: 1.6, 4.3. You’ll be able to build a bigram model by counting, work out the probability of a sentence, explain temperature, and read a model’s loss and perplexity.

A language model is a very good guesser of the next word. Guess, write the word down, and guess again: that loop is all an LLM does.

1. Why this matters

Every LLM, from a phone keyboard to a chatbot, does one thing. Given the text so far, it gives a probability to each possible next token (a word or a piece of a word), picks one, and repeats. Answering questions, writing emails and writing code all come out of that loop.

Seeing the loop on a tiny model explains two things at once: why LLMs sound so fluent, and why they can be confidently wrong.

2. Counting: n-gram models

The simplest language model just counts. A bigram model (“bi” = two) counts which word follows which in some training text. If “the” is followed by “cat” 4 times out of 20, then P(cat|the)=4/20=0.2. Read the bar ”|” as “given”: the chance of “cat”, given that the last word was “the”. A trigram model looks at the two words before, and so on.

Everyday picture: the word suggestions above your phone keyboard. After “see you”, it offers “soon”, “tomorrow” and “later”, because those are what usually come next.

Try it: write like a language model

The 14 training sentences

the cat sat on the mat. the cat ate the fish. the dog sat on the rug. the dog ate the bone. a cat likes fish. a dog likes bones. the cat likes the sun. the dog likes the park. the sun is warm. the park is big. the mat is warm. the fish is fresh. the cat sleeps on the mat. the dog sleeps in the park.

The model only knows which word followed which in its training text: after "the", it saw "cat" 4 times out of 20, so it gives "cat" a probability of 4/20 = 0.2. Adding the most likely word every time soon loops ("the cat likes the cat likes ..."). Drawing at random by the probabilities gives more varied sentences. Temperature reshapes the odds before drawing: below 1 the likely words get likelier; above 1 the odds flatten. A large language model does the same thing, but its probabilities come from a network that reads the whole text so far, not just the last word.

To write, start from the start-of-sentence marker, pick a next word, and repeat. Always taking the single most likely word gives dull loops. Drawing at random by the probabilities gives variety. Temperature reshapes the odds first: below 1, likely words get likelier; above 1, the odds even out and surprises creep in.

A sentence’s probability is each word’s probability multiplied together. Watch the bigram model score “the cat sat”:

First word. 12 of the 14 training sentences start with "the", so P(the | start) = 12/14 = 0.857.

Second word. "the" appears 20 times and is followed by "cat" 4 times: P(cat | the) = 4/20 = 0.2.

Third word. "cat" appears 5 times and is followed by "sat" once: P(sat | cat) = 1/5 = 0.2.

Multiply. P(the cat sat) = 0.857 × 0.2 × 0.2 = 0.034. Each extra word can only shrink the total.

An unseen pair. "flew" never follows "cat" in the training text, so P(flew | cat) = 0, and "the cat flew" gets probability 0, however natural it sounds.

3. From counts to neural language models

Counting breaks down fast. With a vocabulary of 50,000 words there are 125 trillion possible three-word sequences, and most never appear in any text, so most counts are zero.

A neural language model swaps the count table for a network:

  1. embed the previous words (Lesson 10.2);
  2. pass them through layers;
  3. output one score per vocabulary word, turned into probabilities by softmax (which makes the scores positive and adds them up to 1).

Because similar words have similar embeddings, similar contexts give similar guesses, even for word sequences the model has never seen.

Early neural language models read only the last few words, or used recurrent networks (Lesson 9.7) that forgot. Transformers (Lessons 10.4 and 10.5) can use thousands of earlier tokens.

4. Scoring a model: loss and perplexity

How do we know a model guesses well? Show it real text, and at each position look at the probability it gave the word that really came next. The loss for that word is −ln⁡p: 0 if it was sure and right (p = 1), large if it gave the true word a tiny chance.

p
the probability the model gave the word that really came next
−ln p
the loss for that word (cross-entropy): 0 for a sure, right guess; it grows fast as p gets small
L
the average loss over every position in the text
perplexity = eL
how many words the model is, in effect, choosing between at each step
Physical meaning
Perplexity is the model's "number of equally likely options". A perplexity of 20 means it is as unsure as if it picked fairly from 20 words each time. A better model has a lower number, and a model with perplexity 1 would know every next word for certain.
Maths and principle

Principle: a sentence’s probability is the product of each word’s probability given what came before.

Chain rule of probability:

P(w1,…,wn)=∏t=1nP(wt|w1,…,wt−1)

A bigram model keeps only the last word: P(wt|wt−1)=count(wt−1,wt)/count(wt−1).

Softmax with temperature T: from scores zi,

pi=ezi/T∑jezj/T

Loss: the cross-entropy, −ln⁡p(true next word), averaged over every position in the training text. Its exponential is the perplexity: a perplexity of 20 means the model is as unsure as if it chose fairly among 20 words.

Analogy: predictive text on a phone, which suggests the next word from what you usually type after the last one.

Try it: a model gives the true next words probabilities 0.5 and 0.125. Mean loss and perplexity? ((ln⁡2+ln⁡8)/2=ln⁡4=1.386; perplexity e1.386=4.)

5. Worked example

From the 14 training sentences in the simulation: a sentence starts with “the” 12 times out of 14; “the” is followed by “cat” 4 times out of 20; “cat” by “sat” once out of 5.

P(the cat sat)=1214×420×15=0.857×0.2×0.2=0.034

“the cat flew” gets probability 0: “flew” never follows “cat” in the training text. A neural model would give it a small, sensible probability, since “flew” behaves like other verbs it has seen.

“Temperature 0 makes the model correct.” Low temperature only makes it pick its top word every time. If the top word is wrong, it is now wrong every time, just more consistently. Temperature controls variety, not truth.

6. Practice

  1. In the simulation, press “Add the most likely word” until the sentence stops. What goes wrong, and why?
  2. Set the temperature to 0.2 and draw at random several times. How is that like choosing the top word?
  3. Why does a bigram model give zero probability to many perfectly good sentences?
Answers
  1. It loops: “the cat likes the cat likes …”, because after “the” the top word is always “cat”, and so on round the circle.
  2. At low temperature the top word’s probability is close to 1, so a random draw almost always picks it.
  3. It only knows word pairs it has seen. Any sentence with one unseen pair gets probability 0, however natural.

Try it on your own data: the word counter counts the words in your own text, the first step of any n-gram model.

New words: language model, token, bigram, n-gram, chain rule of probability, softmax, temperature, cross-entropy, perplexity.

  • A language model gives a probability to every possible next token; writing is guessing, adding the word and guessing again.
  • A bigram model counts pairs: P(next | last) = count of the pair ÷ count of the last word.
  • A sentence’s probability is the product of its words’ probabilities, so one unseen pair gives a count model 0.
  • Temperature reshapes the odds: low is safe and repetitive, high is varied and riskier; it never makes an answer true.
  • The training loss is −ln p of the true next word; perplexity = eloss is the number of words the model is, in effect, choosing between.

Check yourself

1"the" is followed by "cat" 4 times out of 20 in the training text. A bigram model gives P(cat | the) =
2Raising the temperature when drawing the next word makes the text
3A model gives the true next word probability 0.25. Its loss for that word is:

Sources

  • Jurafsky, D. and Martin, J. H., Speech and Language Processing, 3rd edition draft, chapter 3 (n-gram language models, perplexity) https://web.stanford.edu/~jurafsky/slp3/
  • Bengio, Y., Ducharme, R., Vincent, P. and Jauvin, C. (2003), A Neural Probabilistic Language Model, Journal of Machine Learning Research 3, 1137-1155
  • Goodfellow, I., Bengio, Y. and Courville, A. (2016), Deep Learning, MIT Press, section 12.4 (n-grams and neural language models) https://www.deeplearningbook.org/

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.