Home / Learn / Data & AI / ML foundations / Lesson 10.6
All lessons

Module 10: From words to an LLM · Lesson 10.6 · about 25 min

How an LLM is trained

An LLM is trained in stages: next-word prediction on trillions of words, then example conversations, then people's preferences. See the same question answered after each stage, and estimate the cost of training.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 10 · about 25 minutes · read first: 3.3, 10.3, 10.5. You’ll be able to name the three stages of LLM training, say what data and goal each one uses, work out a model’s loss and perplexity, and estimate the computing cost of pretraining.

An LLM learns in three stages: read a huge library guessing each next word, practise answering requests, then learn which answers people prefer.

1. Why this matters

Knowing how a model was trained explains its habits:

  • why a base model rambles instead of answering;
  • why chat models are polite and sometimes over-cautious;
  • why they know things up to a date and nothing after;
  • why “fine-tuning on your own data” is a much smaller job than training from scratch.

It also explains why only a handful of companies pretrain large models, while anyone can adapt one.

2. Three stages

  1. Pretraining. Guess the next token (Lesson 10.3) over a huge mix of public web pages, books, code and papers: trillions of tokens. The text supplies its own answers, so nobody has to label anything (self-supervised learning). The result, a base model, has soaked up grammar, facts, styles and reasoning patterns. But it continues text rather than answering it.
  2. Instruction tuning (supervised fine-tuning). Train on example conversations: a request and a good reply, many written by people. The loss is the same next-token loss, counted only on the replies. The model learns the shape of a helpful answer.
  3. Feedback tuning. Show people (or a judging model) two answers to the same request and record which is better. Methods such as RLHF (reinforcement learning from human feedback) and DPO (direct preference optimisation) shift the model towards the preferred answers: clearer, more correct, safer, more honest about doubt.
Try it: same question, three stages

Prompt: How do I boil an egg?

Learns from
Goal
After pretraining, the model continues text: asked a question, it may write more questions, because that is what often follows a question on the web. Instruction tuning teaches it to answer; feedback tuning polishes how. All three stages train the same network with gradient descent (Lesson 3.3); only the data and the target change. The answers are written to show typical behaviour, not output from a particular model.

All three stages run the same network with gradient descent (Lesson 3.3) and backpropagation (Lesson 9.4); only the data and the target change. Later, smaller fine-tunes (such as LoRA, in the Offline AI course) adapt a finished model to one company’s style or task with a few thousand examples.

Here is what one tiny step of pretraining looks like, on the text “the cat sat”:

Hide the next word. The model sees "the cat" and must guess what comes next. The answer, "sat", is already in the text: no person needed to label it.

Guess. The model gives "sat" probability 0.20 (and the rest to other words).

Score the guess. Loss = −ln 0.20 = 1.61. A sure, right guess would cost 0.

Nudge every weight. Backpropagation works out which way each weight should move to make "sat" likelier here, and gradient descent moves each one a tiny step.

Next time. The same context now gives "sat" 0.30, so the loss falls to −ln 0.30 = 1.20. Repeat for trillions of positions.

3. What the numbers mean

  • Loss and perplexity (Lesson 10.3) are watched throughout pretraining on held-out text, exactly like validation loss in Lesson 9.5.
  • Scaling: loss falls smoothly and predictably as models, data and computing grow together. Labs use this to plan a run before paying for it.
  • Knowledge cut-off: the model knows only what was in its training text. Events after it are unknown unless they are put in the prompt or found by a search tool.
Maths and principle

Principle: one loss, next-token cross-entropy, applied to different data at each stage.

Pretraining loss over n positions of text:

L=−1n∑t=1nln⁡pθ(wt|w1,…,wt−1),perplexity=eL

Compute rule of thumb: training a model with N parameters on D tokens takes about

C≈6ND arithmetic operations

(2 per weight per token for the forward pass, 4 for the backward pass).

Analogy: a doctor’s training. Years of reading everything (pretraining), then supervised practice on real cases (instruction tuning), then feedback from senior colleagues on which of two approaches was better (feedback tuning).

Try it: a model gives the true next tokens probabilities 0.5, 0.25, 0.1 and 0.8. Mean loss and perplexity? (Losses 0.693, 1.386, 2.303, 0.223; mean 1.151; perplexity e1.151=3.16: as unsure as choosing among about 3 tokens.)

N
the number of parameters (weights) in the model
D
the number of tokens of training text it reads
6
about 2 operations per weight per token to make a guess, and 4 more to work out the nudges
C
the total number of arithmetic operations for pretraining
Physical meaning
C is a bill. Divide it by how fast your graphics cards work and you get weeks of a data centre's time and millions in electricity and hardware. That is why only a few labs pretrain big models, and why most people adapt an existing one instead.

4. Worked example

How much computing does a mid-sized open model need? Take N=8 billion parameters and D=15 trillion tokens:

C≈6×(8×109)×(15×1012)=7.2×1023

A modern data-centre graphics card might keep up about 4×1014 operations a second in practice. That gives 1.8×109 card-seconds: about 500,000 card-hours, or roughly 10 days on 2,000 cards, before any failed runs or experiments. Instruction and feedback tuning add only a small fraction of this. Fine-tuning a finished model with LoRA can run on one good computer.

“Chat models learn their facts from people writing answers in fine-tuning.” Almost all of a model’s knowledge comes from pretraining on huge amounts of text. Instruction and feedback tuning mainly teach it how to answer: format, tone, when to say “I’m not sure”. Fine-tuning on a few hundred examples is a poor way to add new facts; putting the documents in the prompt usually works better.

5. Practice

  1. In the simulation, why does the pretrained model answer a question with more questions?
  2. Why is pretraining called self-supervised, and why does that matter for its scale?
  3. Training loss on held-out text drops from 2.3 to 2.0. What happens to perplexity?
Answers
  1. It continues text. On the web, a question is often followed by similar questions (FAQ pages, forums), so that is a likely continuation.
  2. The next token is the label, so any text can be used without anyone labelling it. That is why it can use trillions of tokens.
  3. It falls from e2.3≈10 to e2.0≈7.4: the model is less unsure.

Try it on your own data: the prompt pruner turns your own PDFs and web pages into lean text that uses fewer tokens, the units every number in this lesson is counted in.

New words: pretraining, self-supervised learning, base model, instruction tuning, supervised fine-tuning, RLHF, DPO, knowledge cut-off, scaling law, LoRA.

  • Pretraining: guess the next token over trillions of tokens; the text is its own label (self-supervised).
  • Instruction tuning: example requests and good replies teach the shape of a helpful answer.
  • Feedback tuning (RLHF, DPO): people’s choices between two answers steer the model to preferred replies.
  • All stages use the same next-token loss and gradient descent; perplexity = eloss tracks progress.
  • Pretraining costs about C ≈ 6ND operations, which is why few labs pretrain and most people adapt.

Check yourself

1During pretraining, the "label" for each position is:
2Instruction tuning teaches a model mainly:
3Feedback tuning (such as RLHF) learns from:

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.