Home / Learn / Data & AI / Offline AI / Lesson 3.5
All lessons

Module 3: Tokens: the unit of everything · Lesson 3.5 · about 20 min

Tokens per second: reading speed and writing speed

Local models have two speeds, a fast reading speed (prefill) and a slower writing speed (decode). What sets each, what is a good number, and how to measure yours.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 3 · about 20 minutes · read first: 3.4. You’ll be able to read the speed numbers your engine shows and know which part of your hardware sets each one.

A local model has two speeds: a fast reading speed set by how much maths the chip can do, and a slower writing speed set by how fast it can read the model from memory.

1. Why this matters

“How fast is it?” has two answers. A model may read a long document in seconds but write slowly, or the other way round. Knowing which speed limits your job tells you whether a bigger graphics card, more memory or a smaller model will help.

2. Two phases

  1. Prefill (reading): the model processes all your input tokens together. Because the work is done in parallel, it is fast, often hundreds to thousands of tokens per second on a graphics card. It is limited by compute, how much maths the chip does per second.
  2. Decode (writing): it writes one token at a time, and each needs a pass through the model’s weights. It is limited by memory bandwidth, how fast those weights can be read.

You see prefill as the pause before the first word (“time to first token”), and decode as the speed of the words after it.

Picture a translator with a long letter. Reading it through takes a few seconds, because the eyes take in many words at once. Writing the translation is slower: one word at a time, and before each word they flip through their dictionary.

You press Enter. Chen's 12-page document is 8,000 tokens of input.

Prefill. The laptop reads all 8,000 together at 500 tokens per second: 8,000 ÷ 500 = 16 seconds of waiting before anything appears.

First token. After 16 seconds the first word shows. That pause is the time to first token.

Decode. Now words appear at 12 tokens per second. A 300-token summary takes 300 ÷ 12 = 25 seconds.

Total: 41 seconds. To cut the 16-second wait, more compute (a graphics card) helps. To speed up the writing, a smaller or more tightly squeezed model helps.

3. What good looks like

  • Reading along: 10 tokens per second or more feels smooth; people read about 4 to 8.
  • Batch jobs: every extra token per second shortens the run.
  • A real example: one test measured gpt-oss-120b writing at about 88 tokens per second on a 128 GB M5 Max Mac (4K context, MLX). It is a Mixture-of-Experts model, so only part of it is read per token (Lesson 2.5).

Speeds fall as the context grows, because there is more to look back over (often by a quarter or more at long lengths).

“A faster graphics chip always means faster writing.” Writing is usually held back by memory speed, not by the chip’s maths. A chip twice as powerful with the same memory bandwidth writes at about the same speed; it only reads your input faster. When you compare machines for chat speed, compare memory bandwidth (GB/s) first.

Maths and principle

Principle: decode reads every active weight once per token, so its ceiling is bandwidth divided by bytes read.

Equation: decode speed ≤ memory bandwidth (GB/s) ÷ active weights (GB).

Analogy: a reader who must flip through the whole reference book before writing each word. A thinner book (smaller model) or faster hands (more bandwidth) means more words per minute.

Try it: a graphics card at 448 GB/s and a 7.7 GB model? (448 ÷ 7.7 ≈ 58 tokens per second at most; real speeds are often 60 to 80% of the ceiling.)

prefill
reading all the input tokens together; limited by compute (maths per second)
decode
writing one token at a time; limited by memory bandwidth
bandwidth (GB/s)
how many gigabytes the memory can deliver each second
active weights (GB)
how much of the model must be read for each token (all of it for a dense model)
Physical meaning
Every written token needs one trip through the whole model in memory. Divide how fast memory can be read by how much must be read, and you have the most tokens per second possible. Real engines reach about 60 to 80% of it.

4. Worked example

Made-up numbers. Chen’s 12-page document (8,000 tokens) takes 16 seconds before the first word appears: prefill at 500 tokens per second on his laptop. The summary then writes at 12 tokens per second. To speed up the start, a graphics card would help most; to speed up the writing, a smaller or more tightly quantised model would.

5. Practice

  1. In Ollama, run ollama run gemma4:e4b --verbose and ask a question. Find “prompt eval rate” (prefill) and “eval rate” (decode).
  2. A Mac with 170 GB/s and a 16 GB model. What is the decode ceiling?
  3. Your job reads 50 long documents and writes one line each. Which speed matters more?
Answers
  1. The prompt eval rate is usually many times the eval rate.
  2. About 10 tokens per second.
  3. Prefill (reading), because almost all the tokens are input.

New words: prefill (reading the input), decode (writing the output), time to first token (the pause before the answer starts), compute-bound and memory-bound.

  • Prefill reads all input at once and is limited by compute; you feel it as the time to first token.
  • Decode writes one token at a time and is limited by memory bandwidth.
  • Decode ceiling ≈ bandwidth (GB/s) ÷ active weights (GB); real speeds are about 60 to 80% of it.
  • About 10 tokens per second or more feels smooth to read along with.
  • Speeds fall as the context grows; in Ollama, —verbose shows both rates.

Check yourself

1What mostly limits writing speed (decode) on a home machine?
2What mostly limits reading speed (prefill)?
3About what writing speed feels comfortable to read along with?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.