Module 17: Pros, cons and limits · Lesson 17.2 · about 20 min
Speed limits and where they come from
Why local models write at the speed they do. The memory bandwidth ceiling, the difference between reading the prompt and writing the answer, why long chats slow down, and what thinking tokens cost in time.
Module 17 · about 20 minutes · read first: 4.2 and 3.5. You’ll be able to explain the speed you see, and know which upgrades would make it faster.
1. Why this matters
Speed decides whether local AI feels pleasant or painful. It is not a mystery: it follows a few physical limits. Once you know them, you can predict the speed of a machine before you buy it, and you will not waste money on upgrades that cannot help.
2. The ceiling: bandwidth
To write each token, the chip reads every active weight once. Memory bandwidth is how many gigabytes per second the chip can read from its memory. So tokens per second can never pass bandwidth ÷ gigabytes read per token. Think of a librarian who must walk the whole library once for every word they write: the walking speed sets the writing speed.
That is why a mixture-of-experts model (Lesson 2.5) with few active weights can be fast even when it is large, and why more memory alone does not mean more speed.
3. Two phases: reading and writing
- Prefill (reading your prompt): all prompt tokens go through at once. Limited by the chip’s raw arithmetic (compute-bound), so a graphics card reads a long prompt far faster than a laptop CPU. This sets the time to first token.
- Decode (writing the answer): one token at a time. Limited by bandwidth (memory-bound), as above.
A machine can be great at one and poor at the other. Macs with high bandwidth write quickly but read very long prompts more slowly than big graphics cards, which matters for document search with long contexts.
4. Long contexts slow both
Every new token compares itself with all earlier tokens (attention, Lesson 2.4) and reads their KV cache (Lesson 8.4). At 100 tokens that is nothing; at 100,000 it can halve the speed or worse, and prefill of a very long prompt can take minutes on modest hardware.
5. Thinking tokens cost time
A reasoning model (Lesson 10.4) may write 2,000 tokens of thinking before a 200-token answer. At 20 tokens per second, that is 100 seconds of waiting before the first word of the answer. Turn thinking down for easy tasks.
Watch where the seconds go for one 400-token answer on a 16 GB card:
A short chat. A 200-token prompt is read in a blink (0.1 s). The answer is written at 45 tokens per second: 400 ÷ 45 ≈ 8.9 s.
Paste a 40,000-token document. Reading it takes about 15 s before the first word. With so much context, writing also slows to about 30 tokens per second: 400 ÷ 30 ≈ 13.3 s.
Switch thinking on. The model writes 1,200 tokens of thinking first, at the same 30 tokens per second: another 40 s. Total about 68 s.
The fix. Send only the 4,000 relevant tokens (document search, Module 12) and turn thinking off. Reading takes 1.5 s, writing about 10 s: under 12 s in all, with no new hardware.
- P
- prompt tokens: everything sent, including pasted documents and the chat so far
- sread
- prefill speed in tokens per second, set by the chip's arithmetic
- T, A
- thinking tokens and answer tokens the model writes
- swrite
- decode speed in tokens per second
- B ÷ G
- memory bandwidth (GB/s) over gigabytes read per token: the ceiling on writing speed
- Physical meaning
- Reading is a team of clerks scanning a request together; writing is one clerk fetching one book at a time from the stacks. You can speed up the clerks with a stronger chip, but the fetching only gets faster with faster memory or smaller books (a smaller model or quant).
Maths and principle
Principle: time = waiting + reading + thinking + writing, each with its own limit.
Equation: answer time ≈ prompt tokens ÷ prefill speed + (thinking + answer tokens) ÷ decode speed; decode speed ≤ bandwidth ÷ GB per token.
Analogy: a library. Reading the request is quick for a team of clerks working together (prefill); fetching each book is one trip to the stacks at a time (decode).
Try it: 273 GB/s box, 3.1 GB per token (gpt-oss-120b at 4-bit)? (Ceiling ≈ 88 tokens per second; expect perhaps 45 to 70 in practice.)
“More memory will make my model faster.” More memory lets a bigger model fit; it does not make reading it quicker. Writing speed follows memory bandwidth, not size. A machine with twice the memory but the same bandwidth writes the same model at about the same speed.
6. Worked example
Made-up measurements. Jordan’s 16 GB card writes 45 tokens per second with a 12B model in a short chat. With a 40,000-token document pasted in, the first word takes 15 seconds and writing drops to about 30 tokens per second. Turning on a thinking mode adds 40 seconds. Jordan keeps thinking off for summaries and uses document search (Module 12) to send only the relevant 4,000 tokens instead of the whole file.
7. Practice
- Work out the ceiling for your machine and model.
- Your first word takes ages but writing is quick. Which phase is slow?
- Name two ways to make a long-document task faster without new hardware.
Answers
- Bandwidth ÷ (active parameters × bits ÷ 8, in GB).
- Prefill (or a queue, Lesson 15.1).
- Send less text (search for the relevant parts) and turn thinking down; a smaller quant also helps decode.
New words: prefill, decode, time to first token, compute-bound, memory-bound.
- Writing speed can never pass memory bandwidth ÷ GB of active weights read per token.
- Prefill (reading the prompt) is limited by arithmetic and sets the time to first token; decode (writing) is limited by bandwidth.
- Long contexts slow both phases; thinking tokens add their own writing time before the answer.
- Answer time ≈ prompt ÷ reading speed + (thinking + answer) ÷ writing speed.
- Send less text and turn thinking down before buying new hardware.
Check yourself
Sources
- Pope et al., "Efficiently Scaling Transformer Inference" (memory-bound decoding, compute-bound prefill), 2022, https://arxiv.org/abs/2211.05102
- NVIDIA and Apple product pages for memory bandwidth (see Lesson 4.1 sources)
- llama.cpp (official repository; llama-bench reports prompt-processing and generation speed separately), https://github.com/ggml-org/llama.cpp
- Vaswani et al., "Attention Is All You Need" (every token attends to all earlier tokens), 2017, https://arxiv.org/abs/1706.03762
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.