Home / Learn / Data & AI / Offline AI / Lesson 4.2
All lessons

Module 4: The parts that matter for AI · Lesson 4.2 · about 20 min

Memory bandwidth: the real speed limit

Why writing speed depends on how fast memory can be read, not on how fast the chip is, and how to estimate tokens per second for any machine.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 4 · about 20 minutes · read first: 4.1 and 3.5. You’ll be able to guess a machine’s writing speed before you buy it.

To write each word, the chip must read the whole model from memory, so the speed of that memory, not the chip, sets how fast the AI writes.

1. Why this matters

Lesson 4.1 asked “does it fit?”. This lesson asks “how fast will it write?”. The answer is mostly one number on the spec sheet, memory bandwidth, which shops rarely print in big letters. Knowing it saves you from paying for a fast chip that sits waiting.

2. One token, one full read

To write each new token, the chip combines the conversation with every weight in the model (Lesson 2.1). The weights live in memory. So for every token, all of them travel from memory to the chip. A 7.7 GB model means 7.7 GB read per token.

The chip itself could do the maths much faster. It spends most of its time waiting for the next batch of weights to arrive. That is why we say writing is memory-bound: memory is the bottleneck.

Picture a cook who needs every jar in the pantry for every single dish. A faster cook does not help if the pantry door is narrow. A wider door does. Bandwidth is the width of the door.

Token 1. A 7.7 GB model on a card that reads 448 GB/s. Reading every weight once takes 7.7 ÷ 448 ≈ 0.017 s, or 17 ms.
The maths is quick. Once the weights arrive, the sums take only a sliver of that time. For the rest, the chip waits.
Tokens 2, 3, 4 ... Every new token needs the whole read again. In the first 100 ms there is room for about 5 tokens.
Per second. 448 ÷ 7.7 ≈ 58 full reads a second, so at most about 58 tokens per second. Real engines reach about 50 to 80% of this: roughly 30 to 45.
Double the door. The same model on a 936 GB/s card reads in about 8 ms, so the ceiling rises to about 120 tokens per second. Same model, same maths, faster memory.
Try it: the speed limit

Each crossing of the dot stands for one token: the chip must read every weight it uses once per word piece. Top speed ≈ memory speed ÷ GB read per token. Real engines reach about half to four fifths of this ceiling. Memory speeds are makers' figures (Lesson 4.1 sources); the Mac mini figure is rounded, check Apple's spec page.

In symbols, with B the bandwidth in GB/s and M the GB read for each token:

tokens per second≤BM
B (bandwidth)
How many GB the memory can hand to the chip each second. It is on the spec sheet as GB/s.
M
GB read for each token: the whole model for a normal (dense) model, only the active part for a Mixture-of-Experts model.
Memory-bound
The chip spends its time waiting for memory, not doing sums.
Physical meaning
Every word the AI writes is a full trip through its memory. A machine with twice the bandwidth makes that trip twice as fast and writes about twice as many words per second, whatever its chip is called.

3. Reading the speed off a spec sheet

Memory bandwidth is written in GB/s (gigabytes per second):

  • Laptop memory (two DDR5 sticks): about 60 to 100 GB/s.
  • Base Mac mini class: about 120 GB/s; Pro and Max chips: 270 to over 600 GB/s.
  • 128 GB AI boxes (DGX Spark, Ryzen AI Max class): about 256 to 273 GB/s.
  • Graphics cards: RTX 5060 Ti 448 GB/s, used RTX 3090 936 GB/s, RTX 5090 1,792 GB/s.

A Mixture-of-Experts model (Lesson 2.5) reads only its active experts for each token, so M is much smaller than the file. That is why big MoE models can write quickly even on medium bandwidth.

“A faster processor makes the AI write faster.” While writing, the chip mostly waits for memory. A higher clock speed or more “AI TOPS” with the same memory changes little. Look for the GB/s number instead.

4. Reading time versus writing time

Reading your prompt (prefill, Lesson 3.5) is different. The chip handles many prompt tokens together, so each weight it reads is used many times. Prefill is compute-bound: here the chip’s raw power matters, and graphics cards beat Macs and AI boxes by a wide margin. That is why a long document can take a while to “start” on a Mac even when the writing is quick.

Maths and principle

Principle: writing speed is capped by how many times per second memory can hand over every active weight.

Equation: top tokens per second ≈ bandwidth (GB/s) ÷ bytes read per token (GB). Real engines reach about 50 to 80% of it.

Analogy: a cook who needs every ingredient in the pantry for each dish. A faster cook does not help if the pantry door is narrow; a wider door does.

Try it: a 16 GB model on a 1,792 GB/s card? (1,792 ÷ 16 = 112 tokens per second ceiling, about 56 to 90 in practice.)

5. Worked example

Made-up numbers. Arjun compares a 128 GB AI box (273 GB/s) with a Mac Studio (about 614 GB/s) for a 27B model at 4-bit (about 16 GB). The box: 273 ÷ 16 ≈ 17 tokens per second ceiling, so about 9 to 14 in practice. The Mac: 614 ÷ 16 ≈ 38, so about 19 to 31. For a mixture-of-experts model that reads only 3 GB per token, both feel quick, and the box’s lower price wins for him.

6. Practice

  1. Your laptop has 80 GB/s. How fast could Gemma 4 E2B (about 1.6 GB) write at best?
  2. Why does a long pasted report feel slower to start on a Mac than on a graphics card?
  3. Two cards both have 16 GB: one 448 GB/s, one 896 GB/s. Which models can each run, and how do they differ?
Answers
  1. 80 ÷ 1.6 = 50 tokens per second ceiling, about 25 to 40 in practice.
  2. Reading the prompt is compute-bound, and graphics cards have much more raw compute.
  3. The same models (same memory), but the 896 GB/s card writes about twice as fast.

New words: bandwidth (GB moved per second), memory-bound (waiting on memory), compute-bound (waiting on the maths).

  • Every new token needs every active weight read from memory once.
  • Speed ceiling ≈ bandwidth (GB/s) ÷ GB read per token; real engines reach about 50 to 80% of it.
  • Writing is memory-bound, so faster memory helps more than a faster chip.
  • Reading a long prompt is compute-bound, which is where graphics cards pull far ahead.
  • Mixture-of-Experts models read only their active part, so they write fast for their size.

Check yourself

1A graphics card reads memory at 448 GB/s. A model's weights are 7.7 GB. What is the rough ceiling on writing speed?
2Why does a Mixture-of-Experts model with 120B parameters write faster than a dense 27B model on the same box?
3Which change speeds up writing the most on a machine where the model already fits?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.