Module 3: Tokens: the unit of everything · Lesson 3.8 · about 20 min
Saving tokens 3: caching, limits and structured output
Reuse the work a model already did on the same opening text, cap answer length, stop at the right moment, and ask for structured output so nothing is wasted.
Module 3 · about 20 minutes · read first: 3.6 and 3.7. You’ll be able to make repeated jobs faster with caching and stop answers from running long.
1. Why this matters
Batch jobs send nearly the same prompt hundreds of times. Without caching, the model re-reads the same instructions every time. Without limits, a few answers run on far too long. Both waste the slow, expensive part of the work.
2. Prompt caching
When a new request begins with exactly the same tokens as the last one, an engine can reuse the stored keys and values for that shared beginning (the KV cache from Lesson 2.4) instead of reading it again. The shared beginning is called the prefix.
It is like a bookmark. If you always start from the same chapter, you don’t reread the book from page one each time; you open it at the bookmark.
The job. 500 support emails. Every request starts with the same 2,000-token instructions, then 300 tokens of that email.
No caching. Each request reads 2,000 + 300 = 2,300 tokens. 500 × 2,300 = 1,150,000 tokens read.
Request 1 with caching. The engine reads all 2,300 and keeps the keys and values for the first 2,000.
Requests 2 to 500. The first 2,000 tokens match exactly, so they are reused. Only each email's 300 new tokens are read.
Total. 2,000 + 500 × 300 = 152,000 tokens read: about 7.5 times fewer. Change one word near the start and the match breaks, so keep the fixed part first.
- Locally: llama.cpp’s server and Ollama reuse a shared beginning while the model stays loaded. Keep the model loaded between requests (Ollama’s
keep_aliveoption) and keep the start identical. - In the cloud: several providers offer cheaper “cached input” for repeated beginnings; check each provider’s pricing page.
- Order matters: fixed system prompt and reference document first; the changing part (this email, this question) last.
3. Caps and stops
- Max tokens (
num_predictin Ollama,max_tokensin OpenAI-style calls,-nin llama.cpp): a hard ceiling per answer. - Stop sequences: end the answer when a chosen string appears, such as a blank line or
###. - Thinking off for easy tasks (Lesson 10.4).
4. Structured output
JSON with a schema (Lesson 11.3) removes the chatty extras (“Sure! Here is…”) and gives fields your code can use directly. It is usually both shorter and more reliable than free text.
Maths and principle
Principle: with caching, the shared prefix is read once; each request only reads its new part.
Equation: input tokens read ≈ prefix + n × new part (cached) instead of n × (prefix + new part).
Analogy: a teacher who reads the class rules aloud once in the morning, not before every question.
Try it: 500 requests, a 2,000-token prefix and 300 new tokens each? (Without caching 1,150,000 tokens read; with caching 2,000 + 150,000 = 152,000, about 7.5 times fewer.)
- prefix
- the identical opening tokens every request shares: fixed instructions, a reference document
- new part
- the tokens that change each time: this email, this question
- n
- how many requests the job sends
- prefix + n × new part
- tokens read with caching: the prefix once, then only the new parts
- Physical meaning
- Caching turns "read the rule book before every question" into "read it once in the morning". The longer the shared opening and the more requests, the bigger the saving, but only if the opening is exactly the same each time.
“Prompt caching stores old answers and hands them back.” It doesn’t reuse answers at all. It reuses the work of reading an identical opening (the stored keys and values). Every request still gets a fresh answer, written for its own question.
5. Worked example
Made-up numbers. A support team classifies 1,000 tickets overnight with a 1,500-token instruction block. With the instructions first and the ticket last, the engine reuses the instructions; each ticket’s reading time falls from 4 seconds to under 1. With a 40-token cap and JSON output, the whole run drops from 3 hours to 50 minutes.
6. Practice
- Reorder one of your batch prompts so the fixed part comes first.
- Add a max-tokens cap to the folder script in 11.2 (
options={"num_predict": 120}). - Using the box: 200 requests, 3,000-token prefix, 200 new tokens. How many tokens are read with and without caching?
Answers
- Anything that changes per request (names, dates, the document) goes last.
- Answers that hit the cap end mid-sentence; ask for a shorter format so they finish naturally.
- Without: 640,000. With: 3,000 + 40,000 = 43,000.
New words: prompt caching or prefix caching, max tokens, stop sequence, keep_alive (how long Ollama keeps a model loaded).
- Prompt caching reuses the work of reading an identical opening; it never reuses answers.
- Put the fixed instructions and documents first and the changing part last; keep the model loaded.
- Tokens read with caching ≈ prefix + n × new part, instead of n × (prefix + new part).
- Max tokens caps each answer; stop sequences end it at a chosen string; switch thinking off for easy tasks.
- JSON with a schema removes chatty extras and gives your code fields it can use directly.
Check yourself
Sources
- Kwon, W. et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention (KV cache reuse) https://arxiv.org/abs/2309.06180
- Ollama API documentation (keep_alive, num_predict, stop and format options) https://github.com/ollama/ollama/blob/main/docs/api.md
- Anthropic, Prompt caching (official documentation) https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.