Module 3: Tokens: the unit of everything · Lesson 3.7 · about 20 min
Saving tokens 2: trimming and summarising context
Long chats and long documents fill the context window. How to summarise history, send only the parts that matter, and split big documents so nothing important falls out.
Module 3 · about 20 minutes · read first: 3.3. You’ll be able to keep long work inside the context window without losing what matters.
1. Why this matters
The context window is the model’s whole world. Fill it with old small talk and there is no room for the document that matters. Overflow it and the start falls out. Managing it is the biggest lever on quality, speed and cost in long tasks.
2. Long chats: summarise and restart
Every message re-sends the whole conversation. Watch how fast that adds up, with turns of 200 tokens each.
Turn 1. The model reads your 200-token message. Total read so far: 200.
Turn 2. It reads turn 1 again plus the new 200: 400. Total so far: 200 + 400 = 600.
Turn 3. It reads 600. Total: 200 + 400 + 600 = 1,200, which is 200 × 3 × 4 ÷ 2.
Turn 30. The last turn alone reads 6,000 tokens, and the chat has read 200 × 30 × 31 ÷ 2 = 93,000 in all. Each turn is slower than the one before.
Summarise and restart. Ten bullet points of decisions take about 300 tokens. A new chat that starts from them reads 300 instead of 6,000 on its first turn.
After a long chat:
- Ask: “Summarise our decisions and open questions in 10 bullet points.”
- Start a new chat and paste the summary.
You keep the substance in a few hundred tokens instead of thousands. Try it in the simulation: fill the window, then tick “Summarise old messages”.
3. Long documents: send only what matters
- Cut by hand when you know where the answer is: send the section, not the file.
- Search, then send (RAG): split the document into chunks, find the chunks most related to the question, and send only those (Module 12).
- Map-reduce when you need the whole thing summarised: summarise each part, then summarise the summaries.
4. Clean the input
Remove repeated headers and footers, signatures, long email chains already answered, page numbers and boilerplate. A cleaning step in your script (Lesson 11.2) can cut a third of the tokens from typical office documents.
Maths and principle
Principle: every turn re-sends the history, so a chat’s total input grows roughly with the square of its length.
Equation: total input over n turns of t tokens each ≈ t × n(n + 1) ÷ 2.
Analogy: reading a whole notebook from page one every time you add a line. Writing a short summary at the top and starting a new notebook keeps each read short.
Try it: 30 turns of 200 tokens? (200 × 30 × 31 ÷ 2 = 93,000 input tokens in total, though the last turn alone is only 6,000.)
- t
- tokens added in each turn (your message plus the reply)
- n
- the number of turns so far
- n(n + 1) ÷ 2
- 1 + 2 + 3 + … + n: turn 1 is read n times, turn 2 n − 1 times, and so on
- t × n(n + 1) ÷ 2
- all the input tokens the chat has read in total
- Physical meaning
- Doubling the length of a chat roughly quadruples the reading it costs. Locally that is waiting time; in the cloud it is money and plan limits. A short summary resets the count close to zero while keeping what matters.
“Each message only costs the tokens I just typed.” The model has no memory between turns except the text it is sent, so every turn sends the whole conversation again. A one-line question at the end of a long chat can cost thousands of input tokens. That is why long chats get slower and drain plan limits faster.
5. Worked example
Made-up numbers. An engineer summarises a 120-page site report. Pasting it fails: about 80,000 tokens against a 16,000 window. Her script splits it into 12 sections, summarises each in 150 words, then summarises the 12 summaries into one page. Each request stays under 10,000 tokens, and the whole run takes 14 minutes on her laptop.
6. Practice
- In the simulation, fill a 4,000-token window with and without summarising. How many messages are lost?
- Using the box, how many input tokens do 50 turns of 300 tokens send in total?
- Which approach for “What does section 7.3 say about refunds?” in a 90-page policy?
Answers
- With summarising, far fewer are lost; the summary keeps their gist.
- 300 × 50 × 51 ÷ 2 = 382,500 tokens.
- Cut by hand (send section 7.3) or search for it (RAG). Map-reduce is for whole-document summaries.
New words: summarise and restart, chunk (a piece of a document), RAG (retrieval-augmented generation: search, then send), map-reduce (summarise parts, then combine).
- Every turn re-sends the whole history: total input ≈ t × n(n + 1) ÷ 2 over n turns of t tokens.
- After a long chat, ask for a short summary of decisions and open questions, then start fresh with it.
- For long documents: cut by hand, search and send only relevant chunks (RAG), or map-reduce for whole summaries.
- Clean the input: headers, footers, signatures and old email chains can be a third of the tokens.
- A lean window is faster, cheaper and gives better answers.
Check yourself
Sources
- Lewis, P. et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks https://arxiv.org/abs/2005.11401
- Liu, N. F. et al. (2023), Lost in the Middle: How Language Models Use Long Contexts https://arxiv.org/abs/2307.03172
- Dean, J. and Ghemawat, S. (2004), MapReduce: Simplified Data Processing on Large Clusters, OSDI 2004 (the idea behind map-reduce summarising)
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.