Home / Learn / Data & AI / Offline AI / Lesson 3.3
All lessons

Module 3: Tokens: the unit of everything · Lesson 3.3 · about 20 min

The context window: a model's working memory

Everything the model can see at once, instructions, chat history, documents and its own answer, has to fit in the context window. What happens when it is full, and how to set it.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 3 · about 20 minutes · read first: 3.1. You’ll be able to say what fills the context window, what happens when it overflows, and how to set its size.

The context window is everything the model can see at once; instructions, chat, documents and its own answer must all fit, and whatever falls out is simply gone.

1. Why this matters

Most “the model forgot” and “it ignored my document” problems are context problems. Knowing the size of the window, and what is in it, lets you design prompts and scripts that keep the important text where the model can see it.

2. What the window holds

For every new token it writes, the model looks at everything in its context window:

  1. the system prompt (standing instructions);
  2. the conversation so far, your messages and its replies;
  3. any documents or tool results you added;
  4. the answer it is writing, so far.

The context length is the most tokens that can fit. Current open models are trained for long contexts: Gemma 4 31B for 256K tokens, Qwen3.8-27B for 262K, gpt-oss-120b for 128K (K means thousand). But your engine decides how much it actually uses, and many start with a few thousand tokens to save memory.

Think of a desk. The model can only use the papers spread out on it. A bigger desk holds more, but costs more room in the office (memory).

Watch an 8,192-token window fill up, with the numbers from the box below.

An empty window of 8,192 tokens.

Standing instructions. The system prompt takes 300 tokens.

The chat so far. Your messages and its replies: 2,500 tokens.

Room for the answer. Keep 1,000 tokens free for the reply it will write. Left for a document: 8,192 − 3,800 = 4,392 tokens, about 3,300 words.

Overflow. Paste a 9,000-token contract instead and it cannot all fit. The engine drops the oldest text first: the start of the chat, then the start of the contract. The model never sees them.

3. When it is full

When the conversation grows past the context length, engines drop the oldest text. The model is not being careless; it simply can no longer see it. Two fixes: a longer window (if memory allows), or a short summary of the old conversation at the start of a new one (Lesson 3.7).

Even inside the window, very long contexts have a known weakness: models pay less attention to the middle of long text than to its start and end (Lesson 17.4). Put key instructions at the start or the end.

“My model supports 256K tokens, so my app gives it 256K.” Often it doesn’t. The 256K is what the model was trained to handle; the engine sets what it actually uses, and many start at 4K or 8K because every extra token of window needs memory. If a long document seems ignored, check the context length setting first.

4. Setting it

  • Apps: a “context length” setting when you load the model.
  • Ollama: in a chat, /set parameter num_ctx 16384; in code, options={"num_ctx": 16384}.
  • llama.cpp: -c 16384.

Raise it step by step and watch memory (Lesson 8.4 explains why it grows).

Maths and principle

Principle: the window is a fixed-size buffer; what goes in must sum to less than its size.

Equation: system + history + documents + answer ≤ context length.

Analogy: a whiteboard. You can only work with what is written on it; when it is full, you rub out the oldest part to keep writing.

Try it: an 8,192 context with a 300-token system prompt, 2,500 tokens of history and a 1,000-token answer. How big can a document be? (8,192 − 3,800 = 4,392 tokens, about 3,300 words.)

system
tokens of standing instructions at the start
history
tokens of the conversation so far, both sides
documents
tokens of anything pasted or attached, and tool results
answer
tokens you keep free for the reply
context length
the size of the window the engine is set to, in tokens
Physical meaning
The model has no memory outside this window. If the sum is too big, something is cut, usually the oldest text, and the model behaves as if it was never said. Plan the budget like packing a bag with a weight limit.

5. Worked example

Made-up numbers. Omar pastes a 9,000-token contract into a chat set to 8,192 tokens. The model summarises only the last part, because the start was cut. He raises the context to 16,384; memory use grows by about 1 GB and the summary now covers the whole contract.

6. Practice

  1. Find the context-length setting in your app or engine.
  2. With a 4,096 window, a 200-token system prompt and a 500-token answer, how big can the rest be?
  3. Why put the main instruction after a long document as well as before it?
Answers
  1. Write down the default; you will compare it with your memory in Lesson 8.4.
  2. 4,096 − 700 = 3,396 tokens.
  3. Text near the end of a long context gets more attention, and the instruction is less likely to be lost.

New words: context window or context length (how many tokens the model can see at once), system prompt (standing instructions at the start), truncation (dropping the oldest text).

  • The context window holds the system prompt, the conversation, any documents and the answer being written.
  • system + history + documents + answer ≤ context length.
  • When it overflows, the oldest text is dropped and the model can no longer see it.
  • Models may support 128K to 262K tokens, but engines often start at a few thousand to save memory; raise it step by step.
  • Put key instructions at the start or end: the middle of a long context gets the least attention.

Check yourself

1What counts towards the context window?
2Your chat passes the context length. What usually happens?
3A model supports 256K tokens. Why might your engine still be set to 8K?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.