Home / Learn / Data & AI / Offline AI / Lesson 10.3
All lessons

Module 10: Settings and prompting · Lesson 10.3 · about 15 min

Context length settings and their memory cost

Why local engines start with a short context, how to raise it in Ollama, LM Studio and llama.cpp, and how to pick a length your memory can hold.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 10 · about 15 minutes · read first: 3.3 and 8.4. You’ll be able to set a context length that holds your documents and still fits in memory.

The context length you set is how much text the model can hold at once; too short and it silently drops the start, too long and the memory runs out.

1. Why this matters

A model card may say “128K context”, but your engine may start at 4,096 tokens to save memory. Long documents then get cut off without any warning. Setting the right length is one of the most common fixes for “it ignored my file”.

2. Defaults

  • Ollama: depends on graphics memory. A late-2026 guide reports 4,096 tokens under 24 GB, 32,768 for 24 to 48 GB, and 256,000 at 48 GB or more. Check your version’s documentation.
  • LM Studio: shows a context length slider when you load a model, with an estimate of the memory it will use.
  • llama.cpp: -c 16384 sets it; -c 0 uses the model’s maximum.

The context is like a desk. A bigger desk holds more papers at once, but it takes up more of the room (memory). Whatever does not fit on the desk falls off, usually from the oldest end.

The document. A contract of 7,500 words is about 7,500 ÷ 0.75 = 10,000 tokens. Add a 200-token system prompt.

The default is too small. The engine is set to 4,096 tokens. Only the last part fits; roughly the first 6,000 tokens never reach the model. It summarises the second half as if it were the whole contract.

Work out what you need. 10,000 (contract) + 200 (system) + 1,000 (room for the answer) = 11,200 tokens.

Pick a setting. The next common size above 11,200 is 16,384. Set num_ctx to 16384.

Check the memory. The KV cache grows with the context, roughly four times bigger at 16,384 than at 4,096. Use the calculator below: model + cache + 1 GB must fit (Lesson 4.1).

3. How to change it

  • Ollama, whole server: set OLLAMA_CONTEXT_LENGTH=16384 before starting it, or use the context slider in the desktop app’s settings.
  • Ollama, one chat: /set parameter num_ctx 16384.
  • Ollama, from code: options={"num_ctx": 16384}.
  • Ollama, saved model: PARAMETER num_ctx 16384 in a Modelfile.
  • LM Studio: move the slider before loading.

4. Picking a length

  1. Estimate your longest input: words ÷ 0.75 = tokens (Lesson 3.1), plus room for the answer.
  2. Use the calculator to check the KV cache for that length.
  3. Check that model + cache + 1 GB fits (Lesson 4.1).
Calculator: the KV cache

Formula: 2 × layers × KV heads × head size × bytes × tokens. In config.json these are num_hidden_layers, num_key_value_heads and head_dim (or hidden size ÷ attention heads). Some models (Gemma, gpt-oss) look only at a recent window on some layers and need less: the engine's log shows the real figure. 8-bit and 4-bit include the small scale overhead of llama.cpp's q8_0 and q4_0.

“The model card says 128K, so my model reads 128K.” The card gives the most the model can handle. Your engine decides how much it actually reserves, and it often starts much lower to save memory. Check the setting, not the card.

Maths and principle

Principle: reserve the context you need, not the most the model allows.

Equation:

context needed≈input words0.75+answer tokens+system prompt

Analogy: booking a hall: book for the guests you expect, not the building’s fire limit.

Try it: a 9,000-word report, a 1,000-token answer and a 200-token system prompt? (9,000 ÷ 0.75 = 12,000 + 1,000 + 200 = 13,200 tokens, so 16,384 is a good setting.)

context length (num_ctx)
the number of tokens the engine reserves room for: prompt, document, chat so far and the answer together
words ÷ 0.75
turns an English word count into an approximate token count
answer tokens
space you leave for the model to write its reply; it shares the same budget
truncation
what happens when the text is longer than the setting: part of it is cut and the model never sees it
Physical meaning
The context is the size of the model's desk. Make it big enough for today's papers plus a notepad for the answer, but every extra inch costs memory, and if the desk no longer fits in the room the model will not load or will crawl.

5. Worked example

Made-up numbers. Ritu’s summaries miss the first half of long contracts. Her Ollama default is 4,096 tokens; contracts run to 10,000. She sets num_ctx to 16,384 in her script. The KV cache adds about 1 GB for her model, which still fits on her 16 GB card.

6. Practice

  1. Find your engine’s current context length.
  2. Work out the tokens of your longest regular document.
  3. Set a context length with room to spare and test a long document.
Answers
  1. Ollama: ollama ps shows loaded models and their context in recent versions; LM Studio shows it on the model load panel.
  2. Words ÷ 0.75.
  3. If the answer now mentions the start of the document, it worked.

New words: num_ctx, context slider, truncation (cutting off text that does not fit).

  • The model card gives the most a model can read; the engine’s setting decides what it actually reads.
  • Engines often start at a few thousand tokens, and longer input is cut off without warning.
  • Context needed ≈ input words ÷ 0.75 + answer tokens + system prompt; round up to the next common size.
  • In Ollama set num_ctx (per chat, per request or in a Modelfile) or OLLAMA_CONTEXT_LENGTH for the whole server.
  • A longer context means a bigger KV cache: check that model + cache + 1 GB still fits.

Check yourself

1You paste a 20-page report into Ollama and the answer ignores the beginning. Likely cause?
2Why not always set the maximum context the model supports?
3In Ollama, which setting changes the context for one request from code?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.