Home / Learn / Data & AI / Offline AI / Lesson 3.2
All lessons

Module 3: Tokens: the unit of everything · Lesson 3.2 · about 20 min

Tokenizers differ: languages, code and counts

The same text gives different token counts in different models and languages. Why that happens and what it means for Hindi, Malayalam, code and your limits.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 3 · about 20 minutes · read first: 3.1. You’ll be able to predict when your text will cost more tokens and plan for it.

Each model family learns its own set of word pieces, mostly from English, so the same meaning can cost a few tokens in one language or model and several times more in another.

1. Why this matters

If you work in Hindi, Tamil, Malayalam, Arabic or with code and spreadsheets, your token counts can be much higher than the English rule of thumb. That means shorter documents per request and slower jobs, locally or in the cloud.

2. Each family learns its own pieces

A tokenizer’s vocabulary is learned from training text: pieces that appear often become single tokens. It is like a shorthand writer who invented quick symbols for the words they hear most. Their English is fast; a language they rarely hear is spelled out letter by letter. So:

  • Different families split differently. Gemma, Qwen and gpt-oss each have their own tokenizer, so the same document gives different counts.
  • Bigger vocabularies split less. Newer models have vocabularies of about 200,000 to 260,000 pieces and handle more languages efficiently than older ones of about 32,000.
  • Languages common in training text do best. English is cheapest almost everywhere. Languages with less training text, and scripts with many combining marks, are split more finely, often costing several times more tokens for the same meaning.
  • Code and data cost more. Indentation, brackets, long identifiers and numbers all add pieces.

Use the language buttons in the toy tokenizer in Lesson 3.1 to see the effect (exaggerated by its tiny vocabulary).

“A page is a page: it costs the same tokens whatever the model or language.” Token counts belong to the tokenizer, not the text. Switch model family and an English document can change by 10 to 30%. Research comparing many languages (Petrov and colleagues, 2023, in the sources) found the same meaning can take several times as many tokens in some languages as in English. Always measure with the model you will actually use.

3. What to do about it

  • Measure, do not guess: Lesson 3.10 shows how to get exact counts from your engine.
  • Prefer a model whose tokenizer handles your language well; model cards often say which languages were a focus.
  • For mixed documents, keep the instructions in English (cheapest) and the content in its own language.
  • Allow a bigger context for non-English work.
Maths and principle

Principle: token count = characters ÷ average characters per token, and that average depends on how well the vocabulary fits the text.

Equation: tokens = characters ÷ c, where c ≈ 4 for English and lower for text the vocabulary fits badly.

Analogy: packing with boxes made for someone else’s things. When the boxes match your items, few boxes are needed; when they don’t, you use many small ones.

Try it: 8,000 characters with c = 4, then with c = 1.5? (2,000 tokens, then about 5,300.)

characters
how long the text is, counting every letter, space and mark
c
average characters per token: about 4 for English, often 1.5 to 3 for text the vocabulary fits badly
tokens = characters ÷ c
the token count; a smaller c means more tokens for the same text
Physical meaning
c measures how well the model's word pieces fit your text. A low c means your documents fill the context window sooner, take longer to read and cost more on a cloud API, even though they say the same thing.

4. Worked example

Made-up numbers. A Kerala school wants summaries of 2,000-word Malayalam notices. Step through their planning.

In English a 2,000-word notice would be about 2,000 ÷ 0.75 ≈ 2,700 tokens.

In Malayalam their model's tokenizer gives about 7,000 tokens for the same notice: about 2.6 times as many.

Room in the window. Their context is 8,192 tokens. Keep 1,000 for the summary and about 200 for the instructions: 8,192 − 1,200 = 6,992 tokens left for the notice.

Check the fit. 7,000 is just over 6,992, so they trim the summary allowance to 800 tokens and one notice fits. Two notices (14,000) never would.

Plan. The script sends one notice per request, with the instructions in English (cheapest) and the notice in Malayalam.

5. Practice

  1. Write the same sentence in English and another language you know, and compare in the toy tokenizer.
  2. A 5,000-character Python file at c = 3. About how many tokens?
  3. Your model has a 16,000-token context. Roughly how many characters of text with c = 2 fit, leaving 2,000 tokens for the answer?
Answers
  1. The non-English version will show more tokens in the toy; real tokenizers narrow the gap but rarely close it.
  2. About 1,700 tokens.
  3. 14,000 × 2 = 28,000 characters.

New words: vocabulary size (how many pieces a tokenizer knows), characters per token (how efficiently a text is cut).

  • Each model family learns its own vocabulary, so the same document gives different token counts.
  • Bigger vocabularies (about 200,000 to 260,000 pieces) split less and handle more languages well.
  • Languages and scripts less common in training text, and code and data, are split into more tokens.
  • Tokens = characters ÷ c, where c is about 4 for English and lower when the vocabulary fits badly.
  • Measure with your own model, keep instructions in English, and allow a bigger context for non-English work.

Check yourself

1You switch from one model family to another. What happens to the token count of the same document?
2Why does Malayalam text often use more tokens than English text with the same meaning?
3What does a higher token count mean for a local model?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.