What the example proves, and the language gap

The 16 tests in code/mini-bpe check these facts, and I ran them:

• Lossless. decode(encode(text)) == text for English, Hindi, emoji, accents and extra spaces.
• Never fails. An invented word such as zzxqj becomes five single-byte tokens, and still decodes correctly.
• Compresses familiar text. Text like the training text uses far fewer tokens than bytes.
• Deterministic. Training twice gives the same merges.
• Unfamiliar text costs more. Text unlike the training data stays close to one token per byte.

Here is the last point as real numbers. After training on English, the example tokenizer reads English and Hindi like this:

$ python demo.py
'the agent reads the tokens'
  26 chars, 26 bytes -> 4 tokens
'मुझे चाय पसंद है। एजेंट टूल को कॉल करता है।'
  43 chars, 111 bytes -> 111 tokens

The English sentence needs 4 tokens for 26 characters (0.15 tokens per character). The Hindi sentence needs 111 tokens for 43 characters (2.6 tokens per character), because each Devanagari character takes three bytes and the tokenizer never learned any Hindi merges, so it falls back to bytes.

If you then train on English and Hindi together, Hindi drops to 7 tokens (0.16 per character). Be careful with that number. I made the Hindi part of the training text by repeating the same sentence six times, so the tokenizer learned that exact sentence. It proves the direction of the effect (training on a language makes it cheaper), not how much a real tokenizer helps.

The same effect exists in real models, and it matters. A published measurement for GPT-4's tokenizer on the same short sentence in different languages reported a median of 7 tokens in English, 32 in Hindi and 72 in Burmese:

Since price and delay grow with the number of tokens, the same message can cost several times more in some languages. If your users write in Hindi or another Indian language, measure with your own text before you promise a price.