Before an LLM sees words, it sees tokens. Type anything below and watch it dissolve into the subword chunks that models actually process.
Step 1 of 4 BPE steps
Sign in to save progress
Type anything. Each coloured block is one token, roughly what a language model sees.
Real-world tokenizations showing how context, language, and structure affect token count.
| Input | Tokens | Count |
|---|---|---|
| Hello world | Hello·world | 2 |
| 2024 | 2024 | 2 |
| unbelievable | unbelievable | 3 |
| ChatGPT | ChatGPT | 3 |
| Süddeutsche Zeitung | Süddeutsche·Zeitung | 6 |
Token count affects cost, context window usage, and even model quality across languages.
Cost
LLM APIs charge per token. 1K tokens ≈ 750 words, so concise prompts save money.
Context limits
GPT-4o supports 128K tokens ≈ 96K words max. Every token in context counts against this limit.
Rare words are expensive
Foreign names and rare words use more tokens than common English words, raising costs and reducing context.
Byte Pair Encoding merges frequent character pairs to build a vocabulary of ~50K subwords. Step through the process:
Start: character level
BPE begins by treating every character as its own token.
Insight
The word “tokenize” splits as [“token”, “ize”], just 2 tokens. But “détokeniser” (French) splits into 6+ tokens. Inefficient tokenization is one reason models handle non-English text worse: every foreign or rare word burns more context budget and costs more. The bigger driver, though, is training data: the corpora these models learn from are predominantly English.