II · THE IDEA · ARTIFICIAL INTELLIGENCE
What Tokenisation Breaks
▶ Listen · narrated
Ask a model to reverse the letters in a word and watch it stumble. Not because reversal is hard, but because it never saw the letters as separate things to reverse.
At a glance
- What breaks
- Character-level tasks, arithmetic, rhyme detection, and fair treatment of low-resource languages
- Why it breaks
- The model operates on token ids, not letters or meaning
- Measured impact
- Non-English text can require 2–10× more tokens for equivalent content
Imagine you are trying to teach someone to count the letter r in a word, but you are only allowed to hand them the word chopped into random chunks — sometimes whole syllables, sometimes single letters, with no pattern. They can still do it, but they have to first reassemble the chunks back into letters, then count. That is what a model faces when you ask it to count characters: the tokeniser has already chopped the text into pieces that do not line up with letters, and the model has to work backwards from those pieces. The same problem affects arithmetic. If the number 1234 sometimes arrives as one chunk and sometimes as "12" and "34", the model cannot treat it as a four-digit number directly — it has to parse the chunks first. And for languages that were rare in the tokeniser's training data, common words get chopped into many small pieces instead of staying whole, which means more tokens to process, higher cost, and a weaker signal for the model to learn from.
Tokenisation converts text into a sequence of integer ids by matching against a fixed vocabulary, typically built using byte pair encoding on a training corpus. The vocabulary size is usually thirty to fifty thousand entries, and the algorithm iteratively merges the most frequent character pairs until the target size is reached. This means frequent words and subwords in the training corpus earn dedicated tokens, while rare words fragment into multiple tokens. The model never operates on characters directly — it sees only token ids — so any task that requires character-level reasoning must be performed by implicitly decoding tokens back into character sequences within the forward pass. This decoding is learned, not hard-coded, so accuracy depends on how often the model encountered similar patterns during training. For arithmetic, the problem is that numbers tokenise based on digit-sequence frequency, not numerical properties. The number 1275 might be one token if that sequence appeared often, or "12" + "75", or "1" + "2" + "7" + "5". Consecutive integers can have completely different token-level representations, which means the model cannot rely on positional or structural cues at the token level. It must learn to parse these fragments back into positional notation before performing arithmetic, and this parsing is a significant source of errors, especially for multiplication and operations on longer numbers. For multilingual models, the efficiency of tokenisation varies by language according to its frequency in the tokeniser's training corpus. Petrov et al. measured this directly and found that Tamil required 6.5 times as many tokens as English for equivalent content. Since inference cost scales with token count, not semantic content, this creates both a cost inequity and a quality gap: rare tokens have less refined embeddings, having been updated fewer times during training. Bostrom et al. argue that byte pair encoding is suboptimal specifically because it optimises for compression on the training corpus rather than for downstream model performance, and that alternative algorithms could mitigate some of these failures. However, changing the tokeniser requires retraining all embeddings, so the choice made during initial development tends to persist.
Look closer
Counting letters requires reconstructing what was hidden
When you ask a model to count the r's in "strawberry", it must first decode the token ids back into a character sequence, count within that reconstruction, then report the result. If "strawberry" arrived as two tokens — say "straw" and "berry" — the model has no direct access to the fact that the first token contains one r and the second contains two. It can learn to perform this decoding implicitly during training, but it is doing so through a layer of indirection that makes the task harder than it looks. The same applies to reversing spellings, detecting rhymes, or any operation that depends on seeing individual characters.
Numbers fragment unpredictably
A tokeniser trained on text will treat numbers as character sequences, not quantities. The number 4827 might become one token, or it might split into "48" and "27", or "4", "8", "2", "7" — the outcome depends on which digit sequences appeared frequently enough in the tokeniser's training data to earn their own vocabulary entries. This means arithmetically similar numbers can have completely different token-level representations, and a model trying to add them must first parse these arbitrary fragments back into quantities. Multiplication and long division are especially fragile, because they depend on positional structure that the tokenisation has obscured.
Languages underrepresented in training become expensive to process
Petrov and colleagues measured tokenisation efficiency across languages and found that a sentence in Tamil required more than six times as many tokens as the same sentence translated into English, using a tokeniser trained predominantly on English text. Each token costs the same to process regardless of how much meaning it carries, so a Tamil user pays six times as much per sentence and exhausts the context window six times faster. The model also has weaker signal to work with, because frequent Tamil words arrive as chains of low-frequency fragments rather than single well-trained units. This is not a property of the language — it is a property of the training distribution that built the tokeniser's vocabulary.
The story
Tokenisation is supposed to be a preprocessing step, a piece of plumbing that converts text into numbers so the model can begin. But the choice of where to split — which fragments earn a place in the fixed vocabulary — propagates through every layer of the system and determines what the model finds easy and what it finds hard.
Consider arithmetic. A human child learning to add does so with a stable representation of digits: the symbol "7" always means seven. A language model sees an integer as a sequence of token ids, and which ids appear depends on accidents of frequency in the tokeniser's training data. The number 487 might be one token, or it might be "48" + "7", or three separate digits. Neighbouring numbers like 486 and 488 may tokenise completely differently. The model must learn not just arithmetic but also a kind of on-the-fly parsing that reconstructs numerical meaning from these arbitrary fragments. This is why models tend to perform better on addition than multiplication, and why they struggle more with longer numbers: the longer the number, the more tokens, and the more opportunities for the representation to obscure the positional structure that arithmetic depends on.
Character-level tasks fail for a related reason. When you ask a model to count letters, reverse a spelling, or identify a rhyme, you are asking it to operate on units it has never directly seen. The token "berry" is a single indivisible id during processing. To count the r's inside it, the model must implicitly decode that id back into the character sequence b-e-r-r-y, perform the count, then surface the result. It can learn to do this — training data contains enough examples of spelling and letter-counting that the model builds an internal representation of the mapping — but it is doing so through a layer of indirection. The task is harder than it would be if characters were the native unit, and failures are more common with rare words, because those words tokenise into unfamiliar fragments whose character-level structure the model has seen less often.
The inequity across languages is the most consequential failure. Petrov and colleagues measured how many tokens were required to encode the same semantic content across thirty languages. They found that a sentence in Bulgarian required 1.15 times as many tokens as English, German required 1.18 times as many, and Tamil required 6.5 times as many. The variation tracks the language's representation in the tokeniser's training data: languages that appeared frequently earn compact encodings, and languages that appeared rarely fragment into many small pieces.
This is not a cosmetic problem. Every token costs the same to process, so a Tamil user pays 6.5 times as much per sentence of equivalent meaning. The context window fills 6.5 times faster, so less conversation history or document text fits. And because Tamil words arrive as chains of low-frequency token fragments rather than single well-trained units, the model has a weaker, noisier signal to work with. The embeddings for rare tokens are less refined, having been updated fewer times during training. Bostrom and colleagues argue that this is not an inherent limitation of subword tokenisation but a consequence of the specific algorithm — byte pair encoding — and the corpus it was trained on. A tokeniser trained on a more balanced multilingual corpus, or using a different splitting algorithm, would distribute the efficiency more fairly. But changing the tokeniser invalidates every embedding the model has learned, so the decision made at the start — often on an English-heavy corpus — becomes effectively permanent.
Why it mattered then
The choice to use subword tokenisation rather than character-level or word-level encoding was made for practical reasons in the mid-2010s. Character-level models required processing very long sequences, which was computationally expensive and made it harder for the model to capture long-range dependencies. Word-level models had enormous vocabularies and could not handle rare words or morphological variation gracefully. Subword tokenisation, particularly byte pair encoding, offered a middle path: a fixed vocabulary of manageable size, typically thirty to fifty thousand entries, that could represent any text by breaking rare words into familiar pieces. It worked well enough on English that it became the default, and once the default was set, the cost of changing it — retraining embeddings from scratch — meant it persisted even as models grew and multilingual deployment became common. The failures we see now are not failures of the idea but consequences of optimising the tokeniser for a narrow slice of the world's languages and then freezing that choice into the architecture.
Why it matters now
These failures matter more as models are deployed globally and as we ask them to perform tasks that depend on sub-token structure. A model used for translation, content moderation, or customer support in Tamil is both more expensive to run and less accurate than the same model serving English users, for reasons that have nothing to do with the underlying capability of the architecture. The cost multiplier is not small — six times as many tokens means six times the inference cost — and it compounds with the quality degradation that comes from fragmenting words into rare, poorly trained pieces. Arithmetic failures matter because models are increasingly used for tasks that involve numerical reasoning, and the brittleness of tokenised number representations limits what they can reliably do. Character-level failures matter less for most applications, but they surface in spelling correction, code generation, and any task that requires precise manipulation of text. The surprising thing is not that these failures exist but that they are artefacts of a preprocessing decision, not a limitation of the model itself. A different tokeniser, trained on different data, would shift which tasks are easy and which are hard, without changing a single parameter in the model's weights.
The surprising detail
There is a class of tokens that appear in the vocabulary but almost never in the training data, often because the tokeniser and the model were built from different corpora. Their embeddings stay close to random initialisation, and prompting a model with them can produce erratic, sometimes unsettling output. They are called glitch tokens, and they are a direct consequence of the tokeniser being trained separately and then frozen. If you know the token id, you can trigger the behaviour reliably. It is not a bug in the usual sense — the system is working exactly as designed — but it is a failure mode that would not exist if the tokeniser and the model were trained jointly, or if the tokeniser's vocabulary were built from the same distribution as the model's training data.
Remember this
The tokeniser determines what the model finds easy. Failures in spelling, arithmetic and fairness across languages trace back to the same cause: the unit of processing does not match the unit the task requires.
Test yourself
A model performs well at adding two-digit numbers but poorly at multiplying them. Both tasks require the same arithmetic knowledge. Explain why tokenisation makes one harder than the other.
Addition can often be done left-to-right or right-to-left with limited carry, so even if the numbers tokenise inconsistently, the model can sometimes learn patterns that work across different token boundaries. Multiplication requires positional structure — you must track which digit is in the ones place, the tens place, and so on — and tokenisation obscures this structure by splitting numbers into arbitrary fragments. If 487 is one token but 486 is two tokens, the model cannot rely on position within the token sequence to determine positional value. It must first reconstruct the full number from fragments, identify each digit's position, then perform the multiplication. The longer the numbers, the more tokens, and the more this positional information is hidden. This is why models tend to fail more on multiplication than addition, and why accuracy drops sharply as numbers grow longer: the task is not harder arithmetically, but the representation makes it harder to extract the structure the task requires.
Go deeper
- Language Model Tokenizers Introduce Unfairness Between Languages · arXiv · Aleksandar Petrov et al. · 2023-05-17
- Byte Pair Encoding is Suboptimal for Language Model Pretraining · arXiv · Kaj Bostrom et al. · 2020-04-07
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.