Tokens: How Language Becomes Something a Model Can Process

Think of a set of fridge magnets. Not one magnet per letter, and not one per word — a mixed set, where the words you use constantly get their own magnet, and rarer words have to be spelled out from smaller pieces you snap together. "The" is a single magnet. "Tokenisation" might take two or three.

Before a model reads your sentence, it rebuilds it out of these magnets. Then it swaps each magnet for its number in the box — magnet 4383, magnet 991 — because numbers are the only thing the model can actually work with.

That is the whole idea. Two things follow from it, and both are worth carrying around.

The first is that the model does not see letters. It sees the magnets. Asking it how many r's are in "strawberry" is like asking someone to count the letters in a word when they are only allowed to look at two or three tiles that spell it. Often they will know the answer from memory. Sometimes they will not.

The second is that the magnet set was not designed — it was assembled by looking at an enormous amount of text and keeping whichever fragments came up most. So it fits the languages that dominated that text, and fits others badly. Writing in a language the set was not built for means using far more magnets to say the same thing.

Tokenisation maps a string to a sequence of integers drawn from a fixed vocabulary, and it is the boundary between text and tensor. The dominant family is subword segmentation, of which byte pair encoding is the canonical instance.

BPE is trained, not specified. Initialise the vocabulary with the base units — bytes, in modern byte-level variants, which guarantees no out-of-vocabulary input is possible. Count the frequency of every adjacent symbol pair across the training corpus. Merge the most frequent pair into a new symbol and record the merge. Repeat until the vocabulary reaches the target size, typically between roughly 30,000 and 200,000 entries in current models. The learned artefact is the ordered list of merges; encoding applies them greedily in order, so segmentation is deterministic for a given text.

Byte-level BPE is the reason arbitrary Unicode, emoji and malformed input can be represented without special handling: the fallback is always the raw byte. SentencePiece takes a further step by treating the input as a raw stream and encoding whitespace explicitly as a meta-symbol, which removes the dependency on language-specific word segmentation. It also offers a unigram language-model segmenter as an alternative to BPE, which selects a segmentation by likelihood rather than by greedy merge order and admits probabilistic sampling of alternative segmentations during training.

The practical consequences follow from the vocabulary being frozen with respect to the model. Embedding matrices are indexed by token id, so the tokeniser and the model weights are a matched pair; swapping one invalidates the other. Sequence length in tokens — not characters — determines attention cost, which is quadratic in the number of tokens for standard attention, and determines KV cache size, which grows linearly. Fertility, the average tokens per word, varies sharply by language and directly determines both the effective context length and the per-request cost for that language.

Two failure modes are worth naming precisely. Character-level tasks are hard because character identity is not recoverable from a token embedding except insofar as the model has learned it indirectly from text describing spelling. And under-trained tokens — vocabulary entries that survive the merge process but occur near-zero times in the model's pretraining corpus, often because tokeniser and model were trained on different data — retain embeddings close to initialisation and can produce badly out-of-distribution behaviour when prompted.