II · THE IDEA · ARTIFICIAL INTELLIGENCE
Vocabularies, Special Tokens and Chat Templates
▶ Listen · narrated
Download a model from Hugging Face, feed it a prompt, and watch it ignore your instruction or repeat your question back at you. The weights are fine. You used the wrong chat template.
At a glance
- Special tokens
- Reserved ids marking structure: beginning, end, padding, separation between turns
- Chat template
- A formatting recipe that wraps user and assistant messages with the correct tokens and syntax
- Why it matters
- Models are trained on formatted conversations; prompting without that format wastes the training
- Consequence of mismatch
- Instruction-following degrades, output becomes incoherent, or the model simply continues your prompt
Imagine a model as a machine that learned to assemble furniture by watching thousands of instruction manuals. Every manual used the same layout: a title, numbered steps, a parts list at the end. Now you hand the machine a pile of parts and some instructions written as a paragraph of prose. The machine can still read, but it does not recognise this as an instruction manual, so it does not know where to start or what counts as a step. Chat templates are the layout. Special tokens are the visual markers — the bold headings, the step numbers — that the model learned to expect. Without them, the model sees text but not structure, and its behaviour degrades to guessing what comes next rather than following what you asked.
Special tokens are reserved vocabulary entries that do not encode language but mark structural boundaries. Common examples: BOS (beginning of sequence, often id 1), EOS (end of sequence, often id 2), PAD (padding, often id 0), and model-specific tokens like <|im_start|> or [INST]. These ids are added to the vocabulary during tokeniser training and are inserted by the chat template at fixed positions. The template itself is a Jinja2 program stored in tokenizer_config.json under the key chat_template. It takes a list of messages as input and outputs a formatted string. Example: [{"role":"user","content":"Hello"}] might become "<|im_start|>user\nHello<|im_end|><|im_start|>assistant\n". The model was trained on text formatted this way, so the special tokens and role markers are part of the learned distribution. Prompting without them is out-of-distribution input. The model will still generate, but instruction-following, turn-taking, and stopping behaviour degrade because those behaviours were conditioned on the presence of the template structure. Batching requires padding tokens to make sequences the same length; an attention mask then tells the model which positions to ignore. Chat templates are model-specific and not interchangeable. Using the wrong template is a common silent failure mode in local inference.
Look closer
BOS and EOS are not opposites in practice
BOS — beginning of sequence — is often added automatically by the tokeniser, marking the start of any input. EOS — end of sequence — signals that generation should stop. But their behaviour is asymmetric. BOS is usually prepended once, at the very start. EOS appears multiple times in a multi-turn conversation, closing each assistant reply. Some models use a distinct token for end-of-turn rather than reusing EOS. The naming is historical and the behaviour is per-model, so you cannot safely assume symmetry.
Padding tokens exist because batches must be rectangular
When you process multiple sequences at once, they rarely have the same length. The padding token fills the shorter ones so every sequence in the batch reaches the same number of tokens. The model is then told to ignore those positions — usually through an attention mask — so the padding does not affect the output. Padding is invisible during single-sequence generation, but essential for efficient training and batch inference. Its token id is arbitrary; what matters is that the mask marks it correctly.
Chat templates are Jinja2 programs stored in the tokeniser config
The template is a small program, written in Jinja2 templating language, that takes a list of messages and returns a single formatted string. It inserts the special tokens, adds any role labels or XML-style tags the model was trained to expect, and handles system prompts if the model supports them. The template lives in the tokeniser configuration file on Hugging Face, which means it travels with the model. But if you load the weights without the tokeniser config, or write your own prompting code, you bypass it entirely — and the model receives unformatted text it was never trained on.
The story
A language model does not natively understand conversation. It was trained on a continuous stream of tokens, learning to predict what comes next. To make it follow instructions or answer questions, you must first teach it the shape of a dialogue: where the human speaks, where the assistant replies, where one turn ends and another begins.
This is the job of special tokens and chat templates. Special tokens are reserved entries in the vocabulary — integers that do not correspond to words or subwords, but to structural markers. A beginning-of-sequence token might have id 1. An end-of-sequence token might be 2. A padding token, used to fill out shorter sequences in a batch, might be 0. The exact ids vary by tokeniser, and the names are conventional rather than enforced.
During training, the model sees these tokens in consistent positions. Every sequence starts with BOS. Every assistant reply ends with EOS. User and assistant turns are separated by role markers or special tokens that signal the handoff. The model learns these patterns as part of the data distribution, the same way it learns that a full stop usually precedes a capital letter.
A chat template encodes this structure. It is a formatting recipe, usually written in Jinja2 templating language, that lives in the tokeniser configuration file. You give it a list of messages — each with a role like "user" or "assistant" and some text — and it returns a single string with all the special tokens, role labels, and separators inserted in the right places. For a model trained on ChatML format, the template wraps each message in XML-style tags. For a Llama-style model, it might use different tokens and no tags at all. The template is model-specific, because the training data was formatted in a specific way.
When you prompt a model without applying its chat template, you are handing it text in a shape it never saw during training. The model may still generate something — it is a language model, after all, and it can continue any sequence — but the instruction-following behaviour, the turn-taking, the ability to stop cleanly at the end of a reply, all of that was learned on formatted examples. Strip the format and you strip the behaviour. The model might echo your question back, or generate in the style of a web scrape rather than a helpful assistant, or simply ignore the instruction because it does not recognise the structure as a prompt.
Why it mattered then
Special tokens were present from the earliest transformer models, used to mark segment boundaries in tasks like translation or question-answering. BERT used [CLS] and [SEP] tokens to separate sentences. GPT models used a single end-of-text marker. But the proliferation of chat-tuned models from 2022 onward — InstructGPT, ChatGPT, open-weight instruction-following models — made the formatting problem acute. Each lab adopted its own convention. OpenAI used a specific prompt structure internally. Anthropic's Claude models expected a different format. Open-weight models released on Hugging Face each came with their own template, often underdocumented, sometimes contradicting the example code in the model card. The chat template field in the tokeniser config was Hugging Face's attempt to standardise the metadata, so the formatting logic could travel with the model rather than being scattered across forum posts and GitHub issues.
Why it matters now
Every local model you download has a chat template, and using it correctly is the difference between a model that follows instructions and one that does not. Inference libraries like llama.cpp and vLLM now read the template automatically, but many users still write their own prompting code, or copy examples from a different model, and wonder why performance is poor. The problem is silent — no error is raised, the model simply behaves as though it was not instruction-tuned. This matters more as models grow more capable and as users run them locally, outside the guardrails of a hosted API. It also matters for fine-tuning: if you train a model on conversations formatted one way, then prompt it another way at inference, you are testing it on out-of-distribution inputs. The template is not a cosmetic detail. It is part of the model's interface, as load-bearing as the architecture itself.
The surprising detail
The chat template is executable code stored as a string in a JSON configuration file, which means it can contain logic errors, infinite loops, or behaviour that diverges from the model card's examples. Some templates have been released with bugs — missing tokens, incorrect conditionals — that went unnoticed because most users relied on the library's default behaviour rather than inspecting the template itself. There is no formal specification for what a template must do, and no validation beyond whether the Jinja2 syntax parses. The template is also mutable: you can edit it locally, and some users do, either to fix a bug or to experiment with alternative formatting. But once you change it, you are prompting a model in a way it was not trained for, and the results are unpredictable. The template is both essential and surprisingly fragile.
Remember this
The chat template is not optional. If you bypass it, you are prompting the model in a format it never learned.
Test yourself
You load a model and prompt it without applying the chat template. The model generates fluent text but ignores your instruction. Why might the output still be fluent?
The model is still a language model, trained to predict plausible continuations of any token sequence. Fluency does not require instruction-following — it only requires that the next token fit the statistical patterns learned from the training data. The instruction-following behaviour was learned on examples that included specific formatting: special tokens, role labels, turn structure. Without that formatting, the model treats your input as generic text, perhaps resembling a web scrape or a document fragment, and continues it in that style. The weights are unchanged, but the prompt is out of distribution, so the behaviour reverts to the base model's default: fluent, but not obedient.
Go deeper
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.