II · THE IDEA · ARTIFICIAL INTELLIGENCE
Models Without Positional Encoding
▶ Listen · narrated
Position seemed fundamental: without it, how would a model know that "dog bites man" differs from "man bites dog"? It turns out the answer was already in the architecture.
At a glance
- What was removed
- All explicit positional encodings — sinusoidal, learned, relative, rotary
- What remained
- The causal attention mask, preventing tokens from attending to future positions
- Result
- Models still learned positional information and performed competently on many tasks
- Where it breaks
- Length generalisation — performance on sequences longer than those seen in training
Imagine you are reading a sentence one word at a time, but you are blindfolded for all future words. The first word, you see only that word. The second word, you see the first and second. By the hundredth word, you see all hundred words so far. Even though no one told you which position you are at, you could guess from how much you can see. A token at position one sees one thing; a token at position fifty sees fifty things. That difference in visibility is a clue. Decoder-only transformers, which are blindfolded to future tokens by the causal mask, can learn position from that clue alone. They infer where they are from how much context is visible. It is not as accurate as being told the position explicitly, and it breaks down on very long sequences, but it works well enough that the model can learn order even when no positional encoding is provided. The mask was meant to enforce causality, but it accidentally leaked position.
In a decoder-only transformer, the causal mask is a lower-triangular matrix applied to the attention scores before softmax, setting all positions beyond the current token to negative infinity. This ensures autoregressive generation: token *i* attends only to positions 1 through *i*. The mask is not learned; it is a fixed structural constraint. However, it is also not uniform across positions. Token one attends to a single position; token *n* attends to *n* positions. The number of non-masked positions is a deterministic function of the token's position, which means the mask encodes positional information implicitly. During training, the model's internal representations can learn to exploit this signal. Probing experiments confirm that linear classifiers trained on the hidden states of models without positional encodings can predict token position well above chance, indicating the information is present and linearly separable. The effect is specific to causal attention; bidirectional encoders, where every token attends to every other token, do not exhibit this behaviour because the mask is symmetric. Performance on in-distribution tasks is often comparable to models with explicit encodings, but length generalisation degrades more sharply. The implicit signal from the mask is weaker and noisier than sinusoidal or rotary encodings, and it extrapolates poorly to sequence lengths outside the training distribution. The finding does not make positional encodings obsolete, but it does show that they are not the sole source of positional information in decoder architectures.
Look closer
The causal mask is not symmetric
Token five can attend to tokens one through five. Token one can attend only to itself. That asymmetry means each position sees a different number of prior tokens, and the model can infer relative position from the shape of what is visible. The first token always attends to exactly one thing; the hundredth token attends to a hundred. The pattern is a signal, even though it was never designed as one.
This only works for decoder-only architectures
Encoder-only models, and the encoder portion of encoder-decoder models, use bidirectional attention. Every token sees every other token, so the attention pattern is symmetric and carries no positional information. The finding does not generalise to BERT-style architectures. It is specific to the causal structure of GPT-style models, where the mask itself has a direction.
Performance degrades on longer sequences
Models trained without positional encodings perform reasonably on sequences of similar length to those in training, but their accuracy drops more steeply than position-aware models when tested on longer inputs. The implicit signal from the mask is weaker and noisier than an explicit encoding, and that weakness compounds as the sequence grows. For tasks requiring precise long-range ordering, the omission still costs something measurable.
The story
Positional encoding was introduced with the original transformer as a solution to an obvious problem: self-attention has no built-in notion of order. Shuffle the input tokens and the attention mechanism produces exactly the same output. For language, where "not guilty" and "guilty not" mean opposite things, that seemed unacceptable. So the 2017 paper added sinusoidal encodings, unique vectors mixed into each token's embedding according to its position in the sequence.
Every transformer architecture since has included some form of positional signal. Learned encodings, relative position biases, rotary embeddings — the designs differ, but the assumption held: you must tell the model where each token sits, or it cannot learn anything about order.
Then two independent research groups, working on length generalisation, tried removing positional encodings entirely from decoder-only models. Both expected catastrophic failure. Both found that the models still learned.
The reason lies in the causal mask. In a decoder, token *n* can attend only to tokens one through *n*. That restriction is implemented as a mask: a triangular matrix of zeros and negative infinities applied before the softmax, ensuring that attention weights for future positions become zero. The mask is necessary to prevent the model from cheating during training by looking ahead at the answer.
But the mask is not uniform. The first token attends to one position. The second attends to two. The hundredth attends to a hundred. Each position therefore sees a different number of prior tokens, and that count is a signal. The model can learn to infer relative position from the width of the attention window, even though no explicit position was provided. The causal structure, introduced to enforce autoregressive generation, accidentally encodes the information it seemed to make unnecessary.
The effect is clearest in probing experiments. Train a small classifier on the internal representations of a no-position-encoding model, asking it to predict which position a given token occupies. The classifier succeeds well above chance. The information is present in the activations, recoverable by a linear layer, which means the model learned it during training.
This does not mean positional encodings are useless. Models trained without them perform worse on length generalisation: they struggle more when tested on sequences longer than anything in training. The implicit signal from the mask is noisier and less robust than an explicit encoding. For production systems, removing positional encodings is usually a step backward. But the finding matters because it shows that the architecture is doing more than we thought. The components interact. A constraint introduced for one reason creates a signal exploited for another.
Why it mattered then
The result mattered because it clarified what the architecture was actually learning. Transformers had been described, even by their creators, as position-agnostic except for the explicit encodings. The finding showed that description was incomplete. The causal mask, treated as a mere training constraint, was part of the inductive bias. That realisation led researchers to look more carefully at what other implicit signals might be present — whether layernorm placement, residual connections, or attention head initialisation were contributing information the model could exploit. It also suggested that some of the engineering effort spent on better positional encodings might be addressing a problem the architecture had already partly solved, albeit imperfectly. The boundary between what we designed and what the model discovered from data became less clear.
Why it matters now
The finding remains relevant because it warns against over-interpreting ablation studies. If you remove a component and performance drops, you have learned that the component was useful, but not necessarily that it was doing what you thought. Positional encodings improve length generalisation, but that does not mean they are the only source of positional information. The model has other paths. That matters for interpretability: if you want to understand how a model represents order, you cannot simply point at the positional encodings and stop. It also matters for architecture search. The result suggests that some design choices we treat as essential may be partially redundant, and that redundancy may itself be useful — giving the model multiple weak signals rather than one strong one. Finally, it is a reminder that the inductive biases of transformers are richer and stranger than the components listed in the paper. The interactions matter as much as the parts.
The surprising detail
The result is often overstated in popularisations, which sometimes claim that positional encodings are unnecessary or that transformers "understand order naturally". Neither is quite right. The models do learn positional information from the causal mask, but they learn it less robustly than from explicit encodings, and the effect is specific to decoder-only architectures. Encoders, which use bidirectional attention, do not have an asymmetric mask and cannot exploit this signal. The finding is genuinely surprising, but it is not a licence to discard positional encodings in production systems. It is evidence that our mental model of the architecture was incomplete, not that the component was redundant.
What is disputed
The papers demonstrate that positional information is learned, but they do not fully explain the mechanism by which the model extracts it from the causal mask. Probing classifiers show the information is present in the representations, but whether it is encoded in attention patterns, residual streams, or some interaction between layers remains an open question. The effect is also sensitive to model size and training duration; very small models or undertrained ones may not develop the same implicit positional awareness.
Remember this
The causal mask enforces order during generation, but it also leaks positional information during training. The architecture does more than we thought.
Test yourself
If the causal mask leaks positional information, why does removing explicit positional encodings still hurt length generalisation? What is the explicit encoding providing that the mask does not?
The causal mask tells each token how many positions came before it, but that signal is indirect and noisy. A token at position one hundred sees a hundred prior positions; a token at position two hundred sees two hundred. The model must infer absolute position from that count, and the inference is imperfect. Explicit positional encodings provide a direct, unambiguous signal: a unique vector for each position that does not depend on the model learning to decode the width of the attention window. When the sequence is longer than anything seen in training, the indirect signal from the mask is less reliable, because the model is extrapolating from counts it never encountered. The explicit encoding generalises better because it continues to provide a distinct, consistent signal even at novel positions — though it too eventually degrades. The mask gives the model something to work with, but the explicit encoding gives it something clearer and more stable.
Go deeper
- The Impact of Positional Encoding on Length Generalization in Transformers · arXiv · Amirhossein Kazemnejad et al. · 2023-05-31
- Transformer Language Models without Positional Encodings Still Learn Positional Information · arXiv · Adi Haviv et al. · 2022-03-30
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.