II · THE IDEA · ARTIFICIAL INTELLIGENCE
Position: Attention Has No Sense of Order
▶ Listen · narrated
Attention weights depend on content alone. A question word and its answer could be adjacent or a thousand tokens apart, and the mechanism itself cannot tell the difference.
At a glance
- The problem
- Attention is permutation-equivariant: reorder the input, outputs reorder the same way
- Original solution
- Sinusoidal position encodings added to input embeddings
- Alternative
- Learned position embeddings, now more common in practice
- Frequency range
- Sinusoids from wavelength 2π to 10000·2π in the original paper
Imagine you have a sentence written on index cards, one word per card. If you shuffle the cards, a person reading them will get confused, because word order matters. But the attention mechanism in a Transformer looks only at what each card says, not at where it sits in the deck. It compares every card to every other card and decides how much attention to pay based on content alone. Without something extra, it cannot tell whether "not" comes before or after "good".
Position encodings solve this by writing a number on the back of each card — not the word's meaning, but its place in the line. The model adds that position number to the word's representation before doing anything else. Now the mechanism can tell first from last, because each position has a unique signature mixed into it. The original Transformer paper used a mathematical formula to generate these position numbers, waves that oscillate at different speeds. Most models now just learn the numbers during training, treating position as another lookup table. Either way, the model has to be told where things are, because the architecture itself has no notion of order.
Attention computes scores as the dot product of query and key vectors, then applies those scores as weights to a sum over value vectors. The operation is permutation-equivariant: if you permute the input sequence, the output sequence permutes identically, because no position index appears in the scoring or aggregation steps. To break this symmetry, position information must be injected before attention is computed.
The original paper added position encodings directly to the input embeddings. For a position p and embedding dimension i, the sinusoidal encoding is PE(p, 2i) = sin(p / 10000^(2i/d_model)) and PE(p, 2i+1) = cos(p / 10000^(2i/d_model)), where d_model is the embedding dimension. This produces a unique vector for each position, and the authors noted that PE(p+k) can be expressed as a linear function of PE(p), which might allow the model to learn relative offsets. Empirically, learned embeddings — a simple lookup table from position index to vector, trained end-to-end — performed nearly as well.
Learned embeddings are now more common. They require fixing a maximum sequence length during training, and positions beyond that length have no encoding. Techniques like position interpolation or fine-tuning with longer sequences can extend the range, but they require additional training. Sinusoidal encodings extend indefinitely by definition, but models trained with them do not necessarily generalize well to lengths far beyond training, because the model's learned parameters may not have seen those encoding values.
Relative position representations, introduced by Shaw et al., modify the attention mechanism itself to include a learned offset term that depends on the distance between query and key positions. This removes the need for input-level encodings and can improve length generalization, but it changes the attention equation and requires re-implementing the scoring step. Rotary Position Embeddings (RoPE), used in some recent models, apply a rotation to query and key vectors that encodes position in a way that naturally expresses relative distance through the dot product. The choice of position encoding affects both the model's ability to generalize to longer sequences and the complexity of the implementation.
Look closer
Permutation-equivariance is a precise claim
If you take a sequence of tokens and permute them — swap positions 2 and 5, say, or reverse the entire order — then run attention, the output vectors permute in exactly the same way. The operation has no internal notion of before and after. This is not a bug in early implementations; it follows directly from the mechanism. Attention computes a weighted sum over value vectors, and the weights come from comparing query and key vectors. None of those comparisons involve position indices.
Sinusoidal encodings use different frequencies for each dimension
The original paper defined position encodings as sine and cosine functions of the position, with a different frequency for each dimension of the embedding. Dimension 0 might oscillate every few positions, dimension 1 more slowly, and so on, up to a wavelength of ten thousand positions for the highest dimensions. The result is that each position gets a unique, deterministic vector, and nearby positions have similar encodings. The functions were chosen so that a relative offset between two positions could in principle be computed as a linear transformation of their encodings, though whether models learn to exploit that property remains unclear.
Learned embeddings treat position as just another lookup
Instead of a formula, many implementations now maintain a table: position 0 maps to one learned vector, position 1 to another, up to some maximum length fixed during training. This is simpler to implement and removes the need to justify a particular functional form. The trade-off is that positions beyond the training length have no encoding at all, whereas sinusoidal functions extend indefinitely. In practice, models with learned encodings struggle with sequences longer than they saw during training, though techniques like interpolation or extrapolation have been proposed to mitigate this.
The story
The attention mechanism compares every token to every other token, computes a weight for each pair based on how relevant they are to one another, then uses those weights to mix together value vectors. Nowhere in that process does the position of a token enter the calculation. If you wrote "The cat sat on the mat" and "mat the on sat cat The", the attention scores would be identical — only permuted to match the scrambled input.
This is called permutation-equivariance, and it is a direct consequence of how attention is defined. The query vector for token i and the key vector for token j produce a score by taking their dot product. That score depends on the content of i and j, but not on where they appear in the sequence. The same is true for the subsequent weighted sum over values. Position is simply not a variable in the equations.
For many tasks, this is unacceptable. In language, word order carries meaning. "The dog bit the man" and "The man bit the dog" contain the same tokens, but they describe different events. A model that cannot distinguish them cannot model language. The original Transformer paper acknowledged this and proposed a solution: add a position encoding to each token's embedding before any attention computation begins.
The paper introduced sinusoidal position encodings, vectors computed by evaluating sine and cosine functions at each position, with a different frequency for each dimension. The idea was to give each position a unique signature that varied smoothly, so that nearby positions would have similar encodings. The functions were also chosen with a particular mathematical property in mind: the encoding for position p+k could, in theory, be expressed as a linear function of the encoding for position p, which might allow the model to learn relative position relationships more easily. Whether models actually learn to use this property has not been conclusively demonstrated, and the authors noted that learned position embeddings performed nearly as well in their experiments.
Learned embeddings treat position as a straightforward lookup table. Position 0 gets one vector, position 1 gets another, and so on, up to the maximum sequence length seen during training. These vectors are initialized randomly and updated during training like any other parameter. Most large language models now use learned position embeddings rather than sinusoidal ones, partly because they are simpler and partly because they perform at least as well in practice. The cost is that they do not naturally extend beyond the training length, whereas sinusoidal functions can be evaluated at any position. Techniques for extending learned embeddings to longer sequences exist, but they require additional design choices.
Later work introduced relative position representations, where the attention score between two tokens depends directly on their distance rather than on encodings added at the input. This shifts the position information into the attention mechanism itself, and some recent architectures use this approach instead. The field has not converged on a single best method, and the choice often depends on the trade-offs a particular model is willing to make between simplicity, length generalization, and empirical performance.
Why it mattered then
The Transformer architecture was designed to replace recurrent networks, which processed sequences one token at a time and therefore had position information built into their operation — the hidden state at step t had already seen steps 0 through t-1, in that order. Attention dispensed with sequential processing entirely, allowing every token to attend to every other token in parallel. This brought enormous computational advantages, but it also discarded the implicit position signal that recurrence provided. Without position encodings, early experiments would have failed immediately. A model that could not distinguish "not good" from "good not" could not learn language. The sinusoidal encodings were proposed as a principled solution: deterministic, smooth, and equipped with a mathematical property that might help the model reason about relative distances. The fact that learned embeddings worked nearly as well suggested that the specific functional form mattered less than simply providing any consistent position signal at all.
Why it matters now
Position encodings remain a necessary component of every Transformer-based language model. The architectural choice made in 2017 — to separate position from content and then add them together — is still standard, though the details vary. Models now routinely handle context windows of tens of thousands of tokens, and the question of how position encodings behave at those lengths has become practically important. Learned embeddings require decisions about maximum length and strategies for extending beyond it. Sinusoidal encodings extend indefinitely but may not capture the patterns that matter for very long contexts. Relative position methods, introduced in subsequent work, have gained traction in some architectures because they allow the model to focus on distances rather than absolute positions, which may be more useful for certain tasks. Rotary position embeddings, another variant, encode position by rotating the query and key vectors in a way that naturally expresses relative distance. The fact that multiple approaches coexist and continue to be refined indicates that the problem of representing position in a permutation-invariant architecture is not yet fully solved. It is a small component of a much larger system, but it is one that every model must address, and the choice has measurable consequences for how well a model generalizes to longer sequences than it saw during training.
The surprising detail
The original paper noted that learned position embeddings and sinusoidal encodings produced nearly identical results in their experiments, despite the sinusoidal version having a specific mathematical structure designed to encode relative positions. This suggests that the model may not have been exploiting the theoretical properties of the sinusoidal functions, or that those properties were less important than simply providing a unique, consistent vector for each position. The field largely moved to learned embeddings anyway, which are easier to implement and remove the need to choose frequencies or justify a functional form. It is a reminder that elegant mathematical design does not always translate into empirical advantage, and that the simplest solution that works is often the one that survives.
Remember this
Attention has no built-in sense of order. Position must be added explicitly, or the model cannot tell first from last.
Test yourself
A model is trained with learned position embeddings up to length 2048, then at inference time you give it a sequence of length 3000. Describe what happens to positions 2048 through 2999, and name one technique that has been proposed to mitigate the problem.
Positions beyond 2048 have no learned embedding, because the table only goes that far. Implementations typically either truncate the sequence, refusing to process the extra tokens, or reuse embeddings in some way — repeating the last one, wrapping around to the start, or crashing. None of these options preserve the correct position signal. One mitigation is position interpolation: during fine-tuning, the learned embeddings are stretched to cover a longer range by interpolating between existing positions, so that the model can be adapted to longer contexts without retraining from scratch. Another is to switch to a relative position scheme, which does not depend on absolute position indices and therefore does not have a maximum length. Both require additional training or design work; there is no free extension.
Go deeper
- Attention Is All You Need · arXiv · Ashish Vaswani et al. · 2017-06-12
- Self-Attention with Relative Position Representations · arXiv · Peter Shaw et al. · 2018-03-06
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.