II · THE IDEA · ARTIFICIAL INTELLIGENCE
Rotary Position Embeddings
▶ Listen · narrated
The transformer needs to know word order, but adding position as a fixed vector wastes an opportunity: what matters is usually how far apart two words are, not their absolute coordinates.
At a glance
- What it encodes
- Position as a rotation angle applied to query and key vectors
- Key property
- Attention score between two tokens depends only on their relative distance
- Where it acts
- Applied to queries and keys before the attention dot product, not to values
- Adoption
- Used in LLaMA, PaLM, GPT-NeoX and most models trained after 2022
Imagine three people standing around a maypole, each holding a ribbon. Person one stands at the zero-degree mark, person two has walked a quarter turn clockwise, and person three has walked halfway around. Now measure the angle between person one's ribbon and person two's: it is 90 degrees. Measure between person two and person three: also 90 degrees. The angle between any two people depends only on how far apart they walked, not on where they started. Rotary position embeddings do something similar in high-dimensional space. Each token's query and key vectors are rotated by an angle proportional to the token's position in the sequence. When the model computes attention, it takes the dot product of two vectors, and that dot product depends on the angle between them. Because both vectors have been rotated by amounts proportional to their positions, the angle between them is proportional to the distance between the tokens. The model therefore learns patterns based on relative distance—how many tokens apart two words are—rather than their absolute positions in the sequence.
RoPE applies a rotation matrix to the query and key vectors before the attention dot product. The embedding dimension is divided into pairs, and each pair (d_i, d_{i+1}) is treated as a two-dimensional subspace. For a token at position m, the rotation matrix for the i-th pair rotates by angle m·θ_i, where θ_i is a base frequency chosen for that pair. The standard choice is θ_i = 10000^(-2i/d), creating a geometric progression of frequencies across dimension pairs.
The rotation for a pair is the standard 2D rotation matrix: [[cos(m·θ_i), -sin(m·θ_i)], [sin(m·θ_i), cos(m·θ_i)]]. When applied to both query and key, the attention score q^T k becomes a function of (m - n)·θ_i for each dimension pair, where m and n are the positions of the query and key tokens. This makes the score dependent on relative distance m - n, not on absolute positions.
The method requires no learned parameters beyond the query and key projection weights already present in the transformer. The computational cost is one rotation per dimension pair per token, which is a small additive cost compared to the attention mechanism itself. The rotation is applied after the linear projection that produces queries and keys, and before the scaled dot-product attention.
One practical limitation: the periodic nature of rotation means that very long distances can alias if the frequency is too high. For a dimension pair with frequency θ, two tokens separated by 2π/θ positions will produce identical rotation angles. This is why the base frequencies decrease geometrically: low-frequency pairs handle long-range dependencies without aliasing, while high-frequency pairs provide fine resolution for nearby tokens.
Look closer
The rotation happens in paired dimensions
The embedding vector is divided into pairs of dimensions. Each pair is treated as a point in a two-dimensional plane and rotated by an angle θ that depends on the token's position. Different pairs rotate at different base frequencies, so dimension pairs 0–1 might rotate slowly across the sequence while pairs 2–3 rotate faster. This creates a unique rotation signature for each position, encoded entirely in the direction of the vector rather than added as separate numbers.
The dot product extracts relative distance automatically
When you rotate the query at position m by angle θ_m and the key at position n by angle θ_n, their dot product depends on the difference θ_m minus θ_n. That difference is proportional to m minus n, the relative distance between the tokens. The model never sees absolute positions explicitly; it recovers spacing through the geometry of rotated vectors. This is why RoPE is called relative: the attention mechanism naturally attends based on how far apart two tokens are, not where they sit in the sequence.
Each frequency band captures a different scale of distance
The base frequencies are chosen so that low-frequency pairs rotate slowly and encode long-range dependencies, while high-frequency pairs rotate quickly and distinguish nearby tokens. The original RoFormer paper uses frequencies that decrease geometrically across dimension pairs, creating a spectrum of positional resolution. A token ten positions away produces a small phase shift in the slow dimensions and a large shift in the fast ones, giving the model information at multiple scales simultaneously.
The story
Transformers are famously position-agnostic. Without help, the attention mechanism treats a sequence as a bag of tokens, unable to distinguish the first word from the last. Early solutions added a position embedding to each token embedding before processing began. These embeddings were either learned vectors or fixed sinusoidal patterns, and they worked, but they encoded absolute position: token five always got the same positional signal regardless of context.
Rotary position embeddings, introduced in the RoFormer paper by Jianlin Su and colleagues in 2021, take a different approach. Instead of adding position information to the token embedding, RoPE rotates the query and key vectors by an angle that depends on position. The rotation is applied in two-dimensional subspaces: dimensions zero and one are rotated together, dimensions two and three are rotated together, and so on through the embedding. Each pair rotates at a different base frequency.
The geometry does the work. When the model computes attention, it takes the dot product of a query and a key. If the query has been rotated by θ_m and the key by θ_n, the dot product depends on θ_m minus θ_n. Because the rotation angle is proportional to position, that difference θ_m minus θ_n is proportional to the distance m minus n between the two tokens. The attention score therefore depends on relative distance, not absolute position.
This has practical consequences. A model using RoPE can generalise to sequence lengths it never saw during training, because the mechanism does not rely on learned position indices. If it learned that adjacent tokens often attend to each other, that pattern holds whether the tokens are at positions three and four or positions ten thousand and ten thousand and one. The rotation simply continues.
The method also avoids adding extra parameters. Absolute position embeddings require a learned vector for every position up to the maximum sequence length. RoPE requires only the choice of base frequencies, which are set once and applied through trigonometric functions. The cost is a rotation operation applied to queries and keys before each attention computation, but that cost is modest and the operation parallelises well.
Why it mattered then
The RoFormer paper appeared in 2021, during a period when extending context windows had become a priority. Models trained on sequences of a few hundred tokens were being asked to handle thousands, and absolute position embeddings did not transfer gracefully. A model with learned embeddings for positions zero through 512 had no representation for position 513; researchers resorted to interpolation or extrapolation, neither of which worked reliably. Relative position methods existed before RoPE, but most added complexity. Some required learning a separate set of biases for every possible relative distance, which scaled poorly. Others modified the attention mechanism itself, making implementation less straightforward. RoPE offered relative position encoding through a geometric operation that required no learned parameters beyond the model's existing query and key projections, and that could be added to any transformer with minimal code changes. The method was mathematically clean and computationally cheap, which mattered for models whose size and cost were growing quickly.
Why it matters now
RoPE has become the dominant position encoding method in large language models trained since 2022. LLaMA, PaLM, GPT-NeoX and their descendants all use it. The reason is partly its elegance, but mostly its behaviour at scale. Models using RoPE generalise to longer contexts more reliably than those using learned absolute embeddings, and the mechanism does not require retraining when the target context length increases. A related technique called positional interpolation, introduced by Chen and colleagues in 2023, extends RoPE further by scaling the rotation frequencies down when a longer context is needed. This allows a model trained on sequences of a few thousand tokens to handle tens of thousands with only a brief fine-tuning phase. The combination of RoPE and interpolation has become the standard approach for models that must process long documents, and it is one reason why context windows have grown from a few thousand tokens in 2022 to over a hundred thousand in some models by 2024.
The surprising detail
The rotation is applied only to queries and keys, not to values. This asymmetry is deliberate. The attention mechanism uses the dot product of queries and keys to decide what to attend to, and that is where position information is needed. The values themselves carry the content that gets aggregated, and adding positional rotation to them would interfere with that aggregation. The result is that position information influences which tokens attend to each other, but not what information flows once attention has been computed. This separation between routing and content is central to how the transformer works, and RoPE respects it.
Remember this
Position becomes rotation angle. The dot product of rotated vectors depends only on the angle between them, which encodes relative distance, not absolute location.
Test yourself
A model using RoPE was trained on sequences up to 2,048 tokens. You now ask it to process a sequence of 4,096 tokens without any fine-tuning or interpolation. Describe one specific way its behaviour is likely to degrade, and explain why that degradation happens in terms of the rotation mechanism.
The model will likely struggle with long-range dependencies beyond the 2,048-token range it saw during training. Here is why: each dimension pair in RoPE rotates at a fixed base frequency, and higher-frequency pairs complete multiple full rotations within the training length. When the sequence doubles, those high-frequency dimensions continue rotating and may complete additional full cycles, causing tokens at distance 3,000 to produce similar rotation angles—and therefore similar attention patterns—to tokens at distance 1,000. The model loses the ability to distinguish distances it never encountered, because the periodic nature of rotation causes aliasing. Lower-frequency dimensions fare better because they rotate slowly enough that even the longer distances remain within their useful range, but the model loses the fine-grained distance resolution that the high-frequency dimensions provided during training.
Go deeper
- RoFormer: Enhanced Transformer with Rotary Position Embedding · arXiv · Jianlin Su et al. · 2021-04-20
- Extending Context Window of Large Language Models via Positional Interpolation · arXiv · Shouyuan Chen et al. · 2023-06-27
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.