II · THE IDEA · ARTIFICIAL INTELLIGENCE
Transformer Positional Encoding Alternatives
▶ Listen · narrated
A transformer trained on short passages can fail on long ones for a reason that has little to do with attention itself: the way it was told where each token sits.
At a glance
- Problem
- Fixed absolute positions limit context length and poor extrapolation
- Transformer-XL
- Relative positional encodings with segment-level recurrence
- ALiBi
- Static linear biases on attention scores; train short, test long
- Gain
- Better length generalisation and simpler positional machinery
Think of a long queue of people. If you name each person by seat number — seat 1, seat 2, seat 3 — those names only make sense in one room of a fixed size. Move everyone into a longer hall, or slide the first half of the queue into the next room, and the seat numbers lie.
Relative position works more like saying “three places to my left” instead of “seat 14”. The relationship stays true when the queue moves. Transformer-XL uses that idea inside attention, and also reuses what it already computed for the previous chunk of text so the model is not stuck inside one short window.
ALiBi goes further. It does not give anyone a seat label at all. When the model decides how much to listen to another token, it simply knocks the score down a little for every step of distance — a straight-line penalty. Near tokens stay loud; far tokens fade. Because the rule is only about distance, it still works when the line of tokens grows longer than anything seen in training.
So the lesson is not that sinusoids were wrong, but that they were one choice. Changing how order is signalled can let a model trained on short text behave more sensibly on long text, often with less positional machinery rather than more.
Vanilla transformers add an absolute positional embedding — sinusoidal or learned — to each token embedding before the stack. Attention then operates on content already mixed with index-specific vectors. That design does not compose cleanly with segment-level caching: reused states carry stale absolute indices, and lengths beyond the trained range require extending or interpolating a positional table the model may not have internalised.
Transformer-XL replaces absolute embeddings with a relative positional encoding inside attention, so scores depend on the offset between query and key. Coupled with segment-level recurrence (caching prior-segment hidden states as extended memory), the model can propagate information across segment boundaries without positional contradiction. The relative term keeps the attention geometry coherent under shifts.
ALiBi removes positional embeddings from the residual stream entirely. For each head h a fixed slope m_h defines a bias b_{i,j} = −m_h · (i − j) added to the pre-softmax logits (with causal masking as usual). Slopes are not learned; they are set by a simple schedule across heads. Training can use a short context; at inference the same linear rule applies to larger i − j, which is the mechanism behind train-short/test-long extrapolation. Cost is a few additions on the score matrix — no extra embedding parameters, no sinusoidal generation at new lengths.
Limitations remain. Relative schemes still interact with depth, head count, and the memory length used at train time. ALiBi’s linear distance prior is strong and language-modelling-friendly but is an inductive bias, not a learned map of arbitrary structure. Neither method by itself fixes quadratic attention cost; they address how position is represented, not the asymptotic expense of full attention.
Look closer
Absolute positions collide with recurrence
Sinusoidal or learned absolute embeddings assign each index a fixed vector. When a model reuses hidden states from a previous segment, those states still carry the old absolute indices. Relative schemes encode distance between query and key instead, so the same content remains valid when the window slides forward.
Bias instead of embedding
ALiBi leaves the token embeddings free of position. A fixed, non-learned penalty is added directly to the attention logits, growing linearly with the distance between query and key. Near tokens keep their scores; distant ones are steadily discouraged, head by head, before the softmax.
Train short, evaluate long
Because the bias is a simple function of distance rather than a table of trained vectors, the same rule applies past the training length. A model can be trained inside a modest context window and, at inference, attend over longer sequences without retraining the positional component.
The story
The original transformer injected position by adding a sinusoidal vector to each token embedding. That choice made order visible to a model that otherwise treats its input as a bag of vectors. It also tied every representation to a fixed index. Once the sequence length exceeded the range seen in training, or once one tried to chain segments together, those absolute signals became a liability rather than a help.
Transformer-XL attacked the fixed-length barrier on two fronts. First, it introduced a segment-level recurrence: hidden states computed for one segment are cached and reused when the next segment is processed, so information can travel further than a single window. Second, it replaced absolute positional embeddings with a relative scheme. Attention then depends on how far apart two positions are, not on which absolute slots they occupy. Relative distances remain meaningful when cached states are shifted into a new segment; absolute indices do not. The combination lets an attentive language model operate beyond a single fixed context without the positional bookkeeping collapsing.
A later line of work asked a narrower question: can length extrapolation be improved with almost no positional machinery at all? ALiBi — Attention with Linear Biases — answers by refusing to add position vectors to the embeddings. Instead, each attention head receives a static bias on its logits. The bias is a linear function of the distance between the query and the key, with a different slope per head. Closer keys are favoured; farther keys are penalised in proportion to how far they sit. Nothing about that rule is learned during training, and nothing about it is bound to a maximum index.
The practical consequence is the pattern named in the paper’s title: train short, test long. A model can be optimised inside a comfortable context length and, at inference, run on substantially longer inputs because the same distance-based penalty still applies. There is no table of positional embeddings to resize, and no sinusoidal schedule that must be extended past the frequencies the model actually saw. Computational cost stays modest: a few additions to the attention scores rather than a second embedding pathway.
Neither approach claims that position has been solved once and for all. Relative encodings still interact with the rest of the architecture, and linear biases are a particular inductive bias — useful for many language-modelling regimes, not a universal law of sequence structure. What both make plain is that the original sinusoidal choice was a design decision, not a necessity. Changing how position enters the attention computation can improve length generalisation and strip away learned parameters that do not generalise cleanly past the training window.
Why it mattered then
Early transformers inherited a hard context limit from their training setup and from absolute positional signals that did not travel well across segment boundaries. Language modelling beyond a fixed window looked expensive or unstable. Transformer-XL showed that relative position plus recurrence could push attentive models past that wall without discarding self-attention. ALiBi later showed that a still simpler intervention — linear biases on the attention scores — could buy extrapolation when the goal was to train cheaply on short sequences and evaluate on long ones. Together they reframed position as an engineering surface rather than a settled component.
Why it matters now
Context windows keep growing, and so does the cost of filling them during training. Schemes that generalise past the lengths seen in training reduce the need to pay for every inference length up front. Relative and bias-based methods also keep the parameter count and the implementation surface smaller than large learned absolute tables. Anyone choosing a backbone, extending a window, or comparing open-weight models is still choosing, implicitly, how position is represented — and that choice still governs whether longer prompts behave like longer versions of the training distribution or like something the model has never quite seen.
The surprising detail
ALiBi’s positional signal is not an embedding at all. No vectors are added to the tokens. Position appears only as a hand-specified linear penalty on attention logits, with fixed slopes per head. The model never learns a positional table, yet length extrapolation improves over methods that do. The cheaper inductive bias outperforms the richer learned one on the very axis — going past training length — that the richer one was supposed to handle.
What is disputed
Reported gains in length extrapolation depend on the task, the training length, and the rest of the architecture. Relative encodings and ALiBi are not interchangeable fixes, and neither paper claims a single scheme is optimal for every modality or objective. Treat the improvement as well-supported for the language-modelling settings studied, not as a universal ranking of positional methods.
Remember this
How position enters attention — absolute vector, relative distance, or linear bias — governs length generalisation as much as depth or width does.
Test yourself
A model uses segment-level recurrence, reusing hidden states from the previous segment. Why do absolute sinusoidal embeddings cause trouble here, and what does a relative scheme change about the attention computation?
Absolute embeddings stamp each state with an index from the segment in which it was computed. When those states are reused beside a new segment, the old indices no longer describe the true layout, so attention receives inconsistent position signals. A relative scheme scores pairs by how far apart query and key are, not by which absolute slots they occupy. Distances remain valid when cached states slide forward, so recurrence and position stay compatible.
Go deeper
- [1901.02860] Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context · arxiv.org
- [2108.12409] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.