II · THE IDEA · ARTIFICIAL INTELLIGENCE
Attention, Before the Maths
▶ Listen · narrated
You understand a sentence by holding earlier words in mind and deciding which ones matter as new ones arrive. A transformer does something structurally similar, and the mechanism has a name.
At a glance
- What it does
- Lets each token assign weights to all previous tokens, deciding what to attend to
- Origin
- Neural machine translation, 2014–2015, before the transformer architecture
- Core operation
- Compute a score between the current position and every earlier position, then normalise
- Result
- A weighted combination of earlier representations, tailored to what this position needs
Imagine reading a detective novel. When a new clue appears, you mentally flip back through earlier chapters, weighing which earlier details now seem important. You do not reread everything equally; you focus on the bits that connect. Attention is the model's version of that. Each token, as it is processed, looks back at all the tokens that came before and assigns each one a relevance score. Those scores are turned into weights, and the token's representation becomes a mixture of earlier representations, weighted by relevance. "Bank" in isolation is ambiguous. When "steep" arrives, the model can weight "bank" heavily and let the combination clarify the meaning. The weights are not fixed; they are recalculated for every sentence, so the same word can attend to different things in different contexts.
Attention computes a score between a query representation at the current position and a key representation at each earlier position. The scoring function is typically a dot product or a scaled dot product, though other variants exist. The scores are passed through a softmax to produce a probability distribution over positions — the attention weights. Those weights are then used to take a weighted sum of value representations from the earlier positions, and that weighted sum becomes the attention output for the current position. The query, key and value representations are all linear projections of the token embeddings, with learned weight matrices. The mechanism is differentiable end-to-end, so the projection matrices are trained by backpropagation. In a causal decoder, positions can only attend to earlier positions; a mask sets the scores for future positions to negative infinity before the softmax, ensuring their weights are zero. The cost is quadratic in sequence length, because every position scores against every other position. Attention is applied in parallel across all positions during training when the full sequence is available, but at inference time in a generative model, each new token attends to all previous tokens and the attention pattern grows incrementally.
Look closer
The weights are computed, not stored
Attention weights are not parameters that training adjusts directly. They are calculated fresh for every token in every input, based on the current representations. The learned parameters control how representations are transformed before the scoring happens, but the decision about which tokens matter right now is made at inference time, for this sentence. That is why attention can handle dependencies it has never seen before.
Every token looks at every earlier token
In a decoder producing text one token at a time, the model cannot look ahead — that would be cheating, because those tokens do not exist yet. But it can look at every token that came before. The attention mechanism computes a score between the current position and each earlier position, then converts those scores into weights that sum to one. The current token's representation becomes a weighted mixture of what came before, with the weights reflecting relevance.
The same word can attend differently in different contexts
Consider "bank" in two sentences: "The bank was steep" and "The bank was closed." In the first, the attention weights on "bank" will likely emphasise "steep" once that word appears, because the combination disambiguates toward riverbank. In the second, "closed" pulls "bank" toward the financial sense. The mechanism does not store a fixed meaning for "bank"; it computes which surrounding tokens to weight heavily, and that weighting is context-dependent.
The story
Take the sentence "The bank was steep." A human reader holds "bank" in a state of ambiguity — financial institution or riverbank — until "steep" arrives and resolves it. The model needs to do something analogous. When it reaches "steep", it must be able to look back at "bank" and weight that token heavily, because the combination matters. When it processes "was", it might weight "The" and "bank" about equally, because "was" is just continuing the structure. Attention is the mechanism that makes this selective weighting possible.
The idea appeared in neural machine translation before transformers existed. Early sequence models processed input one token at a time, maintaining a hidden state that was supposed to summarise everything seen so far. For short sentences that worked. For longer ones, the hidden state became a bottleneck: too much information compressed into a fixed-size vector. Attention was introduced to let the model look back at the entire input sequence when producing each output word, rather than relying on the summary alone.
The mechanism works by computing a score between the current position and every earlier position. The scores are then normalised into weights that sum to one, and those weights determine how much each earlier token contributes to the representation at the current position. High weight means high relevance; low weight means the token is ignored for this particular decision.
The scoring itself is learned. The model does not arrive knowing that adjectives matter for disambiguating nouns. It learns, from examples, which patterns of co-occurrence are meaningful. After training on enough sentences where "steep" clarifies "bank" and "closed" clarifies it differently, the parameters that compute attention scores will tend to produce high scores for those pairings. The mechanism is general — compute, score, weight, combine — but the specific weights emerge from data.
This is why attention is sometimes described as soft lookup. A hard lookup would retrieve one specific earlier token. Attention retrieves a weighted blend of all of them, with the weights tailored to the current need. Most weights end up near zero, so in practice only a few tokens matter strongly for any given position, but the mechanism does not decide in advance which few. It computes all the scores and lets the data determine the pattern.
Why it mattered then
The problem attention solved was concrete: neural machine translation models were failing on longer sentences because they tried to compress the entire source sentence into a single fixed-size vector before generating the translation. That vector became an information bottleneck. Attention let the model keep the entire source sequence available and decide, for each word it generated, which source words to focus on. The improvement was immediate and measurable. Models with attention produced better translations, especially for long sentences, and the attention weights themselves were often interpretable — you could visualise which source words the model was attending to when it generated each target word, and the patterns frequently matched human intuition about which words should align.
Why it matters now
Attention is the central operation in the transformer architecture, which now underpins almost every large language model. The mechanism that began as a fix for translation bottlenecks turned out to be a more general solution to the problem of context-dependent representation. Transformers apply attention not just once but in many layers, and not just to earlier tokens in the input but within the model's own intermediate representations. The ability to dynamically weight what matters, rather than relying on fixed or sequential processing, is why transformers can handle long-range dependencies and why they parallelise efficiently. Understanding attention is not optional for understanding how these models work; it is the operation that makes them transformers.
The surprising detail
The original attention papers presented the mechanism as an alignment tool: a way to figure out which source-language words corresponded to which target-language words during translation. The weights were a means to an end, and the interpretability was a useful side effect. It was not immediately obvious that the same mechanism, applied to a single language and stacked in many layers, would become the foundation of models that do far more than translate. The generality was discovered, not designed in advance.
Remember this
Attention lets each token look back and decide which earlier tokens to weight heavily. The weights are computed fresh for every input, not stored.
Test yourself
A model is generating text one token at a time. At position fifty, it computes attention weights over the previous forty-nine tokens. Why does the mechanism need to normalise the scores into weights that sum to one, rather than using the raw scores directly?
Normalising to sum to one makes the result a weighted average, which keeps the scale stable and comparable across positions. Without normalisation, raw scores could grow arbitrarily large or small depending on how many tokens came before or how confident the scoring function is, and downstream layers would have to cope with wildly varying magnitudes. The weighted average also has a clean interpretation: you are blending earlier representations in proportions that reflect relevance, and the proportions must account for the whole. Normalisation is typically done with a softmax, which also has the effect of amplifying differences — a score slightly higher than others becomes a substantially higher weight — so the model attends sharply to a few tokens rather than diffusely to many.
Go deeper
- Neural Machine Translation by Jointly Learning to Align and Translate · arXiv · Dzmitry Bahdanau et al. · 2014-09-01
- Effective Approaches to Attention-based Neural Machine Translation · arXiv · Minh-Thang Luong et al. · 2015-08-17
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.