II · THE IDEA · ARTIFICIAL INTELLIGENCE
Multi-Head Attention
▶ Listen · narrated
A single attention mechanism could work. The transformer uses eight, or sixteen, or more — not because the architecture demands it, but because parallel specialisation turned out to help.
At a glance
- What it is
- Splitting query, key and value into groups, computing attention separately for each, then concatenating
- Introduced
- Attention Is All You Need, Vaswani et al., 2017
- Typical setup
- Eight or sixteen heads per layer, each operating on a slice of the full representation
- Total parameter count
- Same as single-head attention with the same total dimension
Imagine you are trying to understand a sentence, and you need to pay attention to several things at once: the grammar, the meaning of individual words, and the overall topic. Doing all of that with a single focus is hard. Multi-head attention splits the job. It divides the representation of each word into several pieces, and each piece gets its own attention mechanism. One might learn to focus on nearby words, another on words that are grammatically related, another on rare or important words. After each mechanism does its work, the results are glued back together. The model ends up with a richer understanding than it would get from a single attention operation, because different parts of the mechanism have specialised in different aspects of the input.
Multi-head attention splits the query, key and value representations into h groups along the feature dimension, applies scaled dot-product attention independently to each group, then concatenates the results and applies a final learned linear projection. If the model dimension is d_model and there are h heads, each head operates on d_model / h dimensions. The per-head projections are learned separately, so head i has its own W^Q_i, W^K_i and W^V_i matrices. After computing attention for each head, the h outputs are concatenated and multiplied by W^O. The total parameter count is the same as single-head attention with the same output dimension: h * (3 * (d_model / h) * (d_model / h)) + d_model * d_model simplifies to 4 * d_model^2. The advantage is representational: each head can attend to different positions or relationships. Empirically, some heads learn interpretable patterns — positional, syntactic, or semantic — but many do not. Pruning studies (Voita et al., 2019) show that in a twelve-head model, fewer than half the heads are necessary for maintaining performance, and in some cases removing heads improves results slightly. The mechanism provides capacity for specialisation, but training does not always use it efficiently.
Look closer
The split happens before attention is computed
If the model dimension is 512 and there are eight heads, each head works with a 64-dimensional slice. The query, key and value projections are learned separately for each head, so head three's query matrix is different from head seven's. After each head computes its own attention-weighted sum, the eight results are concatenated back into a 512-dimensional vector and passed through a final learned projection. The total parameter count is the same as it would be for a single attention operation over the full 512 dimensions.
Heads do not have predefined roles
There is no instruction in the architecture that head one should attend to syntax and head two to semantics. The specialisation, where it occurs, emerges during training. Some heads learn to focus on adjacent tokens, others on tokens separated by specific distances, others on particular syntactic relationships. But many heads show no clean interpretable pattern at all, and pruning experiments have shown that a substantial fraction can be removed with minimal loss in performance.
Most of the work is done by a minority of heads
Voita and colleagues found that in a twelve-head translation model, only a small number of heads were responsible for most of the useful attention patterns. Some heads specialised in positional attending — looking at adjacent words or words at fixed offsets. Others focused on syntactic structure. The rest contributed little, and removing them barely affected translation quality. The finding suggests that multi-head attention provides redundancy and a larger search space during training, but the final learned model does not use all the capacity it was given.
The story
Multi-head attention is a way of running several attention operations in parallel, each over a different slice of the representation. The mechanism was introduced in the original transformer paper in 2017, and it has been a fixture of the architecture ever since.
The split happens at the start. If the model's hidden dimension is 512 and the design specifies eight heads, the representation is divided into eight slices of 64 dimensions each. Each head has its own learned projection matrices for query, key and value, so the attention operation in head three is independent of the one in head seven. After each head computes its weighted sum, the eight results are concatenated and passed through a final shared projection matrix, returning the output to the full 512 dimensions.
The total number of parameters is the same as it would be for a single attention mechanism operating over the full dimension. The advantage is not efficiency but specialisation. Different heads can learn to attend to different aspects of the input: one might focus on the previous token, another on syntactic dependencies several words away, a third on semantic relationships that span long distances.
That is the theory. The reality is more complicated. When researchers began probing trained models to see what each head had learned, they found that clean interpretations were the exception. Some heads did show consistent patterns — attending reliably to the next word, or to the subject of a sentence, or to closing punctuation. But many heads had no obvious function at all. Their attention weights were diffuse, their behaviour changed unpredictably across different inputs, and when they were removed the model's performance barely dropped.
Voita and colleagues tested this directly in 2019. They trained a twelve-head translation model, then systematically pruned heads to see which ones mattered. A small number turned out to be essential: heads that attended to adjacent positions, heads that tracked rare words, heads that encoded positional information. The rest could be removed with almost no loss in translation quality. In one configuration, they pruned more than half the heads and saw the model's performance improve slightly, suggesting that the extra heads had been adding noise rather than signal.
This does not mean multi-head attention is a mistake. The redundancy may be useful during training, giving the optimiser more paths to explore and more chances to discover useful patterns. The final model may not need all the heads it was trained with, but the training process may need them to reach that final state. It is also possible that heads which look uninterpretable are doing something useful that the probing methods are not sensitive enough to detect.
What is clear is that the heads do not divide the labour evenly, and most of them do not have roles that a human can name. The architecture provides the capacity for specialisation, and training sometimes produces it, but there is no guarantee and no blueprint.
Why it mattered then
The original transformer paper offered multi-head attention as a way to let the model attend to information from different representation subspaces at different positions. The intuition was that a single attention mechanism might struggle to capture everything: syntactic structure, semantic relationships, positional patterns, rare word handling. Splitting the representation and running parallel attention operations gave the model more flexibility, and the experiments in the 2017 paper showed that it worked. Translation quality improved compared to single-head variants, and the computational cost was the same. The design also made the architecture more parallelisable. Because each head operates independently, all the attention computations for a given layer can run at the same time on suitable hardware. This mattered in 2017, when the transformer was being pitched as a replacement for recurrent networks that had to process sequences one step at a time. Multi-head attention fit the moment: it was theoretically motivated, it improved results, and it played to the strengths of the hardware researchers had available.
Why it matters now
Multi-head attention remains standard in nearly every transformer variant, from language models to vision transformers to protein structure predictors. The mechanism is well understood, the implementations are optimised, and the empirical results are strong enough that there is little pressure to replace it. But the pruning results have changed how researchers think about what the heads are doing. The finding that most heads contribute little has led to architectures that reduce the head count, or share parameters across heads, or route information dynamically rather than splitting it evenly. It has also informed quantisation and distillation strategies: if half the heads can be removed without much loss, then they are a natural target for compression. The interpretability work has been less conclusive. Some heads do have clear roles, and visualising their attention patterns has become a standard diagnostic tool. But the hope that each head would learn a distinct, human-interpretable function has not been borne out. Most heads remain opaque, and attempts to steer model behaviour by editing individual heads have had limited success. The mechanism provides capacity, and training fills some of it usefully, but the internal organisation is messier than the clean parallel structure suggests.
The surprising detail
When Voita and colleagues pruned heads from their translation model, they found that removing certain heads actually improved performance slightly. The effect was small, but it suggests that some heads were contributing noise rather than useful signal — possibly because they had learned to attend to spurious patterns in the training data, or because they were interfering with the more specialised heads. The finding complicates the usual story about model capacity: more parameters do not always mean better results, even when those parameters have been fully trained. It also raises the question of why the architecture includes so many heads if most of them are not needed. One possibility is that the redundancy helps during training, giving the optimiser more chances to find useful patterns, even if the final model does not use all the capacity it was given.
Remember this
Most heads do not have clean interpretable roles, and pruning experiments suggest that a minority do most of the useful work.
Test yourself
If a model has twelve attention heads per layer and you remove six of them uniformly at random, what are two distinct reasons the performance loss might be smaller than you expect?
First, the heads do not contribute equally. Pruning studies have shown that a small number of heads do most of the useful work, so removing six at random is likely to remove several that contributed little. Second, the remaining heads may partially compensate. Attention is computed from learned parameters, and if the model has seen similar patterns during training, the heads that remain may already encode overlapping information. A third reason, less certain: some of the removed heads may have been adding noise, in which case their removal could even improve performance slightly, as Voita and colleagues observed in specific configurations.
Go deeper
- Attention Is All You Need · arXiv · Ashish Vaswani et al. · 2017-06-12
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned · arXiv · Elena Voita et al. · 2019-05-23
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.