II · THE IDEA · ARTIFICIAL INTELLIGENCE
Queries, Keys and Values
▶ Listen · narrated
Attention does not compare tokens directly. It transforms each one three times first, creating separate representations for searching, being found, and contributing to the answer.
At a glance
- Query matrix
- Projects each token into a representation of what it is looking for
- Key matrix
- Projects each token into a representation by which it can be found
- Value matrix
- Projects each token into what it will contribute when attended to
- All three
- Applied to the same input, learned during training, separate per head
Imagine a room of experts, each holding a card. You walk in with a question written on your card. Each expert also has two other cards: one describing what questions they can answer, and one containing their actual answer. You compare your question card to every expert's 'what I answer' card, scoring each match. Then you collect a bit of every expert's answer card, taking more from the high-scoring matches and less from the low scorers. You leave with a blended answer. In attention, your question card is the query, the 'what I answer' cards are keys, and the answer cards are values. Every token plays every role: it asks questions of others, advertises what it can answer, and offers content when matched.
Each attention head learns three matrices: W_Q, W_K and W_V, which project the input embeddings into query, key and value spaces of dimension d_k (typically d_model / num_heads). For an input X of shape (seq_len, d_model), we compute Q = X W_Q, K = X W_K, V = X W_V. Attention scores are (Q K^T) / sqrt(d_k), passed through softmax row-wise to yield an attention matrix A of shape (seq_len, seq_len), where A[i,j] is the weight position i assigns to position j. The output is A V, shape (seq_len, d_k). In a multi-head architecture, this happens in parallel across h heads, each with its own W_Q, W_K, W_V, and the outputs are concatenated then projected back to d_model. The scaling by sqrt(d_k) prevents the dot products from growing large in magnitude as dimensionality increases, which would push the softmax into regions of vanishing gradient. Masking, when used, is applied by setting certain scores to negative infinity before the softmax, zeroing out the corresponding attention weights. The three projections are learned end-to-end; their structure emerges from the training objective, not from prior linguistic or architectural constraints.
Look closer
The database analogy holds until the weighting
In a database, you issue a query and retrieve rows where the key matches. Here, you compute a query vector for each token, then compare it against every key vector using a dot product. High dot products mean strong matches. But instead of returning one row, you return a weighted blend of all the value vectors, where the weights come from those dot products after a softmax. Every token attends to every other token to some degree; the question is only how much.
The projections are smaller than the input
If the model's residual stream has dimension 768, each head might project down to dimension 64 for queries, keys and values. This compression is deliberate: it lets multiple heads operate in parallel, each learning a different 64-dimensional subspace in which to perform its comparisons. The outputs are later concatenated and projected back up to 768. The dimension reduction also makes the attention mechanism computationally cheaper than it would be if it operated in the full residual dimension.
The three matrices are learned, not designed
Nothing in the architecture specifies what queries should look for or what keys should advertise. The matrices are initialised randomly and adjusted by gradient descent. Over training, they specialise: one head might learn to match pronouns with their antecedents, another to associate adjectives with nouns. The same input is transformed three different ways because the role of searching, being found, and contributing content are distinct, and conflating them into a single representation would force compromises that weaken all three.
The story
Attention begins with a single sequence of token embeddings. Each embedding is a vector, perhaps 768 numbers. The mechanism needs to decide, for each position, which other positions are relevant and how much each should contribute to the updated representation. It does this by transforming every embedding three times.
The first transformation multiplies each embedding by a learned matrix called the query projection. The result is a query vector, typically smaller than the original—64 dimensions is common. This vector encodes what the token is looking for. The second transformation, the key projection, produces a key vector for each token. This encodes how the token should be found, what it advertises about itself. The third, the value projection, produces a value vector: what the token will contribute if attended to.
Once all three sets of vectors exist, the mechanism compares each query against every key by taking their dot product. A high dot product means the query and key are aligned in the learned space, that the searching token finds the other token relevant. These dot products are scaled by the square root of the key dimension—a normalisation that keeps gradients stable—then passed through a softmax, which converts them into a probability distribution. The result is a set of attention weights for each query: non-negative numbers that sum to one, indicating how much to attend to each position.
Finally, each query's attention weights are used to compute a weighted sum of all the value vectors. This weighted sum is the output of attention for that position. It is a blend of information from across the sequence, with the proportions determined by the learned queries and keys.
The entire operation is differentiable. During training, gradients flow back through the softmax, the dot products, and into the three projection matrices. The matrices learn to create query, key and value spaces in which useful patterns of attention emerge. Nothing is hand-coded about what counts as useful; the loss function, applied to the model's final predictions, is the only teacher.
The database analogy is instructive but inexact. In a database, a query retrieves rows where a key matches some criterion, and you get back the values from those rows. Here, every key is considered, and every value contributes, weighted by a continuous measure of match quality. There is no hard threshold, no single retrieved row. The output is always a mixture.
Why it mattered then
The original Transformer paper introduced queries, keys and values as a way to generalise prior attention mechanisms that had used a single learned comparison. Earlier sequence-to-sequence models with attention typically computed a score between a decoder hidden state and each encoder hidden state directly, sometimes with a single learned weight matrix. The three-projection design made attention symmetric and more expressive: every position could simultaneously search for information and offer it, and the content returned could differ from the representation used to decide relevance. This was not a large conceptual leap—key-value stores and database query languages provided an existing mental model—but the specific choice to learn all three projections jointly, and to apply them in parallel across multiple heads, was new. It allowed the model to learn specialised attention patterns in different subspaces without manual feature engineering. The ablation studies in the original paper showed that removing any one of the three projections hurt performance, confirming that the separation of roles was load-bearing.
Why it matters now
Queries, keys and values remain the standard interface for attention across nearly all transformer variants. Understanding them is necessary for reading attention weights, for interpreting what a head has learned to attend to, and for diagnosing failures. When a model attends to the wrong context—when it hallucinates a fact by blending information from irrelevant positions—the error is visible in the attention matrix, which is computed from queries and keys. Techniques for steering models, such as adding vectors to the residual stream or patching activations, often target the query or key projections specifically, because changing what a token searches for or advertises changes the entire downstream flow of information. The three projections are also the reason attention is expensive: the cost is quadratic in sequence length because every query must be compared to every key. Approximations that reduce this cost—such as sparse attention or linear attention—work by changing how queries and keys interact, not by eliminating them. The separation of query, key and value is so fundamental that it appears in attention mechanisms outside transformers, including in diffusion models and in some reinforcement learning architectures. It has become the default way to implement learned, data-dependent routing of information.
The surprising detail
The query, key and value terminology comes from information retrieval and database systems, but the mechanism behaves quite differently in practice. In a database, keys are discrete and unique; here, every key is compared to every query, and the results are continuous and overlapping. The attention weights are also not sparse by default: even after softmax, most positions receive some non-zero weight. This means every token's output is influenced by almost every other token, though the influence may be small. Empirical work on attention head behaviour has found that heads do not always specialise cleanly: some heads show interpretable patterns—attending to the previous token, or to the subject of a sentence—but others produce diffuse attention weights that are hard to characterise. The three projections give the model the capacity for clean specialisation, but the training process does not always use it. There is also a subtle point about the value projection: because the output is a weighted sum of value vectors, and because those vectors are learned, the model can in principle store arbitrary information in the value space that is only retrieved under specific query-key matches. This makes values a potential site for memorisation, distinct from the attention pattern itself.
Remember this
Queries search, keys are searched, values are returned. The same input is projected three ways because the roles are distinct, and the projections are learned.
Test yourself
You freeze the query and key projections of a trained attention head but continue training the value projection. What capability is preserved, and what is lost?
The attention pattern is preserved: which tokens attend to which other tokens remains unchanged, because that depends only on the query-key dot products. What is lost is the content of what gets communicated. The model can still route information to the right places, but the information being routed is now random or poorly fitted to the task, because the value projection determines what each token contributes when attended to. In practice, this would likely degrade performance severely, because even if the routing is correct, the messages being passed are not. This thought experiment reveals that attention has two separable components: the pattern (where to look) and the content (what to retrieve), and the value projection controls the latter.
Go deeper
- Attention Is All You Need · arXiv · Ashish Vaswani et al. · 2017-06-12
- A Mathematical Framework for Transformer Circuits · transformer-circuits.pub
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.