II · THE IDEA · ARTIFICIAL INTELLIGENCE
Graph Attention Networks
▶ Listen · narrated
Convolution on graphs usually blends neighbours with fixed weights. Attention replaces those weights with scores computed from the nodes themselves, so the blend can change with the content.
At a glance
- What it is
- Masked self-attention so each node weights its neighbours by feature content
- Core move
- Learned attention coefficients replace fixed normalised adjacency weights
- Multi-head
- Parallel heads, concatenated in hidden layers and often averaged at the output
- Known limit
- Original scoring yields static attention; GATv2 restores dynamic attention
Think of a meeting in which you may only hear the people sitting next to you. The seating plan is the graph: it decides who is allowed to speak to whom. In an ordinary graph convolution, you average those neighbours with fixed weights — perhaps one vote each, or votes scaled by how many connections they have. You never ask whether this neighbour’s message is more relevant to you than that one’s.
A graph attention layer keeps the seating plan but adds a quick judgment. Looking at your features and each neighbour’s features, it assigns a score to every allowed speaker, turns those scores into shares that add to one, and builds your new representation as a weighted mix. Important neighbours push harder; less relevant ones fade for this step. Several such judgments can run in parallel (multi-head attention) and be joined afterward.
Later work found that the original scoring recipe was less flexible than it sounded: the order of operations could make the ranking of neighbours stubbornly stable even when you changed. A revised scorer, often called GATv2, fixes that ordering so your own features can truly reshuffle who matters.
GAT replaces the fixed normalised adjacency weights of a GCN-style update with coefficients α_ij produced by masked self-attention. Node features h_i are linearly mapped to z_i = W h_i. For each edge (j → i) in the (usually self-looped) neighbourhood N_i, an unnormalised score e_ij = LeakyReLU(a^T [z_i ‖ z_j]) is computed with a shared vector a. Softmax over j ∈ N_i yields α_ij, and the update is h'_i = σ(Σ_j α_ij z_j), or the multi-head analogue: concatenation of head-wise sums in hidden layers, often averaging on the final layer.
Because scores exist only on edges, complexity tracks edges rather than n² node pairs, and the same W and a apply to unseen nodes in inductive settings. The original e_ij, however, is not a universal approximator of attention over a neighbourhood. Rewriting the score so the query participates inside the nonlinearity before interacting with each key — the GATv2 form — restores dynamic attention: the arg-sort of keys can change with the query. Practitioners choosing a GAT block for tasks that need query-conditioned neighbour selection should prefer that family of scorers, while still remembering that stacking layers, residual paths and normalisation dominate much of the empirical behaviour on real graphs.
Look closer
The mask is the graph
Attention is not computed over the full node set. For each node i, scores are produced only for j in its neighbourhood (and typically i itself via a self-loop). A softmax then normalises those scores so they sum to one. Edges therefore decide who may influence whom; features decide how much. Two graphs with identical features but different connectivity yield different updates, because the candidate sets differ before any weight is learned.
A shared score, then a local softmax
The original mechanism builds a score from the pair of node features: each feature vector is linearly transformed, the two results are concatenated, and a single feed-forward scorer with a LeakyReLU produces one unnormalised coefficient per edge. Softmax is taken only inside the neighbourhood of i. The same scorer parameters are shared across all edges, which keeps the layer applicable to graphs of varying size and degree, and to inductive settings where new nodes appear at test time.
Static ranking beneath the softmax
A later analysis showed that the original scoring function is less query-dependent than full self-attention suggests. Because of how the linear maps and concatenation are ordered, the ranking of keys can fail to change with the query in the expected way — a regime termed static attention. Swapping the order of operations, as in GATv2, makes the score a universal approximator of attention functions and recovers dynamic attention, where the preferred neighbour can genuinely shift with the query node’s own features.
The story
A recurring design question in graph neural networks is how a node should assemble information from the nodes it touches. Spectral and early spatial convolutions answered with structure-derived weights: a normalised adjacency entry, fixed once the graph is known, blind to what the endpoints contain. Every neighbour of equal degree pulled with equal strength.
Graph Attention Networks give a different answer. They keep the graph as a hard constraint on who may talk to whom, but they let the content of the conversation set the volume. For a target node i, each neighbour j is scored by a shared attentional mechanism that looks at the features of i and j after a common linear transform. A neighbourhood-wise softmax turns those scores into coefficients that sum to one. The updated representation of i is then a weighted combination of the transformed neighbour features. Structure selects the candidates; attention allocates the mass.
The attentional mechanism in the original formulation is deliberately small: a single-layer feed-forward network applied to the concatenated pair of transformed features, with a LeakyReLU nonlinearity. Parameters are shared across edges, so the layer does not grow with graph size and can be evaluated wherever an edge exists. That sharing is what makes the method practical on graphs whose node sets differ between training and test time.
Stability and expressivity are further helped by multi-head attention. Several independent attention heads run in parallel, each with its own transform and scorer. In intermediate layers their outputs are concatenated; at the final layer they are often averaged. The effect is familiar from transformer multi-head attention: different heads can specialise on different relational patterns without forcing a single ranking of neighbours.
The graph still matters. Because attention is masked to the neighbourhood, a node cannot borrow signal from a distant vertex in one hop. Stacking layers expands the receptive field in the usual way, at the cost of the smoothing and over-squashing pressures common to message-passing models. The architectural bet is narrower than full self-attention over all nodes: GAT spends capacity on deciding how to weight existing edges, not on inventing new ones.
A subsequent analysis asked how attentive these networks really are. It showed that the original scoring order — linear transform, then concatenation, then the scorer — can collapse into static attention, in which the ranking of keys does not truly depend on the query. GATv2 reverses a critical ordering so that the query participates inside the nonlinearity before comparison with each key. With that change, the family of functions the scorer can represent becomes strictly richer, and the preferred neighbour can shift with the query’s own features. The lesson is not that the first design failed in practice, but that “attention” on graphs is sensitive to algebraic detail that is easy to overlook when the mask already looks like a transformer.
Why it mattered then
When the original work appeared, graph convolutions were already effective on citation networks, molecular graphs and related benchmarks, yet most operators still mixed neighbours with weights fixed by degree or by a global spectral construction. Attention offered a middle path: keep the sparse, inductive message-passing template that scales with edges rather than dense node pairs, while letting features modulate the mix. Multi-head masked attention was already proving its worth in sequence models; porting a lightweight version to neighbourhoods gave practitioners a drop-in layer that needed no Laplacian eigenbasis and no hand-tuned polynomial filters. The design was simple enough to implement on top of existing sparse gather-scatter kernels, which helped it spread quickly through the toolkits of the time.
Why it matters now
Message-passing backbones remain the default for many graph learning stacks, and attention-weighted aggregation is still one of the standard alternatives to mean or sum pooling over neighbours. The static-versus-dynamic distinction matters for anyone choosing or auditing a GAT variant: if the task requires the importance of a neighbour to flip when the query node’s features change, the original scoring function may not deliver what the name suggests, and a GATv2-style scorer is the safer default. More broadly, the episode is a reminder that masking full self-attention down to a graph is not a free way to inherit every property of transformers; the precise bilinear or feed-forward form of the score still governs what the layer can express.
The surprising detail
Despite the branding, the original GAT scoring function does not compute the kind of query-dependent attention many readers assume. The analysis behind GATv2 shows a regime of static attention in which, for a fixed set of keys, the ranking induced by the scores can be independent of the query. Only after reordering the linear maps and the nonlinearity does the mechanism become a universal approximator of attention over a neighbourhood. The gap is algebraic rather than empirical theatre: a small change in formula, a large change in the function class.
What is disputed
The static-attention critique applies to the original scoring function’s expressive capacity; it does not by itself say that every trained GAT model behaves as if attention were constant. Practical heads can still look selective after learning, even when the function class is more limited than full dynamic attention. Treat static versus dynamic as a property of what the layer can represent, not as a claim about every checkpoint.
Remember this
GAT keeps the graph as a hard mask and learns how much each edge should carry; whether that weighting is truly query-dependent depends on the scorer’s algebra, not on the word attention alone.
Test yourself
A node i has three neighbours whose features are fixed. You change only the features of i and recompute one GAT layer. In the original scoring formulation, why might the ranking of those three neighbours stay the same, and what modification allows the ranking to change with i?
In the original scorer, linear transforms and concatenation are ordered so that the induced ranking over keys can fail to depend on the query — static attention. The neighbours can therefore keep the same order even though i has changed. GATv2-style scoring applies the query inside the nonlinearity before comparison with each key, restoring dynamic attention in which i’s features can reorder the neighbourhood.
Go deeper
- [1710.10903] Graph Attention Networks · arxiv.org
- [2105.14491] How Attentive are Graph Attention Networks? · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.