II · THE IDEA · ARTIFICIAL INTELLIGENCE
One Word, Many Meanings
▶ Listen · narrated
A model that assigns 'bank' one fixed vector cannot tell whether you are depositing money or standing beside water. Contextual representation solves this, and the solution is also the clearest reason attention mechanisms exist.
At a glance
- The problem
- One word, many meanings — but static embeddings give each word exactly one vector
- The fix
- Contextual representation: the vector for 'bank' changes depending on the surrounding words
- Key mechanism
- Attention weights let each token's representation incorporate information from others
- First major systems
- ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018)
Imagine you have a notebook where each word gets one page, and on that page you write down everything the word could mean. When you read a sentence, you look up each word and get the whole page, with all the meanings mixed together. You have to figure out which meaning is right from context, but the notebook doesn't help — it gives you the same page every time.
Contextual embeddings work differently. Instead of looking up a fixed page, the model reads the whole sentence first, then decides what each word means in that particular sentence. 'Bank' next to 'river' gets a different representation than 'bank' next to 'money', because the model looks at the neighbours before settling on a meaning. The mechanism that does this looking is called attention: each word asks the others for information, and the ones that are relevant send it back. By the time the process finishes, 'bank' has a vector that reflects which sense you meant, not a compromise between all of them.
Static embeddings — Word2Vec, GloVe, FastText — map each vocabulary entry to a fixed vector, typically learned by predicting co-occurrence patterns in a corpus. For a polysemous word, the learned vector is a weighted average over contexts, positioned to minimise loss across all occurrences. This is optimal for the training objective, but it conflates senses: the vector for 'bank' sits between its financial and geological meanings, and downstream tasks must disentangle them without additional information.
Contextual embeddings compute a different vector for each token occurrence. ELMo uses a bidirectional LSTM: the forward pass encodes left context, the backward pass encodes right context, and the two are concatenated. BERT uses stacked Transformer layers with multi-head self-attention. Each token starts with a static embedding, then at each layer computes a query and uses it to attend over all other tokens' key-value pairs. The attention weights are normalised scores from the dot product of query and key, and the output is a weighted sum of values. After multiple layers, tokens in different contexts have accumulated different information, so their representations diverge.
The mechanism is differentiable end-to-end, so the attention weights and the embeddings they operate on are learned jointly during pre-training. The model is not explicitly supervised to disambiguate polysemy — the training objective is typically masked language modelling or next-token prediction — but contextual separation emerges because it reduces loss. A representation that distinguishes 'bank' senses makes better predictions about surrounding words.
Computational cost scales as O(n²d) per layer for sequence length n and model dimension d, because each token attends to all others. This is the reason context windows are finite and inference is expensive. Approximations exist — sparse attention, linear attention, state-space models — but they trade coverage for speed, and the trade-off is not always favourable.
Look closer
Static embeddings are context-free
Word2Vec and GloVe, the dominant embedding methods before 2018, learn one vector per vocabulary entry. That vector is fixed: 'bank' gets the same 300 or 512 numbers whether it appears in "river bank" or "bank account". The vector is trained to sit near words that co-occur with it in a corpus, so it ends up somewhere between all its senses — a compromise that satisfies none of them well. Polysemy, the property of having multiple meanings, is ubiquitous: 'set' has dozens, 'run' has dozens, even 'good' shifts. A static embedding cannot represent this.
Contextual embeddings are computed, not looked up
ELMo and BERT do not store one vector per word. Instead, they start with a static embedding as a seed, then pass the entire sequence through layers that let each token's representation change in response to the others around it. By the time the representation reaches the upper layers, 'bank' in "river bank" and 'bank' in "bank account" have diverged into different vectors, because the model has routed different information to each one based on context. This is not a lookup; it is a computation that happens every time the sequence is processed.
Attention is the mechanism that makes context flow
To build a contextual representation, the model needs a way for each token to gather information from others. Attention provides this: each token computes a weighted sum over all the other tokens' representations, with the weights determined by learned queries and keys. A token next to 'river' will attend strongly to it, pulling in semantic information that pushes 'bank' toward the geological sense. A token next to 'account' will do the opposite. Attention is often introduced as a general-purpose routing mechanism, but polysemy is the clearest single justification for why it is necessary at all.
The story
The word 'bank' has at least two common meanings in English: a financial institution and the sloping land beside a river. A human reading a sentence knows which sense is intended because the surrounding words make it clear. But a model using static word embeddings — the standard approach before 2018 — assigns 'bank' the same vector in both cases, because the vector is retrieved from a fixed table. That vector was trained on a corpus where 'bank' appeared in both senses, so it ends up positioned somewhere in between, closer to words like 'money', 'river', 'account' and 'shore' than it is to any one meaning. This is a compromise, and it is a poor one. The representation has lost information that was present in the sentence.
The problem is not limited to obvious homonyms. 'Play' means something different in "play a role", "play a game" and "play a recording". 'Record' shifts between noun and verb, and between meanings even within the noun sense. 'Light' can be a noun, verb, or adjective, and each of those branches further. A 2018 analysis of common English words found that polysemy is the norm, not the exception: most frequent words carry multiple meanings, and context is the only reliable way to choose between them.
Contextual embeddings, introduced at scale by ELMo in early 2018 and BERT later that year, compute a different vector for each occurrence of a word. The process starts with a static embedding, but then passes the sequence through multiple layers of a neural network. In these layers, each token's representation is updated based on the representations of the tokens around it. The mechanism that performs this update is attention: each token forms a query, and uses it to compute a weighted sum over the keys and values of all other tokens. Tokens that are semantically relevant to the query receive high weights, so their information flows into the updated representation.
In "I walked along the river bank", the token 'bank' will attend strongly to 'river', and its representation will shift toward the geological sense. In "I deposited money at the bank", the same word attends to 'money' and 'deposited', and the representation moves in a different direction. By the final layer, the two occurrences of 'bank' have diverged into vectors that are no longer interchangeable. The model has used context to disambiguate.
This is not a minor refinement. Contextual representation changed what embeddings could express. Tasks that require understanding which sense of a word is active — question answering, translation, coreference resolution — saw immediate gains when models switched from static to contextual embeddings. ELMo improved the state of the art on six benchmarks when it was released, and BERT improved it further on eleven. The gains were large enough that within two years, static embeddings had mostly disappeared from competitive systems.
The architecture that made this possible was not invented for polysemy. Attention was introduced in 2014 for machine translation, and the Transformer architecture that BERT uses was published in 2017 for the same task. But polysemy is the clearest single motivation for why attention is necessary. If every word had only one meaning, a static lookup table would suffice. Because words shift, the model needs a mechanism to route different information to different occurrences, and attention is that mechanism. The problem is old — linguists have studied polysemy for centuries — but the solution is recent, and it is the foundation of every large language model in use today.
Why it mattered then
By 2018, static word embeddings had been the standard representation for half a decade. Word2Vec, released in 2013, and GloVe, released in 2014, were fast to train and easy to use, and they captured enough semantic structure to power a generation of NLP systems. But progress on tasks that required fine-grained understanding of meaning — reading comprehension, entailment, translation — had slowed. The problem was not the architecture of the models downstream of the embeddings; it was the embeddings themselves. A representation that gives 'bank' the same vector in every context cannot support a system that needs to know which bank you mean. ELMo, published by the Allen Institute for AI in February 2018, demonstrated that contextual embeddings could be trained at scale and dropped into existing architectures with minimal changes. The system used a bidirectional LSTM to compute representations, and the improvements on benchmark tasks were large enough to be unmistakable. BERT, published by Google in October of the same year, used the Transformer architecture instead of an LSTM, and introduced a pre-training method that let the model learn from unlabelled text before being fine-tuned on specific tasks. The combination of contextual representation and large-scale pre-training set a new standard. Within a year, most competitive NLP systems had adopted one or the other. The shift was not only technical. Contextual embeddings made it possible to build general-purpose language models that could be adapted to many tasks, rather than training a separate model for each one. This was the beginning of the pre-training paradigm that now dominates the field.
Why it matters now
Every large language model in use today builds contextual representations. GPT, BERT, T5, LLaMA, Claude, and their descendants all use Transformer layers with attention mechanisms to compute token representations that depend on the surrounding sequence. The specific architectures differ — some are encoder-only, some decoder-only, some both — but the core principle is the same: a token's vector is not retrieved from a table, it is computed from context. This matters for more than disambiguation. Contextual representation is also what allows a model to track information across a long sequence. If you mention a person's name at the start of a paragraph and refer to them with a pronoun two sentences later, the model can route information from the name to the pronoun through the attention mechanism. If you establish a premise and then ask a question about it, the question's representation can attend back to the premise. Static embeddings cannot do this, because each word's vector is independent of where it appears. The cost is computational. A static embedding is a single lookup per token, which is effectively free. A contextual embedding requires passing the entire sequence through multiple Transformer layers, each of which computes attention over all token pairs. For a sequence of length n, this scales as n squared in memory and time, which is why context windows are a constrained resource and why inference is expensive. But the trade is worth making, because a model without contextual representation cannot understand language at the level users now expect. The problem of polysemy is not optional to solve; it is central to meaning.
The surprising detail
The BERT paper reports that the model's attention heads do not cleanly specialise by function. Some heads attend to syntactic structure, some to semantic relationships, and some to positional patterns, but most heads do a mixture, and the mixture changes depending on the input. This was unexpected: the hope in 2018 was that interpretability would come for free, with each head learning a distinct role. Instead, the model distributes the work across heads in ways that are hard to summarise. Polysemy is solved, but the mechanism that solves it is not as modular as the architecture diagram suggests.
Remember this
A word's meaning depends on context, so its vector must as well. Attention is the mechanism that makes this possible.
Test yourself
A static embedding for 'set' is trained on a large corpus and ends up near 'collection', 'group', 'tennis' and 'harden'. Why is this evidence of the problem, not a solution?
The embedding has learned that 'set' co-occurs with all those words, but it cannot represent the fact that different occurrences mean different things. It sits in a compromise position that is close to all the senses but faithful to none of them. When the model encounters 'set' in "a set of keys", it gets a vector that is also pulling toward tennis and hardening, which is irrelevant at best and misleading at worst. The static embedding has recorded the ambiguity, but it has not resolved it. A contextual embedding, by contrast, can attend to 'keys' and shift the representation toward the collection sense, leaving the others behind.
Go deeper
- Deep contextualized word representations · arXiv · Matthew E. Peters et al. · 2018-02-15
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · arXiv · Jacob Devlin et al. · 2018-10-11
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.