II · THE IDEA · ARTIFICIAL INTELLIGENCE
Embeddings: Meaning as Direction
▶ Listen · narrated
The model never compared "king" and "queen" as strings. It learned their relationship by noticing the contexts they share, then encoded that pattern as relative position in a space with hundreds of dimensions.
At a glance
- What it is
- A learned mapping from token ID to a point in high-dimensional space
- Typical dimensions
- Several hundred to several thousand real numbers per token
- Training signal
- Tokens appearing in similar contexts are pushed closer together
- Result
- Similarity becomes measurable as distance; relationships become directions
Imagine a city where every word is a building, and buildings that serve similar purposes are built near each other. All the grocery shops cluster in one neighbourhood, all the hospitals in another. You never told the city planner what each building was for—you just gave them records of which buildings people visited in sequence. If someone goes to the bank, then the café, then the bookshop, those three buildings get nudged a bit closer together. After watching millions of trips, the planner has built a city where nearness means something: buildings near each other are used in similar ways. That is what an embedding space is. Each word is a point, and the position is learned by watching context. "King" and "queen" end up near each other not because someone labelled them as related, but because they appear in similar sentences.
An embedding is a learned mapping from a discrete token ID to a dense vector in ℝⁿ, where n is typically between 300 and several thousand. The mapping is stored as a lookup table—a matrix where each row corresponds to one token in the vocabulary—and the vectors are learned parameters, updated during training via backpropagation. Early methods like Word2Vec trained embeddings with a shallow neural network using either continuous bag-of-words (predict a word from context) or skip-gram (predict context from a word) objectives. GloVe factorised a co-occurrence matrix, minimising the difference between dot products of embeddings and the logarithm of co-occurrence counts. Modern transformers initialise embeddings randomly and learn them end-to-end as part of the full model, but the mechanism is the same: token ID in, vector out. Similarity is measured by cosine similarity (the cosine of the angle between vectors) or Euclidean distance. The space has no interpretable axes—each dimension is a linear combination of learned features—but the overall geometry encodes distributional similarity. Vector arithmetic occasionally works (king - man + woman ≈ queen) when the training data contains consistent relational patterns, but accuracy is typically 60–70 per cent on analogy benchmarks, not near 100. The embeddings inherit every bias in the training corpus, and tokens that appear rarely or never during training have poorly optimised or random vectors, leading to unpredictable behaviour.
Look closer
The space has no natural axes
Each dimension is a learned parameter, not a hand-labelled feature like "animacy" or "formality". Dimension 47 does not mean anything you could name. What matters is the overall pattern: two embeddings are similar if their vectors point in nearly the same direction, measured by cosine similarity or Euclidean distance. The space organises itself during training so that tokens used in similar contexts end up near each other, and that clustering is what gives the geometry its semantic structure.
Context defines similarity
The training objective for early embedding methods like Word2Vec was to predict a word from its neighbours, or neighbours from the word. If "dog" and "cat" both appear after "the" and before "slept", their embeddings are nudged closer. The definition of similarity is therefore distributional: words are similar if they are used similarly, not because a human labelled them as synonyms. This means embeddings capture usage patterns, including connotation, register and frequency, not just dictionary meaning.
Arithmetic sometimes works, but not reliably
The famous example is "king" minus "man" plus "woman" yielding something close to "queen". When it works, it suggests the space has learned a consistent direction for gender. But the result depends on which embeddings you use, and many analogies fail. Mikolov's 2013 paper reported accuracies around 60–70 per cent on analogy tasks, not near 100. The geometry is real, but it is approximate and task-dependent. Treat vector arithmetic as a demonstration of structure, not a reliable inference method.
The story
Once the tokeniser has converted text into a sequence of integer IDs, the model looks up each ID in a table. The table returns a vector: a list of several hundred or several thousand real numbers. This is the token's embedding. The integers themselves carry no information—ID 5482 is not "close" to 5483 in any meaningful sense—but the vectors do. Two tokens with similar embeddings are, by design, similar in meaning or usage.
The embeddings are learned during training. Early methods like Word2Vec, published by Mikolov and colleagues in 2013, trained embeddings by predicting context: given a word, predict the words around it, or vice versa. If "dog" and "cat" both appear in the same kinds of sentences—after "the", before "ran", near "pet"—the training process nudges their embeddings closer together. The objective is purely statistical, but the result is semantic structure. Words that are used similarly end up near each other in the high-dimensional space.
That nearness is measurable. Cosine similarity treats each embedding as a direction and measures the angle between two vectors; Euclidean distance measures the straight-line gap. Either way, "king" ends up closer to "queen" than to "apple", because the contexts in which monarchs appear overlap far more than the contexts shared by monarchs and fruit.
The famous arithmetic—"king" minus "man" plus "woman" equals something close to "queen"—works when it works because the training data contained consistent patterns. If male and female terms appeared in parallel structures often enough, the space learned a direction that corresponds roughly to gender. Move along that direction from any gendered word and you travel toward its counterpart. But the success rate is not high. Mikolov's 2013 paper reported accuracies of 60 to 70 per cent on analogy tasks, depending on the dataset and the dimension count. The geometry is real, but it is noisy, and it reflects the biases and irregularities of the training corpus as much as any abstract semantic truth.
GloVe, published by Pennington, Socher and Manning in 2014, took a different approach: it factorised a matrix of co-occurrence counts, trying to preserve the ratios of how often words appeared together. The result was similar—a space where similarity is geometry—but the training objective was global rather than local. Both methods, and the many variants that followed, share the same core idea: meaning is use, and use can be encoded as position.
Modern language models do not use static embeddings like Word2Vec or GloVe. They learn contextual embeddings, where the vector for "bank" changes depending on whether the sentence is about rivers or finance. But the principle remains. The first layer of a transformer is still an embedding table, and the rest of the model still operates on vectors in a high-dimensional space where similarity is distance and relationships are directions. The geometry is more complex now, and it is dynamic, but it is still geometry.
Why it mattered then
Word2Vec and GloVe arrived at a moment when natural language processing was dominated by hand-engineered features: part-of-speech tags, dependency parses, named-entity labels. Embeddings offered an alternative. Train on raw text, let the model discover structure, and use the resulting vectors as input to whatever task you cared about. The approach was not entirely new—latent semantic analysis had explored similar ideas decades earlier—but the combination of neural methods, larger corpora and efficient training made it practical. The analogy tasks were a way to demonstrate that the learned space had structure. If you could recover "queen" from "king" minus "man" plus "woman", the model had learned something about gender as a consistent direction, not just a collection of isolated word pairs. The results were mixed, but they were enough to show that unsupervised learning on text could extract relational patterns without explicit supervision. That mattered because labelled data was expensive and task-specific, while raw text was abundant.
Why it matters now
Embeddings are still the first layer of every language model. The mechanism has changed—contextual embeddings replace static ones, and the table is now just the input to a transformer—but the principle is the same. A token ID is looked up, a vector is retrieved, and the rest of the model operates on that vector. The geometry of similarity is still foundational. The limitations matter more now than they did. Embeddings encode the biases of their training data, and those biases propagate through every downstream task. If "doctor" is closer to "he" than to "she" in the embedding space, that pattern will influence the model's behaviour in ways that are hard to correct after the fact. The distributional definition of similarity also means that embeddings conflate meaning with frequency, register and co-occurrence. A word that appears rarely, or only in narrow contexts, will have a poorly trained embedding, and the model's performance on that word will suffer. The arithmetic, meanwhile, has become a cautionary tale. The famous examples work often enough to be memorable, but not reliably enough to be useful. The geometry is approximate, and the directions are not as clean as the demonstrations suggest. Understanding that the space is learned, noisy and corpus-dependent is now more important than celebrating the cases where the arithmetic happens to succeed.
The surprising detail
There is a well-documented failure mode involving embeddings that were initialised randomly but never trained properly because the corresponding tokens almost never appeared in the training data. Their vectors stay close to their random starting values, and prompting a model with them can produce strikingly erratic output. They are sometimes called glitch tokens. The failure is not in the model's later layers—it is in the embedding table itself, where a token that should mean something is represented by a vector that means nothing. The problem is a mismatch between the tokeniser's vocabulary, which was built from one corpus, and the model's training data, which was built from another.
What is disputed
The 60–70 per cent accuracy figures for analogy tasks come from Mikolov's 2013 Word2Vec paper and depend heavily on which analogies are tested, how the vocabulary is filtered, and how many dimensions the embeddings use. The tasks themselves are somewhat artificial—recovering "queen" from "king - man + woman" is a demonstration, not a real-world use case—and performance varies widely across different embedding methods and training corpora. The geometry is real, but the arithmetic is unreliable enough that it should not be treated as a robust inference method.
Remember this
Similarity is geometry. The model never compared words as strings; it learned their relationships by noticing where they appear, then encoded those patterns as distance and direction in a high-dimensional space.
Test yourself
You train embeddings on a corpus of medical journals, then use them in a model that processes casual social media text. Name two distinct ways the mismatch could degrade performance, beyond simple vocabulary gaps.
First, the embeddings will have learned distributional patterns from formal, technical writing, so words that appear in both domains may still have misleading geometry. "Sick" in medical text clusters near pathology terms; on social media it clusters near slang for "excellent". The vector represents the wrong usage pattern. Second, the embedding space will have devoted its capacity to distinctions that matter in medical writing—fine-grained symptom terminology, drug names, anatomical precision—while underrepresenting the informal registers, emotional connotations and cultural references that dominate social media. The geometry itself is mis-shapen for the task, even for words that appear in both corpora.
Go deeper
- Efficient Estimation of Word Representations in Vector Space · arXiv · Tomas Mikolov et al. · 2013-01-16
- glove.pdf · nlp.stanford.edu
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.