II · THE IDEA · ARTIFICIAL INTELLIGENCE
Turning Text Into Numbers
▶ Listen · narrated
Every model that appears to read is performing a translation you never see: from the symbols we write to the vectors it actually manipulates.
At a glance
- The problem
- Neural networks operate on arrays of numbers, not strings of text
- The solution
- Map each word to a learned vector of real numbers
- Vector dimension
- Typically hundreds of dimensions per word
- Key property
- Similar words end up near each other in the vector space
Imagine you are organising a library, but instead of shelves you have a large open room and you place each book at a specific coordinate. You put all the cookbooks near each other, all the biographies near each other, and so on. If someone asks for a book about French cooking and you do not have that exact title, you can point them to the region where cookbooks are and they will find something close. Word embeddings work the same way. Each word is placed at a coordinate in a high-dimensional space, and words with similar meanings end up near each other. The network learns these coordinates by training on a simple task—predict nearby words—and the structure emerges from the statistics of how language is actually used.
Word2vec learns a dense vector representation for each word in a fixed vocabulary by training a shallow neural network to predict context. The two main architectures are continuous bag-of-words, which predicts a target word from surrounding context words, and skip-gram, which does the reverse. The network has an embedding matrix as its first layer: each row is the vector for one word. During training, the weights of this matrix are adjusted to minimise prediction error on a large corpus. The result is that words appearing in similar contexts receive similar vectors, measured by cosine similarity or Euclidean distance. The dimensionality is a hyperparameter, typically between 100 and 300. Negative sampling, introduced in the second Word2vec paper, makes training practical by approximating the full softmax with a much smaller set of negative examples per update. Modern transformer models learn embeddings jointly with the rest of the network rather than in a separate pre-training step, and they operate on subword tokens rather than whole words, but the principle of learned dense representations remains central.
Look closer
The representation is learned, not designed
Early approaches assigned words to vectors by hand or by simple rules. Word2vec, published by Mikolov and colleagues in 2013, showed that a shallow neural network trained on a simple prediction task would learn vectors that captured semantic relationships without being explicitly told what those relationships were. The network tried to predict a word from its neighbours, and the internal representation it built in order to succeed turned out to encode meaning in a way that generalised.
Distance in vector space corresponds to meaning
If you measure the angle or Euclidean distance between word vectors, you find that synonyms cluster together and that analogies hold as geometric relationships. The often-cited example: the vector for king minus man plus woman lands near queen. This is not a trick built into the system. It emerges from the statistics of how words co-occur in text, compressed into a space where proximity means something consistent.
Each dimension has no single meaning
It is tempting to imagine that one dimension encodes gender, another encodes royalty, and so on. In practice the dimensions are not interpretable in that way. Meaning is distributed across all of them, and any single dimension participates in encoding many different distinctions. The geometry works, but the individual axes do not correspond to concepts a human would name.
The story
A neural network has no native capacity for text. It is built from layers of weighted sums and nonlinear functions, all of which require numbers as input. If you want the network to do anything with language, you must first solve a representational problem: how do you turn a word into an array of numbers in a way that preserves the information the network will need?
The simplest approach is to treat each word as an atomic symbol and assign it an arbitrary integer. The word cat becomes 47, dog becomes 103, and so on. This is called one-hot encoding when expressed as a vector: a list as long as the vocabulary, with a single 1 in the position corresponding to that word and zeros everywhere else. It works in the sense that it gives the network something to process, but it throws away all structure. The distance between cat and dog is the same as the distance between cat and telescope. The network must learn every relationship from scratch, with no prior hint that some words are more similar than others.
Word2vec, introduced by Mikolov and colleagues in 2013, took a different approach. Instead of assigning each word a fixed, arbitrary vector, it learned vectors by training a shallow neural network on a prediction task. The network was shown a word and asked to predict the words that appeared near it in a large corpus, or it was shown the context and asked to predict the missing centre word. The task itself was simple, but in order to perform it well the network had to build an internal representation that captured something about meaning.
That internal representation was the word embedding: a vector of typically a few hundred real numbers for each word in the vocabulary. Words that appeared in similar contexts ended up with similar vectors. The network had never been told that dog and cat were both animals, but because they appeared near words like pet, fur, and animal in the training text, their vectors ended up close together in the high-dimensional space.
The geometry of this space turned out to have structure that went beyond simple clustering. Relationships between words corresponded to directions in the space. The vector difference between king and queen was roughly the same as the difference between man and woman, or between uncle and aunt. You could perform arithmetic on the vectors and land near sensible answers, not because the system had been programmed with facts about gender or royalty, but because those patterns were implicit in how language was used.
Modern large language models do not use Word2vec directly. They learn their embeddings as part of a much larger network, and they work with tokens rather than words. But the principle remains: text must be converted into vectors before a neural network can process it, and those vectors are learned representations that encode similarity and relationship through their geometry.
Why it mattered then
Word2vec arrived at a moment when neural networks were beginning to succeed at tasks that had resisted earlier machine learning methods, but natural language processing still relied heavily on hand-engineered features. Linguists and engineers would spend months designing rules to capture syntactic structure or semantic categories, and the quality of a system depended on the quality of those rules. Word2vec showed that a simple network trained on raw text could learn representations that outperformed carefully designed features on a range of tasks, and it did so with far less human effort. The method was also fast enough to train on billions of words, which made it practical for real applications. Within a year of publication, word embeddings had become a standard component in nearly every neural language system.
Why it matters now
Every large language model begins by converting tokens into learned vectors, and the principle established by Word2vec underlies that process. The embeddings have grown larger and are now learned jointly with the rest of the model rather than in isolation, but the core idea persists: meaning is encoded as position in a continuous space, and similarity in that space corresponds to similarity in use. Understanding this step is necessary for understanding why models behave the way they do with rare words, why they can generalise from examples, and why certain kinds of reasoning are easier for them than others. It is also the foundation for techniques like prompt engineering and fine-tuning, both of which rely on the fact that the model is navigating a learned geometry rather than applying symbolic rules.
The surprising detail
The analogy examples that made Word2vec famous—king minus man plus woman equals queen—were not part of the training objective. The network was never told to preserve analogies or to encode relationships as vector arithmetic. It was simply trying to predict which words appeared near which other words. The analogical structure emerged as a side effect of compressing the statistics of language into a space where similar contexts produced similar vectors. This remains one of the clearest demonstrations that neural networks can discover structure that their designers did not explicitly specify.
Remember this
Text must become numbers before a network can process it, and the way it becomes numbers determines what the network can learn.
Test yourself
One-hot encoding represents each word as a vector with a single 1 and the rest zeros. Why does this make it harder for a network to generalise, compared to learned embeddings?
Because every word is equally distant from every other word in one-hot space. There is no notion of similarity built into the representation. If the network learns something about the word cat, that knowledge does not transfer to dog or kitten or animal, even though a human would recognise those as related. The network must learn each word's behaviour independently. Learned embeddings solve this by placing similar words near each other, so that a pattern learned for one word partially applies to its neighbours. The geometry itself carries information that the network can exploit.
Go deeper
- Efficient Estimation of Word Representations in Vector Space · arXiv · Tomas Mikolov et al. · 2013-01-16
- Distributed Representations of Words and Phrases and their Compositionality · arXiv · Tomas Mikolov et al. · 2013-10-16
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.