II · THE IDEA · ARTIFICIAL INTELLIGENCE
Living in 4,096 Dimensions
▶ Listen · narrated
Every point in a high-dimensional space is almost exactly the same distance from every other point. That fact shapes which similarity measures work and which fail when embeddings try to capture meaning.
At a glance
- What it is
- The vector space where sentence and word embeddings are positioned
- Typical size
- 384 dimensions (smaller models) to 4,096 or more (larger ones)
- Why so many
- More dimensions allow finer distinctions and richer relationships to coexist
- Standard measure
- Cosine similarity, comparing direction rather than absolute distance
Imagine you are standing in a field, and you want to know which direction is north. The distance you have walked from your starting point does not help — you could be a mile away in any direction. What matters is which way you are facing. High-dimensional embeddings work the same way. Each sentence becomes a point in a space with thousands of dimensions, and the system compares them by direction, not by how far out they are. Two sentences that point in nearly the same direction are considered similar, even if one vector is longer than the other. This is what cosine similarity measures: the angle between two vectors. In high dimensions, almost every point ends up roughly the same distance from the origin, so measuring that distance tells you almost nothing. Direction is what survives.
Embeddings from models like Sentence-BERT and the Universal Sentence Encoder live in spaces of 384 to 4,096 dimensions, and the geometry of such spaces differs sharply from low-dimensional intuition. Concentration of measure ensures that the Euclidean distance between almost any pair of randomly distributed points clusters tightly around the mean, with variance that shrinks as dimensionality grows. This makes raw distance nearly useless for distinguishing similar from dissimilar items. Cosine similarity, defined as the dot product of two unit-normalised vectors, measures the cosine of the angle between them and is invariant to magnitude. It returns values in [−1, 1], where 1 indicates identical direction, 0 indicates orthogonality, and −1 indicates opposition. Both Sentence-BERT and the Universal Sentence Encoder are trained with losses that optimise for angular separation: Sentence-BERT uses a triplet or contrastive loss over sentence pairs, and the Universal Sentence Encoder is trained on a variety of supervised and unsupervised tasks that reward similar sentences having high cosine similarity. In deployment, embeddings are often normalised to unit length, making cosine similarity equivalent to a dot product and enabling efficient approximate nearest-neighbour search with libraries like FAISS or Annoy. The high dimensionality allows many orthogonal subspaces to coexist — topic, syntax, sentiment, style — but it also means that the curse of dimensionality applies: exact nearest-neighbour search scales poorly, and approximate methods trade recall for speed.
Look closer
Almost everything is nearly the same distance apart
In three dimensions, points spread out: some are close, some far. In 4,096 dimensions, almost all pairwise distances cluster tightly around the mean. If you pick two random points, their Euclidean distance will be very close to the distance between any other random pair. This phenomenon, called concentration of measure, means that absolute distance becomes nearly useless for distinguishing similar from dissimilar items. The variance collapses.
Corners disappear, and nearly all the volume is in a thin shell
A high-dimensional sphere has almost no volume near its centre and almost all of it near the surface. The corners of a high-dimensional cube are so far from the centre that a sphere inscribed in the cube occupies a vanishingly small fraction of the cube's volume. Intuitions from 3D — that objects fill space evenly, that corners are nearby — break down completely. For embeddings, this means that the useful signal lives in directions, not in how far out you are.
Cosine similarity ignores magnitude and measures angle
Cosine similarity computes the cosine of the angle between two vectors, returning 1 for identical direction, 0 for perpendicular, and −1 for opposite. Because it normalises out the length of each vector, it is unaffected by the concentration of distance that makes Euclidean distance so uninformative in high dimensions. Sentence-BERT and the Universal Sentence Encoder both produce embeddings designed to be compared this way, and retrieval systems almost universally rely on it.
The story
When a model produces an embedding for a sentence, it returns a list of several hundred or several thousand floating-point numbers. Each number is a coordinate, and together they specify a point in a space with that many dimensions. The Universal Sentence Encoder, published by researchers at Google in 2018, produces 512-dimensional vectors. Sentence-BERT, introduced by Nils Reimers and Iryna Gurevych the following year, offers models ranging from 384 to 768 dimensions, with some variants going higher.
The choice of dimensionality is a trade-off. More dimensions allow the model to encode finer distinctions: nuances of syntax, topic, sentiment and style can each occupy their own subspace without interfering. Fewer dimensions are faster to compute with, cheaper to store, and sometimes generalise better because they cannot memorise as much irrelevant detail. But the number chosen is always large by the standards of anything we can visualise.
That size brings consequences that are not intuitive. In three dimensions, if you place points at random in a sphere, some will be near the centre, some near the surface, and the distances between pairs will vary widely. In 4,096 dimensions, almost every randomly placed point sits near the surface, and almost every pair is separated by nearly the same distance. The distribution of distances becomes a tight spike rather than a spread.
This is concentration of measure, a property of high-dimensional geometry proved rigorously in the mid-twentieth century but still surprising to encounter. It means that if you measure similarity by Euclidean distance — the straight-line length between two points — you will find that everything looks equally far from everything else. The differences that remain are so small, relative to the absolute distances, that noise and rounding error can swamp the signal.
Cosine similarity solves this by ignoring distance altogether. It measures the angle between two vectors instead: two embeddings that point in nearly the same direction receive a score close to 1, even if one vector is longer than the other. Two embeddings that point in opposite directions score −1. Two that are perpendicular score 0. Because direction is preserved even when magnitude becomes uninformative, cosine similarity remains discriminative in high dimensions.
Both Sentence-BERT and the Universal Sentence Encoder are trained with this measure in mind. Sentence-BERT uses a siamese network structure, training the model so that sentences with similar meanings are pushed toward similar directions in the embedding space. The Universal Sentence Encoder, available in two variants — one based on a transformer architecture, one on a deep averaging network — likewise optimises for angular similarity. Retrieval systems that search through millions of embedded documents almost always rank results by cosine similarity, and the embeddings are often normalised to unit length before storage, which makes cosine similarity equivalent to a simple dot product and faster to compute.
The geometry matters because it shapes what the system can and cannot learn. If the space were three-dimensional, there would not be room for the many orthogonal distinctions a language model needs: topic, formality, sentiment, tense, negation, and countless others. In thousands of dimensions, these distinctions can coexist without interfering, each occupying its own subspace. But the price is that our spatial intuition — built for three dimensions — becomes a poor guide, and the tools we use must account for that.
Why it mattered then
The shift to high-dimensional embeddings followed from the practical failure of earlier methods. Bag-of-words representations and early topic models operated in spaces whose dimensionality was tied to vocabulary size or to a small number of hand-tuned topics, and they could not capture the relationships between words or sentences in a way that generalised. Word2vec, published in 2013, demonstrated that a few hundred dimensions were enough to encode useful semantic relationships if the vectors were learned from data, and the approach scaled. By 2018, when the Universal Sentence Encoder appeared, the infrastructure for training and deploying high-dimensional embeddings had matured, and the benefits were clear: models could retrieve relevant documents, cluster similar sentences, and measure semantic similarity with a reliability that earlier methods could not approach. Sentence-BERT, a year later, showed that you could fine-tune a BERT model to produce sentence embeddings efficiently, without the quadratic cost of comparing every pair during training. Both papers treat cosine similarity as the default measure, and neither argues for it at length — by then, it was understood that high-dimensional Euclidean distance was not fit for purpose.
Why it matters now
High-dimensional embeddings are now the standard representation for text in retrieval, ranking, clustering, and semantic search. Every major vector database — Pinecone, Weaviate, Qdrant, Milvus — stores embeddings in hundreds or thousands of dimensions and ranks results by cosine similarity or its equivalent. The infrastructure assumes it. When you ask a retrieval-augmented generation system to find relevant context for a query, the search happens in this space, using these measures. The dimensionality also determines cost: storing a billion 1,536-dimensional vectors requires more than five terabytes, and approximate nearest-neighbour algorithms trade accuracy for speed because exact search does not scale. Understanding why the space is high-dimensional, and why cosine similarity is the tool of choice, is not optional context. It explains why some queries succeed and others fail, why certain kinds of semantic distinctions are easy for a model to learn and others are hard, and why magnitude is routinely discarded in favour of direction.
The surprising detail
One consequence of concentration of measure is that the nearest neighbour and the farthest neighbour of a randomly chosen point are, in expectation, almost the same distance away in high dimensions. This is not a marginal effect. The ratio of the farthest to the nearest distance approaches 1 as dimensionality increases, meaning that the concept of a neighbourhood — points that are distinctly closer than others — becomes fragile. Retrieval systems work because embeddings are not randomly distributed; they are trained to cluster by meaning. But the background geometry is hostile, and a small amount of noise or a poorly trained embedding can place a point where almost nothing is meaningfully close and almost nothing is meaningfully far.
Remember this
High-dimensional spaces concentrate distance and dissolve magnitude. Cosine similarity survives because it measures direction, and direction is what embeddings are trained to align.
Test yourself
You are comparing two embeddings and you find that their cosine similarity is 0.95, but their Euclidean distance is very large. What does this tell you about the vectors, and why is the cosine similarity still the more useful number?
It tells you that the two vectors point in nearly the same direction — they agree on the semantic content — but one or both have a large magnitude. Cosine similarity is more useful because it isolates the directional component, which is where the trained signal lives. Euclidean distance is inflated by magnitude, which in high-dimensional embeddings is often an artefact of training dynamics or normalisation choices rather than a meaningful property. If the embeddings were normalised to unit length, the Euclidean distance would be small too, but the cosine similarity already gives you the information that matters: the two sentences or documents are semantically close.
Go deeper
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · arXiv · Nils Reimers et al. · 2019-08-27
- Universal Sentence Encoder · arXiv · Daniel Cer et al. · 2018-03-29
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.