II · THE IDEA · ARTIFICIAL INTELLIGENCE
Embedding Models
▶ Listen · narrated
If you ask a generative model which of two paragraphs is closer in meaning to a third, it will often guess wrong. The embeddings it uses internally were never optimised for similarity.
At a glance
- Purpose
- Converting text into fixed-length vectors optimised for measuring semantic similarity
- Training method
- Contrastive learning: pull similar sentences together, push dissimilar ones apart
- Typical size
- Smaller than generative models — often hundreds of millions of parameters rather than billions
- Output
- A single vector per input, usually 384 to 1536 dimensions
Imagine a library where every book is represented by a single point in a room. Books about similar topics are placed near each other. If you walk in with a question, you find the nearest point and that is the most relevant book. An embedding model does this with sentences. It reads a sentence and outputs a list of numbers — a vector — that represents its meaning. Sentences that mean similar things get similar vectors. You measure similarity by measuring distance between the vectors, usually with cosine similarity. The model was trained by showing it pairs of sentences: some similar, some not. It learned to place similar pairs close together and dissimilar pairs far apart. This is useful when you want to search a database by meaning rather than by exact words, or when you want to find duplicates, or cluster documents by topic.
An embedding model is typically an encoder-only transformer, often derived from BERT or similar architectures. It tokenises the input, produces contextualised token embeddings through bidirectional self-attention, then pools those embeddings into a single fixed-length vector representing the entire input. Pooling is usually mean pooling across token embeddings, though some models use the [CLS] token's final hidden state. The model is trained with contrastive loss functions: given a batch of sentence pairs, the loss pulls similar pairs closer in cosine similarity and pushes dissimilar pairs apart. Training data can be labelled pairs, weakly-supervised pairs inferred from document structure, or a mix. Sentence-BERT introduced siamese and triplet network structures for this, fine-tuning BERT on Natural Language Inference datasets and Semantic Textual Similarity benchmarks. More recent models use large-scale weakly-supervised contrastive pre-training on billions of text pairs scraped from the web. The output vector is typically 384 to 1536 dimensions. At inference, you embed your corpus once, store the vectors, then embed each query and retrieve nearest neighbours using approximate nearest neighbour search with libraries like FAISS or Annoy. This is faster and more semantically robust than BM25 or TF-IDF for most retrieval tasks, though hybrid approaches that combine lexical and semantic signals often outperform either alone.
Look closer
They do not generate text
An embedding model is usually an encoder-only transformer. It reads the input and produces a vector, then stops. There is no decoder, no sampling, no next-token prediction. This architectural difference is not incidental: training a model to generate coherent text and training it to place semantically similar sentences near each other in vector space are distinct objectives, and pursuing both in one model requires compromise.
Training uses pairs or triplets, not individual sentences
The model sees examples like: this sentence and this paraphrase should be close; this sentence and this unrelated sentence should be far apart. The loss function measures distances in the embedding space and adjusts weights to shrink or widen them. This is contrastive learning. Some methods use weakly-supervised data — pairs scraped from the web where proximity is inferred from context, not hand-labelled. The model never sees a target vector; it learns only from relative distances.
The output is pooled from many token embeddings
The model still tokenises the input and produces one vector per token internally, just as a generative model does. But the final output is a single vector for the entire sentence, usually created by pooling — often mean pooling, sometimes using the special classification token's embedding. That single vector is what you compare with cosine similarity or another distance metric. This is why they are sometimes called sentence embeddings, even though the input can be longer than one sentence.
The story
A generative language model produces embeddings as a side effect of its real work. Every token becomes a vector, and those vectors flow through the network toward the final task: predicting what comes next. The model is never explicitly rewarded for making the embeddings of synonymous sentences resemble each other. If that happens, it is incidental.
This matters when you want to measure similarity. Suppose you have a database of support documents and a user types a question. You want to find the document whose meaning is closest. If you embed both the question and every document using a generative model's internal token embeddings, then average them into sentence vectors, the results are often surprisingly poor. Sentences that mean nearly the same thing land far apart; sentences that share a few topic words but mean different things land close together.
Embedding models are trained specifically to solve this. They are usually smaller encoder-only transformers — no generation, no sampling, just reading and compressing. The training data consists of sentence pairs: some similar, some not. The loss function pulls similar pairs closer in the embedding space and pushes dissimilar ones further apart. This is contrastive learning.
The result is a model that maps sentences to vectors in a space where geometric distance corresponds to semantic similarity. Two paraphrases land near each other. Two unrelated sentences land far apart. You can measure this with cosine similarity: a value near 1 means similar, near 0 means unrelated, and near -1 means opposed, though in practice most trained embedding spaces do not use the negative region much.
The architecture is simpler than a generative model's. There is no decoder, no causal masking, no sampling strategy. The model reads bidirectionally — every token can attend to every other token, including those that come later. This is why the term encoder-only appears: the model keeps the encoder half of the original transformer architecture and discards the decoder half. BERT is the canonical example. Sentence-BERT adapted BERT specifically for sentence embeddings by adding a pooling layer and training with contrastive objectives on sentence pairs.
The training data can come from many sources. Some models use labelled pairs: human annotators mark sentence pairs as similar or dissimilar. Others use weakly-supervised data, inferring similarity from structure. If two sentences appear in the same document, or one is a title and the other is the first sentence of the article, they are probably related. If they come from different documents on different topics, they are probably not. This scales better than hand-labelling, though it introduces noise.
Once trained, the model is fast. Embedding a sentence is a single forward pass, cheaper than generating text token by token. You can pre-compute embeddings for a large corpus, store them, and then search by embedding a query and finding the nearest neighbours. This is the foundation of semantic search.
Why it mattered then
The mismatch between generative models and similarity tasks was evident early. BERT, released in 2018, was a powerful encoder but produced token-level embeddings, not sentence-level ones. Averaging them naively gave poor results on semantic similarity benchmarks. Sentence-BERT, published in 2019, addressed this by adding a pooling layer and training on sentence pairs using siamese and triplet network structures. The improvement was large enough to make semantic search practical at scale. Before this, finding similar documents required either keyword matching, which missed paraphrases, or running a full model inference on every candidate pair, which was too slow for large corpora.
Why it matters now
Embedding models are now infrastructure. Retrieval-augmented generation depends on them: the generative model cannot search a database itself, so an embedding model finds the relevant documents first. Customer support systems, recommendation engines, and code search all use them. The models are small enough to run locally, and many are open-weight. OpenAI, Cohere, and others offer embedding APIs alongside their generative ones, often pricing them separately and cheaper. The separation of concerns — one model for similarity, another for generation — has become standard architecture.
The surprising detail
Embedding models trained on English often transfer surprisingly well to other languages, even when those languages were scarce in the training data. This is not universal translation; it is an artefact of multilingual tokenisers and the fact that semantically similar sentences in different languages often share subword tokens, especially for named entities, numbers, and cognates. Some multilingual embedding models are trained explicitly on parallel corpora — sentence pairs that are translations of each other — which teaches the model to place translations near each other in the embedding space. The result is a single model that can compare sentences across languages, though performance degrades for low-resource languages.
Remember this
Embedding models are trained for similarity, not generation. Use them when distance in vector space matters, not when you need text out.
Test yourself
You have a database of legal documents and you want users to search by describing their situation in plain language. Why is an embedding model a better choice than keyword search?
Keyword search matches only the words that appear. If the user writes "fired without notice" and the relevant document says "terminated without warning", keyword search finds nothing. An embedding model maps both phrases to nearby points in vector space because it learned during training that they mean similar things, even though they share no words. The model generalises over paraphrases, synonyms, and different ways of expressing the same concept. This is why it is called semantic search: it searches meaning, not strings.
Go deeper
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · arXiv · Nils Reimers et al. · 2019-08-27
- Text Embeddings by Weakly-Supervised Contrastive Pre-training · arXiv · Liang Wang et al. · 2022-12-07
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.