II · THE IDEA · ARTIFICIAL INTELLIGENCE
Self-Supervised Learning of Visual Features via Contrastive Loss
▶ Listen · narrated
Labels are scarce; unlabelled images are not. Contrastive self-supervision turns ordinary photographs into training signal by asking only whether two views came from the same image.
At a glance
- Task
- Pull two augmented views of one image together; push other images apart
- Labels used
- None during pretraining — only the fact that two views share a source image
- Loss family
- Normalised temperature-scaled cross-entropy (NT-Xent) in a projection space
- Encoder kept
- Base network retained; the projection head is discarded after pretraining
- Semi-supervised
- Larger pretrained models transfer more cleanly when few labels are later added
Think of a party where nobody wears name tags. Each guest arrives twice, in two different outfits. Your job is to stand people so that the two outfits of the same guest end up close together, while everyone else stands farther away. You never hear anyone’s name — only whether two appearances belong to one person.
Contrastive visual learning does the same with photographs. The computer takes one image, messes it up in two different ways (crop, recolour, and so on), and treats those two versions as a matching pair. All the other images in the current batch are non-matches. A network learns to place matches near each other in a numerical map and non-matches farther apart. After this training, the map’s coordinates become features that later tasks — classifying objects, for example — can use, often with only a thin layer of real labels added at the end.
SimCLR builds minibatches of N images and applies a stochastic augmentation family twice per image, yielding 2N views. A base encoder f (e.g. ResNet) maps each view x to a representation h = f(x). A projection head g, typically a one-hidden-layer MLP, maps h to z = g(h). The loss is NT-Xent on l2-normalised z vectors: for an anchor i and its positive partner j, the logit is sim(z_i, z_j)/τ, and the denominator sums exp(sim(z_i, z_k)/τ) over the other 2N−1 indices k in the batch. Gradients update f and g jointly; after pretraining g is discarded and h is the transferable embedding.
Key levers: stronger colour distortion and cropping than supervised defaults; temperature τ controlling hardness of the softmax; batch size setting the number of in-batch negatives (no memory bank required). Linear evaluation freezes f and trains a logistic classifier on h; fine-tuning updates f end-to-end. SimCLRv2 scales the same objective with larger encoders and deeper projection heads during pretraining, then uses a three-stage semi-supervised path — unsupervised pretrain, fine-tune on few labels, self-distill into a student — showing that capacity compounds when labels are scarce.
Limitations: performance is sensitive to augmentation policy and batch size; very small batches weaken the negative set; the method assumes views share semantics under the chosen transforms, which can fail for domain shifts where colour or crop destroy identity.
Look closer
The pair is made by augmentation, not by class
A single image is passed through a stochastic augmentation pipeline twice. Crops, colour jitter, and related transforms produce two views that still share identity but differ in appearance. Those two views form the only positive pair. Every other image present in the batch supplies negatives. The learning signal is therefore the co-occurrence of two views under the same source image, not a human-provided category name.
Agreement is measured after a throwaway head
The base encoder maps each view to a representation vector. A small multilayer projection head then maps that vector into a lower-dimensional space where the contrastive loss is applied. After pretraining, the projection head is discarded and the encoder’s representation is what downstream tasks actually use. The papers find that this nonlinear head improves the quality of the retained representation even though the head itself is not kept.
Temperature and batch size shape the push
The loss normalises embeddings and scales their similarities by a temperature parameter before a softmax over the batch. Lower temperature sharpens the distribution and focuses the model on harder negatives. Because negatives are drawn from the current batch, larger batches supply more negatives per positive pair, which the framework exploits without a memory bank or specialised architecture.
The story
Self-supervised contrastive learning of visual features begins from a deliberately simple question: given two transformed versions of a photograph, can a network tell that they came from the same source, and separate them from views of every other photograph in the batch?
The pipeline has three moving parts. First, a stochastic data-augmentation module draws two correlated views of each image. The composition of those augmentations matters more than any single transform; random cropping combined with strong colour distortion is especially effective, and the framework benefits from heavier augmentation than is typical in fully supervised training. Second, a base encoder — commonly a residual network — embeds each view. Third, a lightweight projection head maps those embeddings into the space where a contrastive objective is computed.
That objective, a normalised temperature-scaled cross-entropy loss, treats the two views of the same image as a positive pair and every other view in the minibatch as a negative. Cosine similarity is computed between l2-normalised embeddings, divided by a temperature, and turned into a classification problem: for each anchor, identify its true partner among the other samples. Gradient descent therefore pulls matched views together in embedding space and pushes unmatched views apart. Once training finishes, the projection head is thrown away. Downstream classifiers and transfer tasks read the encoder representation, not the space in which the loss was applied.
The simplicity is intentional. The framework does not rely on a memory bank of past embeddings, a specialised architecture, or hand-designed pretext tasks beyond the contrast itself. What it does rely on is scale in the batch: more simultaneous negatives make the discrimination harder and the learned features more useful. Unsupervised pretraining under this recipe can match or approach supervised baselines on standard visual recognition benchmarks when a linear classifier is later trained on the frozen representation, and fine-tuning the whole network widens the advantage further.
A companion line of work scales the same idea. Bigger encoders, deeper projection heads during pretraining, and a multi-step semi-supervised recipe — pretrain on unlabelled data, fine-tune on a thin labelled slice, then distill into a student — show that model capacity which looked wasteful under pure supervision becomes an asset when most of the signal is contrastive and label-free. The editorial point is consistent across both papers: the geometry of embedding space can be organised by agreement under augmentation alone, and that organisation transfers.
Why it mattered then
In 2020, large labelled image collections were still expensive to extend, while unlabelled photographs were abundant. Contrastive self-supervision offered a practical route to representations that rivalled supervised pretraining without requiring class annotations during the long first stage. SimCLR showed that a carefully chosen augmentation policy, a nonlinear projection head, normalised embeddings, and large batches were enough to close much of the gap that earlier pretext tasks had left open. SimCLRv2 then demonstrated that the same family of methods rewards scale: larger models pretrained this way become stronger semi-supervised learners when only a fraction of labels is later introduced. Together they shifted expectation — unlabelled pretraining was no longer a distant second best for visual features, but a first-class training path.
Why it matters now
The same pattern — learn geometry from paired views, discard the head that served the loss, reuse the encoder — remains a template for representation learning beyond the original residual-network experiments. Whenever labels are sparse relative to raw data, pulling augmented views together and pushing unrelated samples apart is still a default way to spend compute. The practical lessons travel: augmentation strength is part of the objective, not mere regularisation; the space in which you optimise need not be the space you keep; and capacity that looks excessive under full supervision can pay for itself when most of the training signal is self-supervised.
The surprising detail
The representation used at test time is not the one the loss directly shapes. A multilayer projection head sits between the encoder and the contrastive objective; after pretraining that head is discarded. Empirically, forcing the loss to operate in this extra nonlinear space improves the encoder features that remain. The network is therefore trained to succeed in a space it will never use again — an architectural slight of hand that proved more effective than applying the loss to the encoder outputs directly.
What is disputed
Reported gains depend on augmentation composition, batch size, temperature, and encoder capacity. Figures from the original papers are tied to their specific residual-network setups and datasets; transferring the recipe to other architectures or domains requires re-tuning rather than assuming the same margins.
Remember this
Two views of one image are pulled together; views of different images are pushed apart. The projection head absorbs the loss; the encoder keeps the features.
Test yourself
After contrastive pretraining you freeze the encoder and train a linear classifier on its outputs. A colleague suggests you would get the same features if the NT-Xent loss had been applied directly to those encoder outputs, skipping the projection head. What does the framework’s evidence imply, and why might the head still matter even though it is thrown away?
The evidence runs the other way: a nonlinear projection head during pretraining improves the quality of the retained encoder representation. One reading is that the head is allowed to discard information that is useful for the contrastive task but harmful or irrelevant for downstream work — for example, low-level cues that make two views easy to match but do not generalise. By absorbing that pressure, the head leaves the encoder freer to keep broader features. Skipping it forces the encoder itself to specialise for the loss, which tends to produce a weaker frozen representation.
Go deeper
- [2002.05709] A Simple Framework for Contrastive Learning of Visual Representations · arxiv.org
- [2006.10029] Big Self-Supervised Models are Strong Semi-Supervised Learners · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.