II · THE IDEA · ARTIFICIAL INTELLIGENCE
The Rise of Self-Supervised Learning
▶ Listen · narrated
A network shown a photograph with no name attached can still learn what is in it, provided the training game it plays cannot be won without seeing real structure.
At a glance
- RotNet task
- Predict which of four rotations was applied to an image
- Rotations used
- 0°, 90°, 180°, and 270°
- SimCLR idea
- Contrastive learning on two augmented views of each image
- Key SimCLR parts
- Strong augmentations, projection head, NT-Xent loss
- Shared aim
- Learn transferable visual features without class labels
Suppose you want to teach someone about photographs without ever telling them what is in one. You could tip each photo on its side and ask which way is up. To answer, they must notice faces, horizons and how furniture usually sits. Or you could hand them two heavily edited copies of one photo — cropped differently, colours shifted — and ask them to pick out that this pair came from the same original, unlike the others in the pile. Either way they never hear an object's name, yet they learn what stays steady when appearance changes.
Self-supervised visual learning runs on that idea. Rotation prediction turns orientation into a four-way quiz: 0°, 90°, 180° or 270°, and the computer already knows the answer because it did the turning. Contrastive learning, as in SimCLR, pulls two edited copies of one image together and pushes other images away. Afterwards it deletes the small projection head used for that pulling and pushing, and keeps the encoder — the part that turns a picture into numbers — underneath. The chores differ; the bargain is the same. Set a task whose answer is free, then keep the machinery that had to understand the picture to solve it.
RotNet frames unsupervised representation learning as four-way classification over geometric transforms. Given an image x, a transform t ∈ {0°, 90°, 180°, 270°} is applied and a convnet fθ is trained to predict t. The target is free because the pipeline chose t. Gradients update the shared backbone; after pretraining the rotation head is dropped and the backbone transferred. The signal is exact and label-free, yet solving it requires orientation-sensitive semantic structure, which is why the backbone becomes useful.
SimCLR instead performs instance discrimination on paired views. Two augmentations t and t' applied to x yield views x̃i and x̃j. An encoder f gives hi = f(x̃i), the representation kept for transfer. A nonlinear projection head g maps it to zi = g(hi). The NT-Xent loss over normalised zi, zj raises the similarity of the positive pair relative to the other elements of the batch, scaled by a temperature τ; those other elements are the negatives. Once training finishes, g is discarded and hi is used for linear evaluation or fine-tuning. The framework reports three factors as empirically important: composing strong augmentations, cropping and colour distortion among them; the nonlinear head; and batches large enough to supply negatives.
The limits follow from the setups. A rotation pretext carries little signal for images with no canonical upright pose. Contrastive quality depends on augmentation design and batch composition, and on the assumption that two views of one image should attract — an assumption that blunts useful distinctions when the augmentations destroy a cue the downstream task needs. Neither paper claims a universal pretext. Both show that a carefully chosen label-free objective can yield transferable visual features.
Look closer
Four angles, not arbitrary spin
The rotation method does not ask a network to regress a continuous angle. It treats orientation as a four-way classification problem: decide whether the input was rotated by 0°, 90°, 180°, or 270°. That discrete choice is easy to implement, yet hard to solve without recognising objects and their upright structure, so the intermediate features become useful far beyond the pretext itself.
Two views of the same image
In the contrastive framework, each training image is turned into a pair of differently augmented copies. The model is trained to recognise that the two copies belong together and to separate them from other images in the batch. The signal therefore comes from agreement across appearance changes, not from a human-provided category name.
The projection head that is thrown away
A nonlinear projection head sits on top of the representation during contrastive training. After training, that head is discarded and the layer beneath it is kept for downstream use. The paper shows that this seemingly small design choice markedly improves the quality of the retained features compared with contrasting the representation layer directly.
The story
For a long stretch, the way to teach a network to see was to pay people to label pictures. Someone writes "dog" under a photograph of a dog, and the network is trained to produce that word. Self-supervised learning took a different route. Invent a question whose correct answer can be worked out from the image itself, with no human asked. Train the network on that question. Then throw the question away and keep what the network learned along the way.
The rotation method shows the trick at its plainest. Take an unlabelled image and turn it by one of four amounts: 0°, 90°, 180° or 270°. The network is told nothing except the turned picture, and must answer which of the four turns was applied. The answer is free to check, because the program applied the turn and therefore already knows it. But answering is not easy. To tell an upright street from a street lying on its side, the network has to find faces, wheels, horizons and roofs, and to know which way up those things normally sit. So the pressure of the task pushes real visual knowledge into the network's inner layers. The final layer that names the angle is disposable. The layers beneath it — the ones that detect parts and shapes — are the prize, and they transfer to tasks where labels do exist.
Contrastive learning arrives at the same prize by a different road. SimCLR, short for a simple framework for contrastive learning of visual representations, works like this. Take one image and make two copies, each mangled in a different way: cropped differently, colours shifted, and so on. These altered copies are called views. Feed both views through the encoder — the stack of layers that turns a picture into a list of numbers, called its representation. Now the training rule. The two views came from the same photograph, so their two number-lists should be close together. Views of any other photograph in the same batch should be pushed far away. That rule is the NT-Xent loss, and the other images in the batch are what supply the examples to push against. Nobody ever says what the photograph contains. The network only learns which things belong together and which do not.
One extra piece sits between the encoder and the loss: a small network called a projection head, which maps the representation into a second space where the pulling and pushing actually happen. When training ends, that head is deleted. The kept representation is the one underneath it, which can then be tested with a simple linear classifier or fine-tuned on a labelled task.
Several choices in this setup matter far more than their plainness suggests. The mangling is not a minor tidying step; combining strong random crops with colour distortion is what forces the encoder to hold on to what survives such abuse. Batch size matters because a bigger batch offers more other images to push away from. And the projection head matters because the layer best shaped for the pulling-and-pushing rule is not the layer best shaped for later reuse.
Rotation prediction and contrastive learning are not the same algorithm. One is a four-way guess about geometry; the other is a matching game between two views. What they share is the move that defined the period: replace human labels with a chore the machine can mark itself, chosen so that no shortcut solves it. Masked modelling — hide part of the input and demand its recovery — belongs to the same family in spirit, but the two papers anchoring this lesson are the rotation work and the simple contrastive framework. Read side by side, they show that the design of the chore, not the supply of labelled categories, was the practical path to features worth keeping.
Why it mattered then
When these methods appeared, gathering and cleaning a large labelled image set was slow, costly work, and for many fields it was the thing standing in the way. Rotation prediction gave a concrete demonstration that a task solvable from the pixels alone could produce features good enough to reuse elsewhere. A few years later, the contrastive framework tightened the recipe into parts anyone could assemble: strong mangling of each image, a projection head that is later deleted, the NT-Xent rule, and batches large enough to supply plenty of images to push away from. No exotic architecture was required. That is what turned self-supervised pretraining from a scatter of one-off tricks into an engineering path others could repeat.
Why it matters now
The same bargain still holds. Whether the chore is turning an image, matching two views of it, or hiding part of it and asking for the rest, the aim is to pull usable structure out of data whose labels are missing, expensive or skewed. The rotation paper and the contrastive framework stay useful as landmarks because they make the mechanism easy to see: set a task the machine can mark without a human, then keep the encoder and discard the head that answered the task. Anyone training or adapting open models repeats that logic whenever unlabelled images outnumber carefully labelled ones.
The surprising detail
In the contrastive framework, the layer the training rule improves is deliberately not the layer kept afterwards. The projection head soaks up part of the objective, and deleting it once training ends leaves a better representation underneath. Part of the machinery that solves the chore is disposable by design, and the experiments treat that as central rather than as tidying.
What is disputed
This lesson draws only on the rotation-prediction paper and the SimCLR contrastive framework. Broader claims about masked modelling, later joint-embedding methods, or industry-wide adoption timelines are outside those sources and are not asserted here.
Remember this
A pretext task works when its answer costs nothing to check, yet cannot be reached without learning something durable about the picture.
Test yourself
A team trains one encoder to classify which of four rotations was applied to each image, and another with a contrastive loss on two augmented views, discarding a projection head afterwards. In both cases no class labels are used. What is each method actually forcing the encoder to keep, and why might the retained features still help a later labelled recognition task?
Rotation prediction forces the encoder to retain cues about object identity, parts and canonical orientation, because those are needed to tell 0°, 90°, 180° and 270° apart. Contrastive learning forces it to retain cues that survive strong augmentation while ignoring brittle appearance differences between two views of the same image. A later labelled recogniser can reuse those features because category distinctions often align with the same stable structure the pretexts already demanded, even though no category names were present during pretraining.
Go deeper
- [1803.07728] Unsupervised Representation Learning by Predicting Image Rotations · arxiv.org
- [2002.05709] A Simple Framework for Contrastive Learning of Visual Representations · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.