II · THE IDEA · ARTIFICIAL INTELLIGENCE

Self-Supervised Learning

Foundations · 2018, BERT paper · Prediction as supervision

▶ Listen · narrated

Training used to mean hiring people to label examples. Self-supervised learning made the data label itself, and suddenly the entire web became a training set.

At a glance

What it is: A training regime where part of the input is hidden and the model learns by predicting it
What it replaced: Supervised learning, which required human-labelled examples for every task
BERT's approach: Mask 15% of tokens at random, train the model to predict them from context
Why it scales: Any unlabelled text becomes training data — no annotation bottleneck

Imagine teaching someone a language by showing them sentences with random words blacked out, then asking them to guess what is missing. They would need to learn grammar, vocabulary and context to fill in the blanks correctly. That is self-supervised learning. The model is shown text with some tokens hidden — replaced with a special [MASK] marker — and trained to predict what belonged there. It learns from its mistakes: every wrong guess produces a signal that adjusts the internal numbers slightly. No human needs to label anything, because the original sentence is both the puzzle and the answer. The model that results has learned enough about language structure to transfer that understanding to other tasks — answering questions, classifying sentiment, extracting names — even though it was only ever trained to fill in blanks.

Self-supervised learning for language models uses a masked prediction objective. During training, a fixed percentage of input tokens (15% in BERT's case) are selected for masking. Of these, 80% are replaced with a [MASK] token, 10% with a random token from the vocabulary, and 10% left unchanged. The model processes the corrupted sequence through its transformer layers and outputs a probability distribution over the vocabulary for each masked position. The loss is computed only at those positions, comparing the predicted distribution to the true token via cross-entropy. Gradients flow back through the entire model, updating all parameters.

The method is self-supervised because no external labels are required — the supervision signal comes from the data itself. The model is not told what any token means or what task it should perform. It simply learns to reconstruct missing pieces, which forces it to build representations that encode syntax, semantics, coreference and context. These representations transfer to downstream tasks via fine-tuning: the pretrained model is initialised with the learned weights, a task-specific head is added, and the entire system is trained on a smaller labelled dataset for that task.

The masking percentage is a hyperparameter balancing two pressures. Too low and training is sample-inefficient, because most positions provide no learning signal. Too high and the context becomes too degraded to support accurate prediction. The 80/10/10 split for mask/random/unchanged addresses a train-test mismatch: [MASK] never appears at inference time, so the model must learn to produce useful representations even at unmasked positions. The random token replacement prevents the model from simply copying its input at unchanged positions.

RoBERTa later showed that some of BERT's other choices — such as the next-sentence prediction auxiliary task — were not necessary, and that training longer on more data with larger batches improved results substantially. The core masked language modelling objective, however, remained.

Look closer

The mask is not always a mask
BERT's masking strategy has a deliberate asymmetry. Of the tokens chosen for masking, 80% are replaced with a special [MASK] token, 10% are replaced with a random token from the vocabulary, and 10% are left unchanged. The model must predict the original token in all three cases. The variation exists because [MASK] never appears during actual use — if the model learned to ignore everything except positions marked [MASK], it would fail when deployed. The random replacements and unchanged tokens force it to maintain useful representations everywhere.
Masking percentage is a trade-off under tension
Mask too little and training is slow — most of the computation goes toward positions the model is not learning from. Mask too much and context collapses — the model cannot infer what belongs in the gaps because too many clues are missing. BERT settled on 15%, which became a widely copied default, though RoBERTa's experiments suggested the choice was less critical than initially thought. The percentage is not a discovered constant; it is an engineering compromise between signal and efficiency.
The objective is local, but the learning is not
The model is only asked to predict specific masked positions, but to do that well it must build representations of everything. A masked verb requires understanding the subject and object around it. A masked noun requires tracking references across sentences. The supervision signal is narrow, but the pressure it creates is broad. This is why self-supervised models transfer so effectively — they were never trained to do one thing, they were trained to maintain enough understanding that any missing piece could be recovered.

The story

Supervised learning worked well when you had labelled data. Someone read an email and marked it spam or not spam. Someone looked at an image and wrote down what it contained. The model learned from those labels, and if you wanted it to do something new, you needed new labels. That meant hiring people, writing guidelines, checking consistency, and waiting. It did not scale to the amount of text that existed.

Self-supervised learning removed the bottleneck by making the data label itself. Take a sentence, hide a word, and ask the model to predict it. The original sentence is both the input and the answer. No one needs to annotate anything — the supervision comes from the structure of the task.

BERT, introduced in 2018, made this concrete. During training, it would take a sequence of tokens and randomly mask 15% of them, replacing most with a special [MASK] marker. The model's job was to predict what belonged in those positions. It had the surrounding context — the words before and after — and from that it had to infer the missing pieces. Every mistake produced a gradient. Every correction made the internal representations a little more useful.

The task sounds narrow, but the learning is not. To predict a masked verb, the model needs to understand the subject and the object. To predict a masked name, it needs to track references. To predict a masked preposition, it needs syntax. The supervision signal touches only the masked positions, but meeting it requires building representations of everything else. You cannot guess what is missing unless you understand what remains.

This is why models trained with self-supervision transfer so well. They were never taught to do sentiment analysis or question answering. They were taught to reconstruct text, which required them to learn about grammar, reference, implication and context. Those capabilities then apply to tasks the training never mentioned.

The method is not unique to language. The same principle works wherever you can hide part of an input and ask a model to recover it. In images, you can mask patches. In video, you can mask frames. In molecules, you can mask atoms. Anywhere structure exists, you can turn prediction into supervision, and supervision into learning, without a single human label.

Why it mattered then

BERT arrived at a moment when labelled datasets were both expensive and limiting. ImageNet, one of the most influential supervised datasets, required years of effort and significant funding to annotate. For language, the problem was worse — tasks were so varied that each one needed its own labels, and transfer between tasks was weak. Models trained on sentiment data did not help much with question answering. Models trained on translation did not help much with named entity recognition. Self-supervision changed the economics. Unlabelled text was effectively unlimited — web crawls, books, articles, all of it became usable. BERT trained on the concatenation of English Wikipedia and a corpus of unpublished books, neither of which carried task-specific labels. The resulting model could then be fine-tuned on small labelled datasets for specific tasks, and it consistently beat models trained only on those task labels from scratch. The BERT paper reported improvements on eleven different language understanding tasks, several of them substantial. The shift mattered beyond performance numbers. It meant a research group without access to large annotation budgets could still train capable models. It meant new tasks could be explored without first spending months labelling data. It redirected effort from annotation pipelines to modelling and data quality, and it made scale — both of models and of training data — a viable strategy again.

Why it matters now

Self-supervised learning is now the default first step for large models. GPT, which uses a different self-supervised objective — predicting the next token rather than filling in masks — demonstrated that the principle scaled further than BERT's original formulation. Models trained to predict text go on to write it, answer questions about it, summarise it, and translate it, all from the same initial learning process. The technique has also migrated. Vision models now use masked image modelling. Protein models use masked amino acid prediction. Speech models mask audio segments. The core idea — hide part of the data, learn by reconstructing it — turns out to be more general than language. It has not solved every problem. Self-supervised models still require fine-tuning or prompting to perform specific tasks reliably, and they inherit the biases present in their training data, which is now drawn from enormous, difficult-to-audit web crawls. The removal of the labelling bottleneck created a data quality bottleneck instead — when you train on everything, you train on everything, including the parts you would have filtered out if someone had been reading it. But the method remains foundational. Any current system that starts with a pretrained model, whether for language or another domain, is starting with something that learned through self-supervision. The label-free regime is no longer an alternative approach. It is the infrastructure.

The surprising detail

BERT's masking is stochastic during training — the same sentence can appear many times across epochs, with different tokens masked each time. This means the model never sees the true unmasked distribution it will encounter during deployment. It only ever sees text with holes in it. The final representations therefore emerge from a training regime that is statistically different from the usage regime, yet they transfer reliably. The mismatch is not a bug that needs fixing; it is the method. The model learns to be useful precisely because it spent all its training time compensating for missing information.

Remember this

Self-supervision turns structure into signal. Hide part of the data, predict it, learn from the errors — no labels required.

Test yourself

Why does BERT replace only 80% of the masked tokens with [MASK], leaving 10% as random tokens and 10% unchanged, rather than masking all of them consistently?

Go deeper

Image: Original diagram, The Daily Triptych. Licence: Original work. Source.

← Back to day 11