II · THE IDEA · ARTIFICIAL INTELLIGENCE
BERT and the Masked Language Modeling Revolution
▶ Listen · narrated
A model that only sees the words left of a blank is solving a different problem from one that can also see the words to the right. BERT made the second approach standard.
At a glance
- Full name
- Bidirectional Encoder Representations from Transformers
- Core objective
- Masked language modelling (a cloze-style fill-in task)
- Secondary task
- Next sentence prediction (later questioned by RoBERTa)
- Architecture
- Encoder-only Transformer, deep and bidirectional
- Released
- 2018 preprint; RoBERTa re-examined the recipe in 2019
Think of a cloze test on a school worksheet: a sentence with a blank in the middle, and you fill the blank using the words on both sides. BERT is pretrained rather like that. Some words in each training sentence are hidden, and the model must guess them from everything that remains, left and right alike.
Older language models more often worked like a person writing a line from left to right, always predicting the next word from what had already been written. That is excellent practice for generation. It is less ideal when you later want a single vector for each word that already knows the whole sentence — for example, to decide whether bank means a river edge or a financial firm.
Because BERT only scores the hidden positions, its Transformer layers are allowed to look at the entire sentence at once without cheating. After this pretraining, the same encoder can be fine-tuned on smaller labelled tasks. A later paper, RoBERTa, kept the fill-in-the-blank idea, dropped a secondary sentence-pair task BERT had used, and showed that heavier, more careful optimisation improved results further.
BERT is an encoder-only Transformer pretrained primarily with masked language modelling (MLM). A subset of WordPiece tokens — fifteen per cent in the original recipe — is selected for prediction. Selected tokens are replaced by [MASK] about eighty per cent of the time, by a random vocabulary token about ten per cent of the time, and left unchanged about ten per cent of the time; the model must recover the original token identity at those positions. Loss is computed only on the selected positions, which permits full bidirectional self-attention in every layer without the information leak that would make bidirectional next-token prediction degenerate.
The original work also used next sentence prediction (NSP): packed sentence pairs with segment embeddings and a classifier on the [CLS] representation that distinguishes true continuation pairs from random pairs. Base (12-layer) and large (24-layer) configurations were released. RoBERTa later removed NSP, adopted dynamic masking, and re-tuned data scale, batch size, and training duration, obtaining stronger downstream results and arguing that BERT had been under-optimised rather than wrongly conceived on the MLM core.
Limitations follow from the design. MLM pretraining does not teach open-ended generation; the encoder has no causal decoder. The pretrain–finetune mismatch around [MASK] is only partly mitigated by the eighty/ten/ten replacement policy. Bidirectional attention also assumes access to the full input span, which is appropriate for understanding benchmarks and less so for token-by-token left-to-right decoding.
Look closer
Masking is not always [MASK]
In the BERT pretraining recipe, roughly fifteen per cent of WordPiece tokens are chosen for prediction. Of those, about eighty per cent are replaced with the special [MASK] symbol, ten per cent with a random vocabulary token, and ten per cent left unchanged. The model must still recover the original identity. The mixed replacements reduce the mismatch between a pretraining input full of [MASK] symbols and a fine-tuning input that contains none.
Bidirectional means joint, not just both ways
Earlier contextual representations such as ELMo combined a left-to-right language model with a right-to-left one. Each direction was trained separately. BERT’s Transformer encoder attends to the full sequence in every layer, so the representation of a token is conditioned on left and right context simultaneously. That is only possible because the training target is a masked subset, not the next token in causal order.
What RoBERTa kept and what it dropped
The RoBERTa study treated BERT’s design as an under-optimised baseline. It removed the next-sentence prediction objective, switched from static to dynamic masking across epochs, trained longer with larger batches on more data, and reported substantial gains on the same downstream suites. The masked language modelling core remained; several surrounding choices did not.
The story
For much of the 2010s, strong neural language models were trained to predict the next token given only the tokens that came before it. That objective is natural for generation, and it scales cleanly. It is less natural when the goal is to build a representation of a whole sentence for classification, span labelling, or pairwise inference. In those settings the words after a token are often as informative as the words before it, yet a left-to-right model is forbidden to use them during pretraining.
BERT, introduced in a 2018 preprint, attacked that mismatch with a different pretraining task. Instead of predicting the future, the model is shown a sequence in which some tokens have been blanked out, and it must recover the originals from the unmasked surroundings. The paper frames this explicitly as a cloze-style procedure: the blank is inside the sentence, and both flanks are visible. Because the loss is computed only at the masked positions, the encoder can be fully bidirectional without the trivial self-prediction that would ruin a bidirectional next-token model.
The architecture underneath is an encoder-only Transformer. Two main configurations were released with the original work: a base model with twelve layers and a large model with twenty-four. Inputs are WordPiece token sequences, packed when useful as sentence pairs separated by special markers, with segment embeddings so the model can tell which token belongs to which sentence. Pretraining used large plain-text corpora — BooksCorpus and English Wikipedia are named in the paper — and produced parameters that could then be fine-tuned with relatively small labelled sets for many downstream tasks.
A second objective sat alongside the mask prediction: next sentence prediction. Given two spans packed into one input, the model had to say whether the second truly followed the first in the original text or was a random substitute. The authors presented this as a way to learn relationships between sentences, useful for question answering and natural language inference. A year later, the RoBERTa paper re-examined that choice carefully. Under matched or stronger optimisation, removing next sentence prediction did not hurt — and often helped — while longer training, larger batches, more data, and dynamic masking improved results further. The headline lesson of BERT survived; the full original recipe did not have to.
What changed in practice was the default starting point for English NLP systems. Rather than training a task model from random weights, practitioners began from a bidirectional encoder that already knew a great deal about how words co-occur in context. Fine-tuning became the ordinary path to competitive numbers on the GLUE-style benchmarks of the period. Generative, left-to-right pretraining continued in parallel for other goals; BERT did not replace it. It carved out a clear regime in which understanding a whole sequence mattered more than writing the next word, and it gave that regime a simple, widely copied training signal.
Why it mattered then
In 2018 the field already had contextual embeddings and Transformer architectures, but the dominant pretrained language models were still essentially unidirectional. Transfer often meant taking hidden states from models trained to generate or score text left to right, or stitching separate forward and backward models together after the fact. BERT offered a single encoder whose every layer was jointly bidirectional, trained with an objective that matched the fine-tuning setting more closely than next-token prediction did. The empirical gains on sentence-level and token-level benchmarks were large enough, and the fine-tuning recipe simple enough, that the approach spread quickly through both research and industry pipelines. It also reframed pretraining as something you could design around understanding tasks rather than only around generation.
Why it matters now
Masked language modelling and encoder-only stacks remain standard tools wherever the job is to classify, retrieve, or label text rather than to open-end generate it. Even where decoder-only models now dominate public conversation, many production rankers, embedders, and token-level taggers still rest on bidirectional pretraining ideas that BERT made ordinary. The RoBERTa follow-up still matters as a caution: headline architecture is not the whole story; batch size, training duration, data volume, and which auxiliary losses you keep can move results as much as the original design. When a new pretraining paper claims a revolution, the BERT-to-RoBERTa arc is a useful reminder to separate the enduring objective from the contingent engineering choices around it.
The surprising detail
The [MASK] symbol the model sees so often in pretraining never appears at fine-tuning time. BERT’s authors did not ignore that gap: of the tokens selected for prediction, only about eighty per cent become [MASK]; ten per cent are swapped for a random token and ten per cent are left as they are, so the model cannot assume that only a blank needs fixing. RoBERTa later kept masking but made it dynamic — newly sampled each time a sequence is shown — rather than fixed once during data preparation. The famous objective is therefore not a single rigid trick but a small family of compromises between train-time blanks and test-time reality.
What is disputed
The original BERT paper reported strong gains from next sentence prediction; RoBERTa, under different optimisation and data regimes, found the objective unnecessary or harmful. The discrepancy is real in the public record and is best read as evidence that auxiliary losses interact with training setup, not as a simple error on either side.
Remember this
BERT made joint left-and-right context ordinary by training on cloze-style masks rather than next-token prediction; RoBERTa later showed that the mask objective endured while next-sentence prediction did not have to.
Test yourself
A left-to-right language model and a BERT-style encoder are both asked to build a representation of the word bank in the sentence The river bank was steep. Why is the BERT-style pretraining setup better aligned with using the word steep as evidence, and what would go wrong if you simply trained a fully bidirectional model with ordinary next-token loss?
In BERT-style masked language modelling the loss sits on blanked tokens and the encoder may attend both left and right, so steep can inform the representation of bank during pretraining. Ordinary next-token loss on a fully bidirectional encoder would let each token see itself (or its identity leaked through context) and make the prediction task trivial, so the model would not learn useful distributions over genuine context.
Go deeper
- [1810.04805] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · arxiv.org
- [1907.11692] RoBERTa: A Robustly Optimized BERT Pretraining Approach · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.