II · THE IDEA · ARTIFICIAL INTELLIGENCE
The Causal Mask
▶ Listen · narrated
A model trained to generate the next word has already seen the entire sentence, thousands of times. The causal mask is what stops that from becoming a simple memory test.
At a glance
- What it does
- Prevents each token from attending to tokens that come after it in the sequence
- How it works
- Sets attention scores to negative infinity before the softmax, zeroing them out
- When it applies
- During training and generation in autoregressive models; not used in encoders
- Shape
- Upper triangle of a square matrix, one row and column per token position
Imagine you are teaching someone to write a story by showing them one sentence at a time, always in order, and asking them to guess the next word. They get better because they learn patterns — what kind of word usually follows "the", what kind of sentence usually follows a question. Now imagine instead you showed them the entire story all at once, including the ending, and asked them to fill in a blank in the middle. They would just copy from the page. They would not learn to predict.
The causal mask is the rule that stops a model from cheating in exactly that way. During training, the model sees whole sentences, but the mask hides the future from each word. Word five can see words one through five, but not six or seven. That forces the model to learn what comes next based only on what came before, even though the trainer already knows the whole sentence. It is a blindfold that stays on during training and comes off during generation — except by then the model has learned to predict without it, so it does not need to cheat anymore.
The causal mask is a binary matrix applied to the attention logits before the softmax normalisation in each transformer layer. For a sequence of length *n*, the mask is an *n* × *n* matrix where element (*i*, *j*) is zero if *j* ≤ *i* and negative infinity if *j* > *i*. This is added to the matrix of query-key dot products, so any attention score pointing from position *i* to a future position *j* becomes −∞. The subsequent softmax maps −∞ to exactly zero, and that position contributes nothing to the weighted sum of values.
The mask enforces an autoregressive factorisation of the joint probability: *p*(x₁, …, xₙ) = ∏ *p*(xᵢ | x₁, …, xᵢ₋₁). Without it, the model could learn to attend to xᵢ when predicting xᵢ, collapsing the task into copying rather than conditional generation. The mask is not learned and does not vary by layer, head, or example; it is a fixed function of sequence length, computed once and broadcast across all attention operations.
At inference time, when generating token by token, the mask is technically redundant because future tokens do not exist in the key-value cache. However, it is still applied for consistency, and in some implementations it is used to mask out padding positions in batched generation where sequences have different lengths. The mask's computational cost is negligible — it is a single addition of a precomputed tensor — but its effect on what the model can learn is total. Every parameter in a causal language model has been optimised under the constraint that information flows only backward in time, and removing the mask after training would not grant the model bidirectional reasoning; it would simply produce incoherent attention patterns because the model was never trained to use future context.
Look closer
The mask is applied before softmax, not after
The attention mechanism computes a score for every pair of tokens, then normalises those scores with a softmax so they sum to one. The causal mask sets the upper-triangle scores — the ones pointing forward in time — to negative infinity before that softmax runs. Negative infinity through a softmax becomes zero, cleanly and exactly. If you masked after the softmax instead, you would have to renormalise, and the operation would be less numerically stable. The choice of negative infinity is not arbitrary; it is the value that makes the softmax do the work.
It is the same mask for every layer and every head
The causal constraint is not learned and does not vary across the model. Every attention head in every layer uses an identical triangular mask, determined only by the sequence length. This is cheaper than it sounds: the mask is computed once and reused, and because it contains only two distinct values — zero and negative infinity, or in some implementations, zero and one applied as a multiplicative mask — it compresses well and costs almost nothing to store or transmit between operations.
The diagonal is visible to itself
Token three can attend to tokens one, two and three, but not to four or five. That means each position can see itself. In practice this matters less than it might seem, because the attention score between a token and itself is one vote among many, and the model learns whether to use it. Some positions attend strongly to themselves — often punctuation or function words — while others largely ignore their own embedding and gather context from elsewhere. The mask permits self-attention but does not require it.
The story
A transformer trained to generate text sees entire sentences at once. You give it a thousand tokens, and in a single forward pass it computes attention scores between all of them — a million pairs. That parallelism is what makes transformers fast to train, but it creates a problem: if the model can see token 500 while it is learning to predict token 500, it is not learning to predict at all. It is learning to copy.
The causal mask solves this by making part of the attention matrix invisible. Before the model computes how much token *i* should attend to token *j*, the mask checks whether *j* comes after *i*. If it does, the attention score is set to negative infinity. When that score passes through the softmax — the function that turns raw scores into a probability distribution — negative infinity becomes exactly zero. Token *i* cannot attend to token *j*. The future is hidden.
The result is a lower triangular matrix of attention weights. Token one attends only to itself. Token two attends to one and two. Token three attends to one, two and three. Each position sees only the context that would have been available if the sequence were being generated left to right, one token at a time, even though during training the entire sequence is present in memory.
This is not masking in the sense of blacking out pixels in an image. The tokens are still there, still embedded, still flowing through the layers. The mask operates on the attention scores, not the tokens themselves. It is a surgical intervention in one specific operation, repeated at every layer: the query-key dot product that decides how much each token should listen to each other token. Everywhere else in the model, the full sequence is visible.
The same mask is used at generation time, but there it is almost redundant. When you are generating token by token, the future tokens do not exist yet, so there is nothing to mask. The causal mask still runs, masking positions that have not been filled, but its real work was during training — ensuring that the model learned to predict without seeing ahead.
Without the mask, a model trained on "The cat sat on the" to predict "mat" would learn that whenever "the" appears near the end of a sequence and "mat" appears one token later, it should output "mat". That is not language modelling. With the mask, the model at position five sees only "The cat sat on the". It has to learn that "the" at the end of that fragment is often followed by a noun, and that "sat on the" suggests a surface. The mask forces the model to extract and generalise, because it removes the shortcut.
Why it mattered then
The causal mask appeared in the original Transformer paper in 2017, described in a single sentence as preventing positions from attending to subsequent positions. Vaswani and his co-authors were building a model for machine translation, which uses an encoder-decoder architecture: the encoder sees the entire source sentence at once and does not need a causal mask, but the decoder generates the target sentence one token at a time and must not see the future. The mask was necessary to make that autoregressive generation work during training, when the target sentence is already known. The insight was not that masking was useful — that was obvious — but that it could be implemented as a simple additive bias in the attention logits, applied before the softmax. Earlier sequence models, particularly recurrent networks, enforced causality by construction: they processed one token at a time, maintaining a hidden state, so the future was never available. The transformer's parallelism required an explicit mechanism, and the mask provided one that was both mathematically clean and computationally cheap. It cost almost nothing, and it allowed the entire decoder to train in parallel, seeing all target positions simultaneously while still learning to generate sequentially.
Why it matters now
The causal mask is now the defining feature of autoregressive language models. GPT, LLaMA, and every other model trained to predict the next token uses it at every layer during training. It is part of the architecture, specified once and frozen, and it is the reason these models can be trained on entire documents at once rather than one token at a time. It also explains some of their limitations. A causal model cannot revise its earlier output based on what comes later, because later context is masked during training. If a sentence begins ambiguously and resolves only at the end, the model must commit to an interpretation before it sees the resolution. This is why autoregressive models sometimes start a sentence in one direction and then awkwardly correct themselves, or why they struggle with tasks that require global coherence across a long document. The mask enforces a left-to-right flow of information, and that flow is baked into every learned parameter. Bidirectional models, like BERT, do not use a causal mask. They mask random tokens instead and train the model to predict them from both directions. That makes them better at understanding, but they cannot generate text autoregressively — they have no notion of sequence order baked into their training objective. The causal mask is not just a detail of implementation; it is a choice about what kind of task the model will be able to perform, and that choice is made before training begins.
The surprising detail
The causal mask is often described as if it makes the model forget the future, but the model never sees the future in the first place during generation — it does not exist yet. The mask's real function is to cripple the model during training, removing information it could otherwise use, so that the task stays hard. You are deliberately making the model worse at the training objective — predicting token 500 would be trivial if token 500 were visible — in order to force it to learn something more general. The mask is a form of regularisation, though it is rarely called that. It prevents a specific kind of overfitting: memorising the training sequences rather than learning the conditional distributions that generated them.
Remember this
The causal mask is not learned. It is a fixed triangular matrix of negative infinity, applied before every softmax, ensuring the model never attends forward in time.
Test yourself
A researcher proposes training a model with a causal mask that is 90 per cent sparse — each token can see 90 per cent of the previous context, chosen randomly, rather than all of it. Would this break autoregressive generation, and if so, how?
It would not break generation mechanically — the model would still produce a sequence one token at a time — but it would severely damage the model's ability to learn long-range dependencies. If token 50 can only see a random 10 per cent of the previous 49 tokens during training, it cannot reliably learn patterns that depend on specific earlier tokens being present, because those tokens might be masked out in any given training example. The model would learn to generate text that depends only on very local context, because that is the only context it can count on having seen consistently during training. The causal mask's full visibility of all previous tokens is not a luxury; it is what allows the model to learn that a pronoun at position 50 might refer back to a noun at position 3. Sparsifying the mask trades off long-range coherence for a small reduction in computation, and the trade is almost never worth it for language.
Go deeper
- Attention Is All You Need · arXiv · Ashish Vaswani et al. · 2017-06-12
- Language Models are Few-Shot Learners · arXiv · Tom B. Brown et al. · 2020-05-28
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.