II · THE IDEA · ARTIFICIAL INTELLIGENCE
Logits and the Softmax
▶ Listen · narrated
Before a model can choose a token, it must first decide how likely each candidate ought to be. The mechanism that performs this conversion shapes every word the model generates.
At a glance
- Input
- One real-valued score (logit) per vocabulary entry
- Output
- Probability distribution: non-negative values summing to 1
- Effect
- Exponential amplification of differences between scores
- Typical vocabulary size
- Tens of thousands of entries
Imagine a teacher scoring four essays and writing 8, 7.5, 7, and 2 out of 10 on them. If you were asked which is best, you would say the first — but if you were asked to bet money on each one winning a prize, you would not put all your money on the first, because the second and third are very close. Softmax does something similar, but with a twist: it does not treat the gaps between scores as equal. A score that is one point higher does not become a little more likely — it becomes about 2.7 times more likely, because the function exponentiates the scores before turning them into probabilities. A gap of two points means 7 times more likely, and a gap of five points means 148 times. This amplification is why the token with the highest score usually gets most of the probability, unless several tokens are scored very close together.
The softmax function is defined as softmax(z_i) = exp(z_i) / sum_j exp(z_j), where z is the vector of logits. It is differentiable everywhere, and its gradient has a convenient form during backpropagation: the derivative of softmax(z_i) with respect to z_j is softmax(z_i) times (delta_ij − softmax(z_j)), where delta is the Kronecker delta. This makes it well-suited to gradient-based training. The function is invariant to additive shifts: adding a constant to every logit leaves the output unchanged, which is why implementations often subtract the maximum logit before exponentiating, preventing overflow without altering the result. The exponential mapping from logits to unnormalised probabilities means that a unit increase in one logit multiplies that token's probability by e, roughly 2.718, relative to the others. This amplification is stronger than a linear mapping would produce, and it ensures that the highest-scoring token usually dominates unless competitors are within about one or two logits. In a vocabulary of fifty thousand tokens, most will have negligible probability after softmax if their logits are more than a few units below the maximum. The temperature parameter, often applied before softmax, divides every logit by a scalar: temperatures above one flatten the distribution, and temperatures below one sharpen it, but the underlying exponential structure remains.
Look closer
The scores are called logits, and they can be any real number
Before softmax, the model outputs one score per token in its vocabulary. These are called logits, and they have no upper or lower bound — a logit can be 3.7, −12.4, or 0.003. They are not probabilities yet. They are not even comparable in a strict sense, because their scale is arbitrary. What matters is their relative ordering and spacing, not their absolute magnitude.
Softmax exponentiates, then normalises
The softmax function raises e to the power of each logit, producing a list of positive numbers. Then it divides each by the sum of all of them, ensuring the result is a valid probability distribution: every entry is between zero and one, and they all add up to exactly one. The exponentiation step is what gives softmax its characteristic behaviour: a logit that is one unit larger than another produces a probability roughly 2.7 times greater, and the gap compounds quickly.
Small differences in logits become large differences in probability
If one token has a logit of 5 and another has a logit of 2, the first does not end up three times more probable — it ends up about twenty times more probable, because e to the power of 5 divided by e to the power of 2 is e cubed, roughly 20. This amplification is intentional. It allows the model to express strong preferences, and it ensures that the highest-scoring token usually dominates the distribution unless other tokens are very close in score.
The story
At the final layer of a transformer, the model has computed a hidden state — a vector of floating-point numbers summarising everything it has inferred so far about what should come next. That vector is then multiplied by a large matrix, one row per entry in the vocabulary, producing a single score for every token the model knows. These scores are the logits.
The logits are not probabilities. They are unbounded real numbers, and their scale is determined by the internal geometry of the model's weights, not by any external constraint. A logit of 10 is not twice as confident as a logit of 5 in any straightforward sense. What the model needs at this point is a way to turn this list of arbitrary scores into something it can sample from: a probability distribution where every value is non-negative and the whole collection sums to one.
Softmax performs this conversion in two steps. First, it exponentiates every logit — raises e, the base of the natural logarithm, to the power of each score. This guarantees that every result is positive, because e to any power is always greater than zero. Second, it divides each of these exponentiated values by their sum, which forces the entire list to add up to exactly one. The result is a valid probability distribution over the vocabulary.
The exponentiation is not merely a mathematical convenience. It amplifies differences. A token whose logit is one unit higher than another's will end up with a probability roughly e times larger — about 2.7 times. A gap of two units means e squared, roughly 7.4 times. A gap of five units produces a ratio of about 148 to one. This is why the highest-scoring token usually dominates the distribution unless several tokens are bunched closely together in logit space.
Once the softmax has run, the model holds a list of probabilities, one per vocabulary entry, and it can sample from that distribution to choose the next token. But the shape of that distribution — how concentrated or spread out it is, how much probability mass sits on the top few tokens versus the long tail — was determined by the gaps between the logits, amplified exponentially by the softmax function.
Why it mattered then
The softmax function appears in the original Attention Is All You Need paper from 2017, where it serves two distinct roles: converting attention scores into attention weights, and converting final-layer logits into token probabilities. Both uses rely on the same property — it transforms arbitrary real-valued scores into a valid probability distribution. The choice of softmax rather than a simpler normalisation reflects a preference inherited from earlier neural network research: the exponential amplification discourages the model from hedging, pushing probability mass toward the most confident predictions. In a classification setting, this was considered desirable. It also made the function's gradient well-behaved during training, which mattered for models trained with backpropagation.
Why it matters now
Softmax remains the standard final-layer activation in nearly every transformer-based language model in production, but its exponential amplification is now widely understood to cause problems during generation. The 2019 paper The Curious Case of Neural Text Degeneration documents how greedy sampling from a softmax distribution — always picking the highest-probability token — produces text that is generic and repetitive, because the model's training objective rewards safe, frequent continuations. The paper argues that human text exhibits more surprisal than maximum-likelihood sampling produces, and it proposes nucleus sampling as an alternative: sample only from the smallest set of tokens whose cumulative probability exceeds a threshold, typically 0.9 or 0.95. This technique, and others like top-k sampling, work by discarding the long tail of low-probability tokens that softmax produces, then renormalising what remains. They are corrections applied after softmax has run, because replacing softmax itself would require retraining the model from scratch. The function is now understood less as an optimal choice and more as an entrenched one.
The surprising detail
Softmax is not the only way to turn scores into probabilities. An alternative called sparsemax, proposed in 2016, produces distributions where many entries are exactly zero rather than merely very small, which can make sampling faster and more interpretable. But sparsemax requires a different backward pass during training, and switching to it mid-stream would invalidate a trained model's weights. The installed base of softmax-trained models is now so large that alternatives remain theoretical curiosities, even when their properties might be preferable. The choice made in 2017 has become effectively irreversible at scale.
Remember this
Softmax converts raw scores into probabilities by exponentiating and normalising. The exponentiation amplifies differences, so small gaps in logits become large gaps in probability.
Test yourself
A model assigns logits of 8.0, 7.5, 7.0 and 2.0 to four candidate tokens. After softmax, roughly what fraction of the total probability mass sits on the top token?
Roughly half. The exponentials are approximately e^8, e^7.5, e^7, and e^2. The first three are e^7 times e, e^0.5, and 1 — so roughly 2.7, 1.6, and 1.0 in relative terms, while e^2 is about 7.4. But e^7 is roughly 1100, so the fourth token contributes almost nothing. The top three sum to about 5.3 times e^7, and the top token is 2.7 times e^7, giving it about 2.7 divided by 5.3, or 51 per cent. The lesson: when several logits are close together, even the highest one does not dominate, because the exponential amplification applies to all of them. The distribution becomes more uniform when the top scores are bunched, and more peaked when one score pulls ahead.
Go deeper
- Attention Is All You Need · arXiv · Ashish Vaswani et al. · 2017-06-12
- The Curious Case of Neural Text Degeneration · arXiv · Ari Holtzman et al. · 2019-04-22
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.