II · THE IDEA · ARTIFICIAL INTELLIGENCE
Temperature
▶ Listen · narrated
A model at temperature 0.2 writes like a committee minutes-taker. The same model at 1.4 writes like someone free-associating after two drinks. One parameter, profoundly different behaviour.
At a glance
- What it does
- Divides every logit by the temperature value before softmax
- Below 1.0
- Sharpens the distribution, concentrating probability on the top choices
- Above 1.0
- Flattens the distribution, spreading probability more evenly
- At exactly 1.0
- No change; the model's original probabilities are preserved
Imagine a bag of coloured marbles. Most are blue, some are red, a few are green. If you draw one marble at random, you will probably get blue. Temperature changes how many of each colour are in the bag before you draw. Low temperature adds more blue marbles and removes most of the green ones, making blue almost certain. High temperature adds more green and red marbles, making the draw much less predictable. The model is the bag, the tokens are the marbles, and temperature adjusts their proportions. At temperature 1.0, you draw from the bag the model originally prepared. Below 1.0, you have made the likely choices even more likely. Above 1.0, you have given the unlikely choices a better chance.
Temperature is applied by dividing each logit by the temperature scalar before the softmax operation. Given logits z and temperature T, the probability of token i becomes exp(z_i / T) divided by the sum of exp(z_j / T) over all tokens j. When T is less than 1, division increases the logits, widening the gaps between them, and softmax exponentiation amplifies those gaps into a sharper probability distribution. When T is greater than 1, division reduces the logits and compresses the gaps, producing a flatter distribution after softmax. At T equals 1, the operation is the identity. At T approaching zero, the model becomes deterministic argmax. At T approaching infinity, the distribution becomes uniform. Temperature is differentiable, so it can be used during training for techniques like knowledge distillation, where a student model learns from the softened outputs of a teacher. In inference, it is applied after the final linear layer and before sampling, and it has no learned parameters. The effective range is typically 0.1 to 2.0; beyond that, output quality degrades rapidly.
Look closer
The arithmetic is applied before softmax
Temperature does not touch the probabilities directly. It divides the raw logits — the unnormalised scores — and then softmax converts those adjusted logits into probabilities. Because softmax is exponential, small changes to logits produce large changes to probabilities, and the effect compounds. At temperature 0.5, a logit of 4.0 becomes 8.0 before softmax sees it, which dramatically increases its final probability. At temperature 2.0, that same logit becomes 2.0, and its advantage shrinks.
Temperature zero is a special case
Dividing by zero is undefined, so temperature 0.0 is implemented as a limit: the model simply picks the highest logit every time, with no sampling at all. This is deterministic argmax decoding, sometimes called greedy decoding. It guarantees the same output for the same prompt, but it also guarantees repetition traps if the highest-probability path leads into a loop, because there is no randomness to break out.
The same temperature behaves differently in different models
Temperature is applied to whatever logits the model produces, and models differ in how confident they are. A model trained with label smoothing or one that has learned to hedge will produce flatter logit distributions to begin with, so temperature 0.7 on that model may feel like 1.0 on a sharper one. There is no universal calibration. The numbers that work well are found empirically for each model, and sometimes for each task.
The story
Imagine the model has just computed logits for the next token. Perhaps "the" has a logit of 8.2, "a" has 7.9, "an" has 5.1, and two hundred other tokens trail off into negative numbers. These are raw scores, not probabilities yet.
Temperature divides every one of those logits by a single number before softmax turns them into probabilities. If temperature is 0.5, the 8.2 becomes 16.4 and the 7.9 becomes 15.8. The gap between them has doubled. When softmax exponentiates those larger numbers, the probability mass concentrates even more heavily on "the". The long tail of unlikely tokens becomes vanishingly unlikely.
If temperature is 2.0, the opposite happens. The 8.2 becomes 4.1, the 7.9 becomes 3.95, and the gap between them has halved. After softmax, "the" is still more likely than "a", but not by much, and tokens that were distant long shots now have enough probability to be sampled occasionally. The distribution has flattened.
At temperature 1.0, nothing changes. Dividing by one is the identity operation, and you get whatever probabilities the model originally implied.
This is why temperature is often described as controlling randomness, but that is not quite right. It controls the shape of the distribution from which you sample. Low temperature makes the model more deterministic because most of the probability sits on one or two tokens. High temperature makes it more exploratory because probability spreads across many candidates. But the randomness itself — the sampling step — remains unchanged. You are simply sampling from a different shape.
The effect is non-linear and it accelerates at the extremes. Temperature 0.1 produces near-identical output to temperature 0.2, both very sharp. Temperature 1.8 and 2.0 both feel quite loose, though 2.0 wanders further. The middle range, roughly 0.7 to 1.2, is where most practical work happens, and where small adjustments produce noticeably different character without breaking coherence.
Why it mattered then
Temperature was borrowed from statistical mechanics, where it describes the energy distribution of particles in a system. In the 1980s, researchers training Boltzmann machines and using simulated annealing found that dividing energy terms by a temperature parameter let them control exploration during optimisation. High temperature allowed the system to jump between states freely; low temperature let it settle into a minimum. When neural language models began using softmax to convert scores into probabilities, the same mechanism applied. Dividing logits by temperature before softmax gave researchers a way to adjust how confidently the model committed to its top choices without retraining. It was a simple, differentiable operation that fit neatly into the existing architecture, and it required no additional parameters to learn.
Why it matters now
Temperature remains the first parameter most practitioners adjust when a model's output feels wrong. If the text is repetitive or dull, raise it. If the model is incoherent or hallucinating, lower it. The adjustment is immediate, requires no retraining, and applies to any model that uses softmax. But temperature also exposes a limitation. It is a single global knob applied uniformly to every token in the vocabulary at every step. It cannot distinguish between positions where the model is genuinely uncertain and should explore, and positions where the model is confident and should not. More recent techniques — top-p sampling, top-k, min-p — attempt to make that distinction by truncating the distribution instead of reshaping it, but temperature remains ubiquitous because it is simple, smooth, and well understood. Every API exposes it, and most users encounter it before they encounter any other sampling parameter.
The surprising detail
The paper that introduced nucleus sampling, also called top-p, includes a striking observation: human text does not come from a constant-temperature distribution. When you write, some words are forced by grammar or idiom — there is only one token that can come next — and others are wide open. A fixed temperature cannot match that variation. It either flattens the forced choices, introducing errors, or sharpens the open choices, making the text feel templated. The authors argue that truncation methods, which adapt to each position's entropy, produce more human-like text than any fixed temperature can. But temperature remains dominant in practice, perhaps because it is easier to explain and reason about than a dynamic probability threshold.
Remember this
Temperature reshapes the distribution before you sample from it. It does not add randomness; it changes where the randomness can land.
Test yourself
You set temperature to 0.5 and generate three completions from the same prompt with different random seeds. Then you set temperature to 2.0 and do the same. In which case are the three outputs more likely to differ from one another, and why?
At temperature 2.0. Higher temperature flattens the probability distribution, so more tokens have non-negligible probability and different random seeds are more likely to select different tokens at each step. At temperature 0.5, the distribution is very sharp — most of the probability mass sits on one or two tokens — so even with different seeds, the sampling process will usually pick the same high-probability token. The outputs at 0.5 may still differ eventually, because any divergence compounds over multiple steps, but they will be much more similar to one another than the outputs at 2.0, which are sampling from a much wider set of plausible candidates at every position.
Go deeper
- The Curious Case of Neural Text Degeneration · arXiv · Ari Holtzman et al. · 2019-04-22
- On Calibration of Modern Neural Networks · arXiv · Chuan Guo et al. · 2017-06-14
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.