II · THE IDEA · ARTIFICIAL INTELLIGENCE
Top-k, Top-p and Min-p
▶ Listen · narrated
A model assigns non-zero probability to thousands of next tokens, including many that would derail the sentence. Sampling strategies decide which of those thousands to keep in the draw.
At a glance
- Top-k
- Keep only the k most probable tokens, discard the rest
- Top-p (nucleus)
- Keep the smallest set whose cumulative probability reaches p
- Min-p
- Keep only tokens above p times the top token's probability
- Typical values
- k around 40–50; p around 0.9–0.95; min-p around 0.05
Imagine you are choosing a card from a deck, but someone has written a probability on each card — how likely it is to be the right choice. Most cards have probabilities so low they are effectively useless. Top-k says: throw away all but the top fifty cards, then choose randomly from what is left. Top-p says: throw away cards until the ones you keep add up to 95 per cent of the total probability, however many that takes. Min-p says: throw away any card whose probability is less than 5 per cent of the best card. All three are trying to solve the same problem — if you keep the whole deck, you will occasionally draw a terrible card, but if you keep only the single best card, you lose all variety.
After the model computes logits for the entire vocabulary, you apply temperature scaling if desired, then convert to probabilities via softmax. Truncation happens next. Top-k sorts the vocabulary by probability, keeps the top k entries, and sets the rest to zero. Top-p sorts by probability, walks down the list accumulating mass, and cuts off when the cumulative probability reaches p. Min-p compares each token's probability to the maximum, keeps those above p times that maximum, and discards the rest. In all cases you then renormalise the kept probabilities so they sum to one, and sample from the resulting categorical distribution. The strategies can be combined: you might apply min-p first to remove the extreme tail, then apply top-p to the remainder. Temperature and truncation interact non-trivially — temperature reshapes the distribution before truncation, so a high temperature with a low top-p can produce a very different nucleus than a low temperature with the same top-p. Implementations differ in whether they apply truncation before or after other transformations like repetition penalties, and the order matters. Top-p is sometimes called nucleus sampling in the literature, after Holtzman et al., but the parameter itself is usually labelled top_p in APIs.
Look closer
Top-k keeps a fixed number regardless of the shape
If you set k to 50, the model samples from exactly fifty tokens every time, whether the top token has 90 per cent of the probability mass or 20 per cent. When the distribution is peaked — one token dominates — you are forcing the model to consider forty-nine implausible alternatives. When the distribution is flat — uncertainty is genuine — fifty may not be enough to capture the reasonable options. The number is rigid; the distribution is not.
Nucleus sampling adjusts the cutoff to the certainty
Top-p sorts the tokens by probability, then walks down the list adding them until their cumulative probability reaches the threshold p. If the model is confident, the nucleus is small — perhaps three tokens cover 95 per cent of the mass. If the model is uncertain, the nucleus grows to include dozens. The strategy was named nucleus sampling by Holtzman and colleagues in 2019 because you are sampling from the probability nucleus, the core that contains most of the mass, and discarding the long tail.
Min-p is relative to the top token
Min-p keeps only tokens whose probability is at least some fraction of the most probable token. If the top token has probability 0.4 and min-p is 0.05, any token below 0.02 is discarded. The threshold moves with the peak, so the strategy adapts to both confident and uncertain distributions, but in a different way from nucleus sampling. It tends to preserve more of the tail when the distribution is flat, which some practitioners prefer for creative tasks.
The story
At each step of generation, a language model produces a probability distribution over its entire vocabulary — often tens of thousands of tokens. Most of those probabilities are vanishingly small, but they are not zero, and if you sample from the full distribution you will occasionally draw a token from the extreme tail. That token may be rare for good reason: it might be a typo the model saw in training data, a word from the wrong language, or simply a choice that makes no sense in context. The result is often called degeneration — text that starts coherent and then veers into repetition, contradiction, or outright nonsense.
The simplest fix is greedy decoding: always pick the single most probable token. This eliminates the tail entirely, but it also eliminates variety. Greedy decoding produces the same output every time, and that output is often flat and repetitive, because the most probable next token is frequently a safe, common word that does not advance the thought.
Top-k sampling, introduced in hierarchical story generation work by Fan and colleagues, offers a compromise. You keep only the k most probable tokens, set the rest to zero, renormalise the probabilities over what remains, and sample from that truncated distribution. If k is 50, you are choosing randomly among the top fifty options at each step, which introduces variety without diving into the implausible tail. The problem is that fifty is an arbitrary number. Sometimes the model is very confident and the top token alone has most of the mass; sometimes the model is uncertain and probability is spread across hundreds of plausible continuations. A fixed k treats both situations identically.
Nucleus sampling, or top-p, solves this by making the cutoff adaptive. Instead of keeping a fixed number of tokens, you keep however many it takes to reach a cumulative probability of p — typically 0.9 or 0.95. If the distribution is peaked, the nucleus might be just two or three tokens. If it is flat, the nucleus expands to include dozens. Holtzman and colleagues showed in 2019 that this substantially reduces degeneration compared to both greedy decoding and fixed top-k, because you are sampling from the part of the distribution the model actually finds plausible, and that part changes size depending on the context.
Min-p takes a different approach to the same problem. Instead of cumulative mass, it uses a threshold relative to the top token: keep any token whose probability is at least p times the probability of the most likely token. If the top token has 40 per cent and min-p is 0.05, anything below 2 per cent is discarded. When the model is very confident, the threshold is high and the set is small. When the model is uncertain and the top token itself has low probability, the threshold drops and more options survive. The strategy is newer and less studied than nucleus sampling, but some users report that it handles flat distributions more gracefully, particularly for creative or exploratory generation where you want the model to consider a wider range.
Why it mattered then
The motivation was empirical and immediate. As models grew larger and were deployed in interactive settings — chatbots, story generators, code assistants — users noticed a recurring failure mode: the output would start well and then collapse into repetition or incoherence. Holtzman and colleagues documented this systematically in 2019, showing that sampling from the full distribution or using a fixed top-k both led to degeneration, and that nucleus sampling reduced it substantially without sacrificing diversity. The paper was titled The Curious Case of Neural Text Degeneration, and the curiosity was that maximising likelihood — the training objective — did not produce the best text. Greedy decoding and beam search, which approximate likelihood maximisation, were measurably worse by human judgement than sampling strategies that introduced randomness but truncated the tail. The tail, it turned out, was where the nonsense lived.
Why it matters now
These strategies are now standard hyperparameters in every inference API and local sampling library. When you adjust the creativity or randomness of a model, you are often adjusting top-p or temperature, and the two interact: temperature flattens or sharpens the distribution before truncation, and truncation decides which part of that reshaped distribution to sample from. The defaults matter, because most users never change them, and there is no single correct setting. A low top-p makes the model more deterministic and factual but also more repetitive. A high top-p allows more variation but risks incoherence. The choice depends on the task, and increasingly on user preference, which is why interfaces now expose these parameters as sliders. The underlying insight remains: sampling from the full distribution is a mistake, and the right truncation strategy depends on how confident the model is at that moment, not on a number you set once at the start.
The surprising detail
Holtzman and colleagues found that human-written text, when evaluated by a language model, tends to fall in the nucleus but not at the very peak. Humans do not always choose the most probable next word. They choose from the plausible set, and that set is usually the nucleus. This suggests that nucleus sampling is not just a trick to avoid bad outputs — it may also be a better model of how humans actually generate language, balancing predictability with surprise in a way that greedy decoding does not.
Remember this
The tail of the distribution is where the nonsense lives. Truncation strategies decide how much of it to keep.
Test yourself
You are generating a story and notice the output has become repetitive, choosing safe common words at every step. You are using top-p set to 0.95. Name two changes you could make to the sampling settings, explain what each one does to the distribution, and say which failure mode each risks introducing.
First, you could raise top-p closer to 1.0, or switch to a higher top-k, or raise the temperature. Raising top-p expands the nucleus to include more of the tail, so the model samples from a wider set of options. Raising temperature flattens the distribution before truncation, shifting probability mass from the peak toward the rest. Both increase variety, but both also increase the risk of incoherence or degeneration, because you are now sampling from parts of the distribution the model considered less plausible. Second, you could lower min-p if you are using it, which has a similar effect: it admits more tokens by lowering the relative threshold. The trade-off is always the same — more diversity, more risk of nonsense. A third option, not strictly a sampling change, is to adjust the prompt or the context to give the model a less obvious continuation, so the distribution itself is less peaked before you sample from it.
Go deeper
- The Curious Case of Neural Text Degeneration · arXiv · Ari Holtzman et al. · 2019-04-22
- Hierarchical Neural Story Generation · arXiv · Angela Fan et al. · 2018-05-13
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.