II · THE IDEA · ARTIFICIAL INTELLIGENCE
Predicting the Next Token
▶ Listen · narrated
Every capability you see — translation, reasoning, code generation — emerges from training on a task that sounds almost trivial until you try to do it well.
At a glance
- The task
- Given tokens 1 through n, output a probability distribution over the vocabulary for token n+1
- Training signal
- Billions of examples extracted from text by sliding a window forward one token at a time
- Loss function
- Cross-entropy between predicted distribution and the actual next token
- No labels required
- The text itself provides supervision — each token is the label for the sequence before it
Imagine you are learning a language by reading thousands of books, but the last word of every sentence is covered. Your task is to guess it. At first you guess randomly. Then you notice patterns: certain words follow others, questions end with question marks, stories have structure. You get better. Now imagine the same task, but instead of guessing one word you must write down how likely you think each word in the dictionary is. That is what the model does. It reads a sequence of tokens and outputs a probability for every token in its vocabulary — typically tens of thousands of numbers. During training, it is shown the actual next token and penalised if it assigned that token low probability. The parameters are adjusted to reduce the error, and the process repeats billions of times. Everything the model learns — grammar, facts, reasoning patterns — it learns because knowing those things helps it predict the next token more accurately.
The model is a function from a sequence of token ids to a probability distribution over the vocabulary. During training, a document is tokenised and then split into overlapping examples: tokens 1 to n are the input, token n+1 is the target. The model's output is passed through a softmax to produce a distribution, and the loss is the cross-entropy between that distribution and a one-hot vector for the target token. Gradients are computed and the parameters updated via backpropagation, typically using Adam or a variant.
The model has no explicit representation of tasks. It does not know whether the input is a question, a code snippet, or the middle of a narrative. It learns that certain token patterns correlate with certain continuations because those patterns recur in the training data with sufficient frequency that ignoring them raises the loss. Question-answering, translation, and reasoning emerge as capabilities because the model has compressed into its parameters the statistical structure of text written by humans performing those tasks.
The architecture is autoregressive: each token is predicted from all previous tokens and no future ones. This is enforced by a causal attention mask, which prevents position i from attending to position j when j > i. During generation, the model produces one token, appends it to the context, and runs a forward pass to predict the next one. The output distribution can be sampled, or the highest-probability token can be chosen greedily, or a more sophisticated decoding strategy applied.
Scaling laws, documented by Kaplan et al., show that test loss falls as a power law in model parameters, dataset size, and training compute, with no sign of saturation at the scales tested. The implication is that next-token prediction is not a task the model solves, but one it continues to improve at with more resources.
Look closer
The output is a distribution, not a single token
The model does not point at one answer. It assigns a probability to every entry in the vocabulary — typically tens of thousands of numbers that sum to one. During training, the model is penalised according to how much probability mass it placed on the token that actually came next. During generation, that distribution is sampled or the highest-probability token is chosen, then the process repeats with the new token appended to the context.
The same task at every position
Whether the model has seen three tokens or three thousand, the objective is identical: predict the next one. There is no separate training regime for questions versus statements, for the first word of a paragraph versus the last. The model learns that certain patterns — a question mark, an instruction, a half-finished argument — correlate with particular continuations, but those patterns are discovered, not labelled.
Correct prediction requires world knowledge
To predict well, the model must compress into its parameters everything that makes one continuation more likely than another: grammar, facts, narrative convention, logical consistency, the typical structure of a Python function. A model trained only to minimise next-token prediction error has no explicit representation of truth or meaning, yet it must approximate both to lower the loss on text written by humans who care about truth and meaning.
The story
The task is stated in one sentence. You are given a sequence of tokens. Predict a probability distribution over the vocabulary for the next one. Repeat.
That is the entire training objective for GPT-3, for LLaMA, for every decoder-only language model in wide use. No human writes labels. No annotator marks which sentences are questions or which paragraphs are summaries. The model is simply shown vast quantities of text, and at each position it guesses what comes next. The actual next token is right there in the data, so the model's prediction can be scored immediately. The loss is higher when the model assigned low probability to the token that actually occurred. The parameters are adjusted to lower that loss. Then the window slides forward and the process repeats.
The method is sometimes called self-supervised learning, because the supervision signal is extracted from the data itself rather than added by a human. It scales in a way that labelled datasets do not. As long as you can find more text, you can generate more training examples — trillions of them, each one a context and a target.
What makes this task hard is that language is not random. The next token depends on grammar, on the topic established three sentences ago, on facts about the world, on the genre and register of the text. To predict well, the model must learn structure at every scale: which letters follow which, which words belong in a clause together, how an argument unfolds, what is plausible given everything said so far.
Kaplan and colleagues, in their 2020 paper on scaling laws, measured how prediction accuracy improves as models grow larger and see more data. The relationship is smooth and predictable across many orders of magnitude. Loss falls as a power law in model size, in dataset size, and in the amount of computation used for training. The implication is that next-token prediction is not a task you solve and finish. It is a task you get better at, incrementally, by scaling.
Brown and colleagues, in the GPT-3 paper published the same year, demonstrated that a model trained only on next-token prediction could perform tasks it had never been explicitly taught: translation between languages, arithmetic, question answering, even generating code from a description. The tasks were specified in the prompt, as context, and the model continued the pattern. The behaviour was not programmed. It emerged because the model had learned, from vast text, that certain patterns of tokens — a question followed by an answer, a problem followed by a solution — are common, and predicting them well requires general capabilities.
There is no point during training when the model is told what a question is, or what it means to summarise. It is only ever told whether its probability distribution was well-calibrated for the next token. Everything else is a consequence of trying to lower that loss on human-written text.
Why it mattered then
The choice of next-token prediction as the training objective was partly pragmatic. It required no human labelling, which meant it could use the entire internet's worth of text rather than a small annotated dataset. It was also theoretically appealing: Shannon had shown in 1951 that a good model of a sequence must capture the statistical structure of the language, and next-token prediction is a direct way to learn that structure. But the critical discovery, made empirically between 2018 and 2020, was that the task generalises. Models trained only to predict the next token developed capabilities that looked like reasoning, translation, and question answering — tasks that researchers had previously built separate systems to handle. The scaling laws published by Kaplan and colleagues suggested that the generalisation would continue as models grew. That changed the research programme. If next-token prediction was sufficient to produce broad capabilities, and if the loss continued to fall with scale, then the path forward was clear: more parameters, more data, more compute.
Why it matters now
Next-token prediction remains the core training objective for the largest and most capable language models in deployment. Instruction tuning and reinforcement learning from human feedback are applied afterward, but they are adjustments to a model that has already learned the structure of language through next-token prediction on raw text. The objective also explains certain behaviours. A model will sometimes continue a pattern even when the continuation is false, because the pattern is statistically strong. It will sometimes refuse a question it could answer, because refusal is a common continuation in text where the question is inappropriate. It will sometimes produce a confident-sounding sentence that contradicts the previous one, because the training signal is local — each token is predicted from the context immediately before it, and long-range consistency is learned only insofar as inconsistency raises the loss. Understanding that the model was trained only to predict the next token helps calibrate expectations. The model has no explicit goal of being truthful, helpful, or consistent. Those properties emerge when they help predict human-written text, and they fail when the statistical signal is weak or misleading.
The surprising detail
The task is not symmetrical. The model predicts the next token given all the previous ones, but it is never trained to predict a previous token given the ones that follow. This is why models are far better at continuing a sentence than at filling in a missing word from the middle. The architecture enforces it: each position can only attend to earlier positions, never to later ones. That restriction, called causal masking, is what makes the model autoregressive — it generates one token at a time, each one conditioned on all the tokens before it and none of the tokens after. The asymmetry is a choice, not a necessity, but it makes generation straightforward and training efficient.
Remember this
Every capability emerges from training on one task: guess the next token, then update the parameters to make that guess better.
Test yourself
A model trained only on next-token prediction is then shown a prompt containing a question it has never seen, followed by the start of an answer. It continues the answer correctly. No human labelled this as a question-answering example. Explain how the training objective alone could produce this behaviour.
The training data contained many examples of questions followed by answers, because that is a common pattern in human-written text — forums, textbooks, interviews, FAQs. To predict the tokens that come after a question, the model had to learn what makes a good answer: relevance to the question, factual accuracy, appropriate register. It was never told 'this is a question' or 'this is an answer'. It learned the pattern because assigning high probability to the actual continuation required recognising the structure. When the prompt provides a question and the start of an answer, the model continues in a way that lowers the loss on similar examples it saw during training. The behaviour generalises because the statistical structure of question-answer pairs is consistent enough that a model optimising next-token prediction must approximate it.
Go deeper
- Language Models are Few-Shot Learners · arXiv · Tom B. Brown et al. · 2020-05-28
- Scaling Laws for Neural Language Models · arXiv · Jared Kaplan et al. · 2020-01-23
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.