II · THE IDEA · ARTIFICIAL INTELLIGENCE
Training Versus Inference
▶ Listen · narrated
When you ask a model a question, no learning is happening. The system consulting those billions of parameters was built in a completely different process, at a completely different scale.
At a glance
- Training
- Adjusts weights by gradient descent across a corpus; weeks to months on clusters
- Inference
- Reads frozen weights to produce output; milliseconds to seconds per query
- Cost ratio
- Training a frontier model costs millions; one inference call costs fractions of a penny
- What changes
- Training mutates every weight. Inference changes nothing in the model itself.
Training is like recording an album in a professional studio over months, with a full production team and a budget in the millions. Inference is like pressing play on the finished recording—it costs almost nothing, happens instantly, and you can do it as many times as you want without changing the album itself. The recording session adjusts every instrument, every mix level, every effect until the song sounds right. Playing it back just reads what was already decided. In the same way, training adjusts billions of numbers inside a model by showing it examples and measuring errors, a process that takes weeks on thousands of computers. Inference reads those frozen numbers to answer your question in seconds on one machine. The model file is the same in both cases, but what you are doing with it is completely different.
Training is stochastic gradient descent over a dataset, typically using the Adam optimiser or a variant. For each batch, the model performs a forward pass to compute predictions, calculates loss (usually cross-entropy for language modelling), then backpropagates gradients through all parameters and updates weights. This requires storing activations for the backward pass, which approximately doubles memory usage compared to inference. For large models, training is parallelised across multiple devices using data parallelism, tensor parallelism, or pipeline parallelism, with gradient synchronisation at each step. The process runs for a fixed number of tokens—GPT-3 saw 300 billion tokens during training—and the compute cost scales as O(N) in model parameters and O(D) in dataset size, following the scaling laws derived by Kaplan et al.
Inference is a single forward pass with frozen weights. The model loads parameters as read-only, processes the input through each layer, and samples from the output distribution. For autoregressive generation, this repeats for each token, with the key-value cache storing attention states to avoid recomputing them. Memory usage is dominated by the cache for long contexts, not by the model weights. Inference cost scales linearly with generation length and context size, but not with the training dataset—the model's behaviour is determined entirely by the learned weights. Quantisation, pruning, and distillation are inference-time optimisations that reduce precision or size; they do not apply during training, where full precision is needed for stable gradient updates. The two regimes use different hardware profiles: training favours high-bandwidth interconnects and FP32 or mixed precision, while inference favours low latency, high throughput, and often lower precision like INT8 or FP16.
Look closer
Training requires backward passes through the entire network
For each batch of examples, the system runs forward to produce predictions, measures error, then propagates that error backward through every layer to compute how each weight should change. This backward pass roughly doubles the memory requirement compared to inference, because you must hold intermediate activations to compute gradients. Inference runs only forward, discarding activations as soon as the next layer consumes them.
Hardware choices diverge completely
Training clusters prioritise high-bandwidth interconnects between thousands of accelerators, because gradient updates must synchronise across the entire corpus. A single training run for a model at GPT-3 scale involved thousands of GPUs coordinated over weeks. Inference deployments prioritise low latency and high throughput per chip, often using different silicon entirely—quantised weights, smaller batch sizes, sometimes CPUs for small enough models. The two regimes optimise for opposite things.
Inference cost scales with generation length, training cost does not
During training, the model sees fixed-length sequences and computes loss across all positions in parallel. Cost per example is essentially constant. During inference, autoregressive generation produces one token at a time, and each new token requires another full forward pass through the network with a longer context. A hundred-token reply costs roughly a hundred times more compute than a one-token reply, which is why inference pricing is usually per token rather than per request.
The story
The word model refers to two completely different computational realities. When a researcher says they are training a model, they mean running an optimisation process that adjusts billions of parameters over days or weeks, consuming more electricity than a small town. When you send a prompt to an API and receive a reply, that is inference: a single forward pass through a frozen set of weights, completed in seconds.
Training is an iterative gradient descent across a corpus. The system reads batches of text, predicts what comes next, measures how wrong it was, then uses calculus to apportion blame backward through every layer. Each weight shifts a tiny amount in the direction that would have reduced the error. Repeat this across trillions of tokens and the weights converge toward a configuration that compresses statistical regularities in the training data. The process requires holding activations in memory for the backward pass, synchronising gradients across thousands of accelerators, and checkpointing state in case hardware fails mid-run. Kaplan et al. derived scaling laws relating compute budget, model size and dataset size: training cost grows predictably with the number of parameters and the length of the corpus, and both must scale together to improve performance efficiently.
Inference uses the result of that process but performs none of the work. The weights are loaded into memory as read-only data. A prompt arrives, is tokenised, and flows forward through the layers. Each transformer block applies its learned attention patterns and feed-forward transformations. The final layer produces a probability distribution over the vocabulary, a token is sampled, and that token is appended to the context for the next step. No gradients are computed. No weights change. The model after a million inferences is byte-for-byte identical to the model before them.
The computational profiles are so different that the hardware often is too. Training a model the size of GPT-3 required approximately 3.14 × 10²³ floating-point operations, spread across thousands of GPUs over weeks. Brown et al. report that training consumed several thousand petaflop-days. Inference for a single query on the same model requires only one forward pass—billions of operations rather than trillions of trillions—and completes in seconds on a single accelerator. The cost per token generated is measured in fractions of a cent. The cost to train the model in the first place is measured in millions of dollars.
This asymmetry explains why pre-trained models are valuable. Training is so expensive that only a few organisations can afford to do it at frontier scale, but once the weights exist they can be copied freely. Inference is cheap enough to offer commercially. The entire ecosystem of model APIs, fine-tuning services, and local deployments rests on the fact that using a model costs five or six orders of magnitude less than creating it.
Why it mattered then
The separation mattered because it determined who could participate. In 2020, when Brown et al. published the GPT-3 paper, training a model of that scale required resources available only to well-funded labs. The compute budget alone—several thousand petaflop-days—translated to millions of dollars in hardware time, and the engineering effort to coordinate that training run across a cluster was substantial. Making the resulting model available through an API rather than releasing the weights was partly a safety decision and partly an economic one: inference was cheap enough to monetise per query, while training was expensive enough to be a significant barrier to entry. The asymmetry created a new market structure in which a small number of organisations trained foundation models and a much larger number built applications on top of them.
Why it matters now
The distinction still governs the economics and accessibility of the technology. Training costs continue to rise as models grow larger—frontier models now require compute budgets that exceed GPT-3 by another order of magnitude—but inference costs have fallen through quantisation, better hardware, and architectural improvements. This divergence has enabled new deployment patterns: small models running locally on consumer hardware for privacy-sensitive applications, and large models served centrally at scale for tasks that justify the cost. It also shapes research priorities. Techniques that reduce training cost—such as more efficient optimisers or better data curation—are valuable but benefit only those running training jobs. Techniques that reduce inference cost—such as speculative decoding or sparse attention—benefit everyone using the model, which is a much larger population. The two regimes remain as separate now as they were when the scaling laws were first derived, and understanding which regime you are working in determines what optimisations matter and what the cost structure will be.
The surprising detail
Inference does not learn, but it is not entirely stateless either. During a single inference session, the key-value cache grows with every token generated, holding the computed attention keys and values for all previous tokens in the context. This cache is essential for efficiency—without it, you would recompute attention over the entire context from scratch for each new token—but it is also memory-hungry. For long conversations, the cache can consume more memory than the model weights themselves. It persists only for the duration of one request, though, and is discarded afterward. The model itself remains unchanged, but the session carries state that is, in a narrow sense, learned from your prompt. This is why context window size is an inference-time constraint, not a training-time one.
Remember this
Training mutates the weights and costs millions. Inference reads them and costs cents. The two processes share a model file and nothing else.
Test yourself
You have a fixed budget of ten thousand dollars and access to a cluster. You can either train a small model from scratch or run inference on a much larger pre-trained model for millions of queries. Name two factors that would make training the better choice, despite the size disadvantage.
First, if your task or domain is badly mismatched to available pre-trained models—perhaps you need a model for a low-resource language, or for a specialised corpus like legal documents or protein sequences—training on your own data may produce better performance than prompting a general-purpose model, even if the trained model is smaller. Second, if you need full control over the model weights for deployment, fine-tuning, or compliance reasons, training gives you an asset you own outright, while API inference leaves you dependent on an external service that may change its terms, pricing, or availability. A third consideration: if your application will require enough inference volume over time, the cumulative cost of API calls can eventually exceed the one-time cost of training, making ownership economically rational in the long run.
Go deeper
- Language Models are Few-Shot Learners · arXiv · Tom B. Brown et al. · 2020-05-28
- Scaling Laws for Neural Language Models · arXiv · Jared Kaplan et al. · 2020-01-23
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.