II · THE IDEA · ARTIFICIAL INTELLIGENCE
Parameters, Weights, Layers, Activations
▶ Listen · narrated
You will hear all four words in the same sentence, sometimes as if they mean the same thing. They do not, and the distinction matters the moment you try to count anything.
At a glance
- Parameter
- Any learnable number in the model
- Weight
- A parameter that multiplies an input
- Layer
- A group of operations applied together
- Activation
- A number computed during forward pass
Think of a recipe that produces a meal. The recipe itself — the instructions and ingredient proportions — is like the parameters: fixed for a particular dish, learned by the chef over time. The weights are the specific proportions that multiply ingredients: two parts flour to one part water. The layers are the steps: mix, knead, bake. The activations are the intermediate states of the dough as it moves through those steps: the texture after kneading, the rise after proofing. The dough is not part of the recipe. It exists only while you are cooking, and it changes depending on what you started with. A network is the same: parameters are learned and saved, activations are computed fresh every time.
In a feedforward network, parameters are the entries in weight matrices and bias vectors, along with any other learned values such as batch normalisation statistics or embedding table entries. Weights specifically are parameters that appear as multiplicative coefficients in linear transformations, typically organised as matrices. Biases are parameters but not weights. A layer is an abstraction grouping one or more operations and their associated parameters; the boundary is a matter of implementation convention rather than mathematical necessity. PyTorch defines a layer as any subclass of nn.Module, which can contain other layers recursively. Activations are the intermediate tensors computed during a forward pass. They are not parameters and are not saved with the model. During training, activations from earlier layers must be retained in memory for the backward pass, which is why training memory scales with depth and batch size while inference memory does not. Gradient checkpointing mitigates this by recomputing activations on demand during the backward pass, trading increased computation for reduced memory. The term activation is also used for nonlinear functions applied elementwise, such as ReLU, GELU or sigmoid, which is a separate meaning that must be distinguished from context.
Look closer
Parameters include things that are not weights
A bias term is a parameter — it is learned during training — but it is added to a sum rather than multiplying anything, so it is not a weight. In a batch normalisation layer, the scale and shift are parameters. In an embedding table, every entry is a parameter. The term parameter is the broadest: it means any number the training process adjusts. Weights are the subset that appear as multipliers in linear transformations.
Layers are organisational, not mathematical
A layer is a named grouping of operations, usually with its own parameters. The boundary is a matter of implementation convenience. What PyTorch calls a single Linear layer — a matrix multiplication followed by a bias addition — could equally be split into two operations or merged into a larger block. The layer is the unit you address in code, not a distinct mathematical object. This is why layer count is a coarse measure: a transformer block in one codebase may be a single layer, while another implementation counts the attention and feedforward sections separately.
Activations are ephemeral unless you save them
During a forward pass, each layer produces a tensor of numbers that flows into the next operation. Those numbers are the activations. They exist only for the duration of the computation, and they are normally discarded once the next layer has consumed them — except during training, when some must be kept in memory so gradients can be calculated on the backward pass. The term is also used for the nonlinear function applied after a linear layer, which is confusing but standard: the ReLU activation function produces ReLU activations.
The story
Consider a small feedforward network with an input of size three, a hidden layer of four units, and an output of size two. The input arrives as three numbers. The first layer multiplies them by a 4×3 weight matrix and adds a bias vector of length four, producing four numbers. A nonlinearity is applied elementwise — say, ReLU, which replaces negative values with zero. Those four numbers are the hidden activations. They are multiplied by a 2×4 weight matrix, a bias vector of length two is added, and the result is two output activations.
The parameters are every number that training adjusts. The first weight matrix contributes twelve parameters. The first bias contributes four. The second weight matrix contributes eight, and the second bias contributes two. The total parameter count is twenty-six. None of the activations are parameters, because they change with every different input and are not learned.
The weights are the entries in the two matrices: twenty parameters in total. The biases are parameters but not weights, because they are added rather than multiplied. If you hear someone say this network has twenty-six weights, they are using the word loosely to mean parameters, which is common in informal speech but wrong in any context where the distinction matters — such as when you are trying to understand why a particular operation contributes to memory usage or computation time.
Layers are the organisational units. In the example above, you might reasonably call it a two-layer network — one hidden layer and one output layer — or a three-layer network if you count the input as a layer, which some conventions do and others do not. The ambiguity is not a fault in the terminology; it reflects the fact that layer is a structural label, not a mathematical one. What matters is that each layer groups a set of operations and parameters together so you can refer to them, initialise them, freeze them or inspect them as a unit.
Activations are the numbers that flow through the network during a forward pass. They depend on the input and on the current values of the parameters. If you change the input, the activations change. If you update the parameters during training, the activations produced by the same input will change. They are not stored in the model file. When you save a trained network, you save only the parameters. The activations must be recomputed every time you run an input through.
Why it mattered then
The terminology solidified as neural networks moved from theory to implementation. Early papers often used weight and parameter interchangeably, because the models were simple enough that nearly every parameter was a weight in a linear transformation. As architectures grew more complex — adding normalisation layers, attention mechanisms, embeddings and gating functions — the need for precision increased. Parameter became the umbrella term, and weight narrowed to its specific meaning. Layer remained somewhat loose, because it serves an engineering purpose rather than a mathematical one: it is the unit of abstraction that lets you build, debug and describe a network without listing every operation. The distinction between parameters and activations mattered most for memory management, because training a deep network requires holding activations from earlier layers in memory until the backward pass completes, and that memory cost scales with batch size and depth in a way that parameter memory does not.
Why it matters now
The distinctions matter more now because models are large enough that imprecise language leads to real confusion. When someone says a model has seven billion weights, do they mean parameters? If so, are they counting only the weights, or everything learnable? The difference can be tens of millions of parameters in models with extensive normalisation. When a paper reports layer count, are they counting transformer blocks, or individual operations within each block? The number can differ by a factor of two or more depending on convention. Activations have become a bottleneck in their own right: techniques like gradient checkpointing trade computation for memory by discarding activations during the forward pass and recomputing them when needed, which only makes sense if you understand that activations and parameters are stored and managed separately. In conversations about model efficiency, quantisation, pruning and distillation, using these four words correctly is not pedantry — it is the minimum precision needed to know what is actually being measured or optimised.
The surprising detail
The word activation has two meanings in common use, and the ambiguity is rarely acknowledged. It can mean the output of a layer — the tensor of numbers flowing forward — or it can mean the nonlinear function applied within a layer, such as ReLU or sigmoid. You will see both in the same paragraph. A paper might say a model uses GELU activations, meaning the function, and then describe storing activations during training, meaning the tensors. The context usually resolves it, but not always. This dual use is inherited from neuroscience, where activation referred both to the firing rate of a neuron and to the process that caused it to fire.
Remember this
Parameters are learned, weights multiply, layers group, activations flow. The words name different things.
Test yourself
A model has 120 million parameters. You run a single input through it and measure peak memory usage, then run a batch of thirty-two inputs and measure again. The memory used for parameters has not changed. What has changed, and why does it scale with batch size?
The activations have changed. During a forward pass, each layer produces a tensor of intermediate values that must be kept in memory until the next layer consumes them — and during training, many must be retained until the backward pass so gradients can be computed. Those tensors scale with batch size because each item in the batch produces its own set of activations at every layer. Parameters, by contrast, are shared across the entire batch: the same weight matrix multiplies every input. This is why training memory usage grows much faster than inference memory usage, and why larger batches eventually hit memory limits even when the model itself fits comfortably.
Go deeper
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.