II · THE IDEA · ARTIFICIAL INTELLIGENCE
What a Neural Network Actually Computes
▶ Listen · narrated
Every layer in a deep network performs the same three operations. The third one—a simple, fixed function—is the only reason the word deep means anything at all.
At a glance
- Per-layer operations
- Matrix multiply input by weights, add bias vector, apply nonlinearity
- Without nonlinearity
- Any number of layers collapses mathematically to a single linear transform
- Common nonlinearities
- ReLU (max of zero and input), sigmoid, tanh, each applied element-wise
- What depth buys
- Hierarchical feature composition, but only if each layer includes the nonlinear step
Think of a layer as a recipe with three steps. First, you take your input numbers and mix them together using a set of weights—like combining ingredients in fixed proportions. Second, you add a small adjustment to each result, like adding a pinch of salt. Third, you apply a rule that bends the result: maybe you set all negative numbers to zero, or you squash everything into a narrow range. Then you hand the output to the next layer, which does the same three things again.
The first two steps—mixing and adjusting—are both straight-line operations. If you do a straight-line operation and then another straight-line operation, the result is still just one straight-line operation. You could skip the middle layer entirely and get the same answer. The third step, the bending rule, is what stops that collapse. It puts a kink in the line. Once you have kinks, stacking layers starts to build something more complicated than a single layer could represent. Without that bending step, depth is an illusion.
A fully connected layer computes y = σ(Wx + b), where x is the input vector, W is the weight matrix, b is the bias vector, and σ is the nonlinearity applied element-wise. The matrix multiply Wx is a linear map; adding b makes it affine. If σ is the identity function, then stacking layers produces (W₃(W₂(W₁x + b₁) + b₂) + b₃), which simplifies to (W₃W₂W₁)x + (W₃W₂b₁ + W₃b₂ + b₃). This is still a single affine transformation. The composition of affine maps is affine, so depth without nonlinearity adds no representational power.
The nonlinearity σ breaks this. ReLU, defined as σ(z) = max(0, z), is continuous but not differentiable at zero and not linear anywhere. Sigmoid is σ(z) = 1/(1 + exp(−z)); tanh is σ(z) = (exp(z) − exp(−z))/(exp(z) + exp(−z)). All are applied pointwise, introducing no additional parameters. The choice affects gradient dynamics—ReLU mitigates vanishing gradients because its derivative is one for positive inputs—but the essential role is the same: nonlinearity allows each layer to compute functions that are not in the span of the previous layer's linear transformations.
In convolutional layers, W is structured as a convolution kernel, but the logic is identical: linear operation, bias, nonlinearity. In attention layers, the linear transforms are the query, key, and value projections; the nonlinearity may appear after the attention-weighted sum. Residual connections and normalisation layers complicate the data flow, but they do not replace the core structure. Every layer that learns still performs some variant of affine transform plus nonlinearity, and removing the nonlinearity still collapses the depth.
Look closer
The matrix multiply is a weighted sum for each output
If a layer has 512 inputs and 256 outputs, the weight matrix has 512 × 256 entries. Each output neuron is computed by taking all 512 inputs, multiplying each by its corresponding weight, and summing. The bias is a single number added to that sum. Repeat 256 times, once per output. The matrix notation bundles all those weighted sums into one operation, but the underlying arithmetic is addition and multiplication, done in parallel for every output position.
The nonlinearity is applied independently to each number
After the matrix multiply and bias addition produce a vector of numbers, the nonlinearity runs on each one separately. ReLU replaces negative values with zero and leaves positive values unchanged. Sigmoid squashes everything into the range zero to one. Tanh squashes to minus one to plus one. None of them look at neighbouring values or mix information across positions. They are pointwise functions, and that simplicity is part of why they work: they introduce nonlinearity without adding parameters or breaking the flow of gradient information during training.
Stacking linear layers without nonlinearity is pointless
Matrix multiplication is a linear operation. If you multiply by matrix A, then multiply the result by matrix B, the combined effect is identical to multiplying once by the product matrix BA. Adding bias vectors does not change this: the composition of two affine transforms is still affine. So a network with ten linear layers, no matter how wide, computes exactly the same set of functions as a network with one layer. Depth without nonlinearity is an illusion. The nonlinearity is what breaks the collapse and allows each layer to contribute something the previous layers could not represent on their own.
The story
A neural network layer takes a vector of numbers as input and produces a vector of numbers as output. The transformation happens in three steps, executed in order.
First, the input is multiplied by a matrix of weights. If the input has 512 dimensions and the layer is supposed to produce 256 outputs, the weight matrix has 512 rows and 256 columns. Each of the 256 outputs is computed as a weighted sum: take all 512 input values, multiply each by the weight in the corresponding row of that output's column, and add them up. This is matrix multiplication, and it is the most expensive part of the operation in terms of arithmetic. A single layer in a large model may involve billions of such multiplications.
Second, a bias vector is added. Each output position gets its own bias term, a single learned number that shifts the result of the weighted sum up or down. Bias allows the layer to produce non-zero outputs even when the input is all zeros, which turns out to matter for training.
Third, a nonlinearity is applied. This is a fixed mathematical function—not learned, not parameterised—that runs independently on each number in the output vector. The most common choice in modern networks is ReLU, which stands for rectified linear unit. ReLU looks at a number and returns the larger of that number and zero. Negative values become zero; positive values pass through unchanged. Other nonlinearities include the sigmoid function, which squashes any input into the range between zero and one, and tanh, which squashes into the range between minus one and one.
That is the entire layer. Matrix multiply, add bias, apply nonlinearity. The output becomes the input to the next layer, and the process repeats. A deep network is this same structure applied dozens or hundreds of times in sequence.
The nonlinearity is not decorative. Without it, the entire stack collapses. Matrix multiplication is a linear operation: if you apply one linear transform and then another, the result is equivalent to a single linear transform. You can compose a hundred matrices into one matrix and get exactly the same function. Adding bias does not save you—affine transforms compose into affine transforms. So a ten-layer network with no nonlinearities computes precisely the same set of functions as a one-layer network with the same input and output dimensions. The depth is doing nothing.
The nonlinearity breaks that collapse. It introduces a kink, a bend, a place where the function is no longer a straight line through high-dimensional space. Once you have that, stacking layers starts to matter. Each layer can now build on the representations from the previous one in ways that are not reducible to a single linear step. Early layers might detect edges or textures; middle layers might combine those into shapes; later layers might recognise objects. That hierarchical composition is only possible because the nonlinearity prevents each layer from being absorbed into its neighbours.
The choice of nonlinearity affects training speed, gradient flow, and which functions the network can approximate efficiently, but the fundamental role is the same: it is the thing that makes depth mean something.
Why it mattered then
The recognition that nonlinearity was essential came early in the history of neural networks, though the implications took time to settle. The perceptron, introduced in the late nineteen-fifties, was a single-layer linear classifier, and its limitations were well understood by the nineteen-sixties: it could not learn functions that were not linearly separable, such as XOR. Adding more perceptron layers without nonlinearity did not help, because the composition remained linear. Multilayer perceptrons with nonlinear activation functions were proposed as a solution, but training them was not practical until backpropagation became widely known in the nineteen-eighties. Even then, the nonlinearities in common use—sigmoid and tanh—had their own problems. They saturate: for large positive or negative inputs, their gradients become very small, which meant that during training, information about the error would fade as it propagated backward through many layers. This was called the vanishing gradient problem, and it limited how deep networks could be trained in practice. ReLU, introduced in its modern form in the early two-thousands and popularised around 2010, mitigated this. Its gradient is either zero or one, so it does not shrink the signal in the same way. That made deeper networks trainable, and deeper networks turned out to perform better on complex tasks, provided you had enough data and compute. The nonlinearity had always been necessary in principle; ReLU made it practical at scale.
Why it matters now
Every major neural network architecture in current use—transformers, convolutional networks, residual networks—relies on this same structure. The matrix multiply may be dressed up as a convolution or an attention operation, but the core remains: linear transform, bias, nonlinearity, repeat. The nonlinearity is still the only reason depth works. Understanding this matters for practical work. If you are debugging a network that is not training, one place to look is whether gradients are vanishing or exploding as they pass through the nonlinearities. If you are trying to understand what a network has learned, you need to remember that each layer's representation is shaped not just by the weights but by the kinks introduced at every nonlinear step. If you are designing a new architecture, you cannot skip the nonlinearity and expect stacking layers to help. It also clarifies what a neural network is not. It is not a black box of inscrutable complexity at the level of individual operations. Each layer does three things, and two of them are linear algebra you could do by hand if the matrices were small enough. The third is a function simple enough to write in one line of code. The complexity emerges from repetition, from scale, and from the interaction of millions of parameters adjusted during training—but the building block is this, and it has been this for decades.
The surprising detail
ReLU is not differentiable at zero. Its gradient is one for positive inputs and zero for negative inputs, but at exactly zero the derivative is undefined. In principle, this should be a problem for gradient-based optimisation, because backpropagation relies on computing derivatives everywhere. In practice, the probability of landing exactly on zero in floating-point arithmetic is negligible, and implementations simply assign the gradient at zero to be either zero or one, arbitrarily. It does not seem to matter which. The function works anyway, and has become the default nonlinearity in deep learning, despite this small mathematical irregularity that would have looked like a flaw on paper.
Remember this
Matrix multiply, add bias, apply nonlinearity. The last step is the only reason stacking layers achieves anything a single layer could not.
Test yourself
You build a three-layer network. Each layer multiplies its input by a weight matrix and adds a bias, but you forget to include any nonlinearity. You then train it on a classification task. What function has the trained network actually learned, and what is the simplest architecture that could have learned the same function?
The trained network has learned a single affine transformation: one matrix multiply plus one bias vector, applied directly from input to output. Because matrix multiplication and bias addition are both linear operations (or affine, if you count the bias), composing three such layers is mathematically identical to applying one layer with a weight matrix equal to the product of the three weight matrices, and a bias equal to the appropriately transformed sum of the biases. The simplest architecture that could learn the same function is a single-layer network with no hidden layers: input directly to output, one matrix and one bias. All the depth was wasted. This is why the nonlinearity is not optional.
Go deeper
- mlp.html · deeplearningbook.org
- Neural networks and deep learning · neuralnetworksanddeeplearning.com
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.