II · THE IDEA · ARTIFICIAL INTELLIGENCE
Activations: ReLU, GELU, SwiGLU
▶ Listen · narrated
The choice between ReLU, GELU and SwiGLU determines not just speed but what a network can represent. Each trades off smoothness, sparsity and the risk that a neuron stops learning altogether.
At a glance
- ReLU
- max(0, x) — simple, fast, sparse, but neurons can die permanently
- GELU
- Smooth approximation weighted by probability x is above zero
- SwiGLU
- Gated variant using two parallel projections, one modulating the other
- Where they appear
- Between the two linear layers in each transformer feedforward block
Think of a dimmer switch. A light switch is either on or off, like a step function. A dimmer lets you turn the light up gradually, creating a smooth range of brightness. An activation function is the dimmer between layers of a neural network. ReLU is almost like a switch—below zero it is off, above zero it passes the signal through unchanged—but that sharp corner at zero is enough to let the network learn curves and boundaries instead of just straight lines. GELU is smoother, like a dimmer that starts turning the light on a little before you reach the zero point, so there is never a completely dead zone. SwiGLU adds a second control: one dimmer decides how bright the light should be, and another decides whether to let that brightness through at all. The network learns to set both, giving it more flexibility in what it pays attention to. The key idea is that without some kind of nonlinearity—some way to bend or gate the signal—stacking more layers does nothing. You can chain ten dimmer switches in a row, but if they all scale the brightness by the same rule, you only get one dimmer's worth of control.
An activation function is applied element-wise to the output of a linear layer, introducing nonlinearity into the network. Without it, composing multiple linear transformations yields another linear transformation, limiting the function class the network can approximate. ReLU, defined as max(0, x), outputs zero for negative inputs and the identity for positive inputs. Its derivative is zero for x < 0 and one for x > 0, with the derivative at zero typically set to zero or one by convention. This creates sparsity—many activations are exactly zero—which accelerates computation and may aid interpretability, but it also means neurons can die if their weights shift such that inputs are always negative, because the gradient becomes permanently zero.
GELU, defined as x times the cumulative distribution function of the standard normal at x, is smooth everywhere. It can be understood as weighting the input by the probability that a Gaussian random variable is less than x. In practice it is computed via approximation, either using the error function or a tanh-based polynomial. GELU asymptotically approaches zero for large negative x and the identity for large positive x, but unlike ReLU it has non-zero gradient throughout, which helps gradient flow during backpropagation. It became popular in transformer models starting with BERT.
SwiGLU restructures the feedforward block. Instead of a single linear projection from dimension d to an intermediate dimension (commonly 4d), followed by an activation and a projection back to d, SwiGLU uses two parallel projections to the intermediate dimension. One branch is passed through Swish, defined as x times sigmoid(x), and the result is multiplied element-wise with the other branch. This gating allows the network to learn which components of the intermediate representation to pass through. The cost is higher: SwiGLU requires two weight matrices for the expansion instead of one, increasing both parameters and computation. Shazeer's experiments in the GLU Variants paper showed that despite this cost, SwiGLU improved language model quality consistently across scales, and it has since been adopted in models including PaLM and LLaMA. The Swish function itself is smooth and non-monotonic, which distinguishes it from ReLU and GELU, though the practical impact of non-monotonicity is not fully understood.
Look closer
ReLU kills exactly half the number line
The Rectified Linear Unit outputs zero for any negative input and the input itself otherwise. This creates exact sparsity: roughly half the neurons in a layer emit zero for any given input, doing no work in subsequent layers. That sparsity makes ReLU fast and makes the network easier to interpret, but it introduces a failure mode. If a neuron's weights drift such that its input is always negative, its gradient becomes permanently zero and it stops learning. This is called a dead neuron, and it is an irreversible event during training.
GELU is smooth and stochastic in origin
The Gaussian Error Linear Unit multiplies the input by the probability that it exceeds a sample drawn from a standard normal distribution. In practice this is approximated as x times the cumulative distribution function of the Gaussian at x. The result is a smooth curve that passes through the origin, asymptotically approaching the identity function for large positive x and zero for large negative x. Unlike ReLU there is no sharp corner, so gradients flow even for small negative values. GELU became popular in BERT and subsequent transformer models.
SwiGLU splits the feedforward layer in two
Swish-Gated Linear Unit, introduced by Shazeer, applies a gating mechanism inside the feedforward block. Instead of one linear projection followed by an activation, SwiGLU uses two parallel projections of the hidden dimension. One branch is passed through the Swish activation, then multiplied element-wise with the other branch. This costs more parameters and computation than ReLU or GELU applied to a single projection, but Shazeer's experiments showed it improved performance enough to justify the cost in large language models.
The story
A neural network without activation functions is just a stack of matrix multiplications, and no matter how many you chain together, the result is still a single linear transformation. You can make the stack arbitrarily deep and it will still only be able to draw straight lines through the data. The activation function, applied after each linear layer, breaks this constraint. It introduces nonlinearity, allowing the network to approximate any continuous function given enough width and depth.
For years the default choice was the sigmoid, a smooth S-shaped curve that squashes any input into the range zero to one. It had an elegant biological motivation—neurons either fire or they do not—and a clean derivative. But sigmoid activations caused training problems in deep networks. Gradients shrank exponentially as they propagated backward through many layers, a problem called vanishing gradients, and most of the neuron's output range was spent in flat regions where the gradient was nearly zero.
ReLU, defined simply as the maximum of zero and the input, solved this. Proposed in the context of deep learning in the early 2010s, it had been used in neuroscience models decades earlier but became standard in computer vision after strong results in convolutional networks. For positive inputs the gradient is exactly one, so it does not vanish. For negative inputs the gradient is zero, which creates sparsity but also the dead neuron problem. In practice ReLU worked well enough that it became the default, and variations like Leaky ReLU—which allows a small non-zero gradient for negative inputs—were proposed to mitigate the failure mode.
GELU, introduced by Hendrycks and Gimpel in 2016, brought back smoothness without the sigmoid's training problems. The function has a probabilistic interpretation: it weights the input by the likelihood that it is greater than a random sample from a Gaussian. The curve is smooth everywhere, so gradients always flow, but it still suppresses large negative values nearly to zero. GELU appeared in BERT and became common in transformers, where its smooth gradients seemed to help with the long training runs required for language models.
SwiGLU, from Shazeer's 2020 paper on GLU variants, takes a different approach. It incorporates gating directly into the activation. The feedforward block, which normally has one linear layer expanding the dimension and another contracting it with an activation in between, is restructured. SwiGLU uses two parallel expansions. One is passed through the Swish activation—a smooth function similar to GELU—and the result is multiplied element-wise with the other. This gating mechanism lets the network learn which parts of the representation to pass through and which to suppress, adding expressiveness at the cost of more parameters. Shazeer's experiments showed consistent improvements in language model perplexity, and SwiGLU has been adopted in several large models since.
Why it mattered then
ReLU's adoption marked a turning point in training deep networks. Before it, networks of more than a few layers were difficult to train reliably because gradients either vanished or exploded. ReLU's simple derivative—one for positive inputs, zero otherwise—meant gradients could propagate through dozens of layers without shrinking to nothing. This enabled the convolutional networks that won ImageNet in 2012 and the deeper architectures that followed. The dead neuron problem was known, but in practice enough neurons survived that the networks still learned. GELU arrived when transformers were becoming the dominant architecture for language, and its smooth gradients suited the long training runs and careful optimisation required for models like BERT. The function's stochastic motivation also fit the broader trend toward viewing neural network components through a probabilistic lens, though the approximation used in practice is deterministic.
Why it matters now
SwiGLU is now the activation of choice in many large language models, including some open-weight models that publish architectural details. The gating mechanism adds flexibility that seems to matter at scale, even though the cost is higher than ReLU or GELU. This reflects a broader shift: as models have grown and as the budget for training them has increased, the field has been willing to spend more computation per parameter if it improves the final result. The choice of activation is no longer just about avoiding training failures—ReLU solved that—but about squeezing incremental quality from a fixed parameter budget. Dead neurons are still a concern during training, but modern optimisers and initialisation schemes have reduced their frequency. The more common question now is whether a given activation's inductive bias—its tendency to create sparsity, or to gate information, or to smooth gradients—aligns with the task.
The surprising detail
The name ReLU is recent, but the function itself was used in neuroscience models in the 1960s and 1970s under the name ramp function. It took decades for the machine learning community to rediscover it and recognise that its simplicity was an advantage, not a limitation. The dead neuron problem, meanwhile, has generated dozens of proposed fixes—Leaky ReLU, Parametric ReLU, ELU, and others—but in practice the original ReLU remained dominant until transformers shifted the field toward GELU and gated variants. The persistence of a function with a known failure mode, simply because it worked well enough, is a reminder that theoretical elegance and empirical performance do not always align.
Remember this
Without a nonlinear activation between layers, a deep network collapses into a single linear transformation. The activation's shape determines what the network can learn and how easily.
Test yourself
A network uses ReLU activations and you notice during training that twenty percent of neurons in a particular layer have stopped producing any non-zero output. Explain why setting the learning rate higher will not fix this, and name one architectural change that might prevent it in future training runs.
A dead ReLU neuron has weights such that its input is always negative, so it always outputs zero. The gradient of ReLU for negative inputs is also zero, which means no error signal propagates back to update those weights. Increasing the learning rate scales the gradient, but zero times any number is still zero—the neuron remains stuck. One prevention is to use Leaky ReLU, which has a small non-zero gradient for negative inputs, allowing the weights to recover if they drift into the dead region. Another is to use GELU or a gated activation like SwiGLU, neither of which has a region of exactly zero gradient. Careful initialisation and a lower learning rate can also reduce the chance that neurons die in the first place, though they do not guarantee recovery once it has happened.
Go deeper
- GLU Variants Improve Transformer · arXiv · Noam Shazeer et al. · 2020-02-12
- Gaussian Error Linear Units (GELUs) · arXiv · Dan Hendrycks et al. · 2016-06-27
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.