II · THE IDEA · ARTIFICIAL INTELLIGENCE
Fourier Features for Positional Encoding
▶ Listen · narrated
Neural networks are surprisingly slow to fit sharp edges and fine texture when the input is just a point in space. A simple trigonometric lift of that point is often enough to change the outcome.
At a glance
- Problem
- MLPs on raw low-dimensional coordinates favour low frequencies
- Mapping
- Fixed sinusoids take coordinates into a higher-dimensional feature space
- Effect
- High-frequency target functions become much easier to fit
- Role
- A positional encoding placed before the network, not inside it
- Scope
- Most relevant when inputs are coordinates in a low-dimensional domain
Think of asking someone to draw a complicated wavy line using only a thick marker and long, sweeping strokes. Broad shapes come out fine; tight wiggles do not. That is roughly what a multilayer perceptron faces when it is given a plain position — a single number, or a pair of coordinates — and asked to match a target full of rapid change.
Fourier features change the brief. Before the network sees the position, you turn that position into a long list of ready-made waves: slow ones, medium ones, fast ones, each recorded as a sine and a cosine. The network’s job is no longer to invent rapid oscillation from scratch. It only has to mix waves it has already been handed.
The mixing is learned. The waves themselves are not. You choose them once, freeze them, and train as usual. In low-dimensional problems — colours at pixel locations, values along a line, densities in a small volume — that single preprocessing step is often enough for fine structure to appear in the fit instead of remaining stuck as blur.
Standard MLPs exhibit a spectral bias on low-dimensional coordinate inputs: as a function of training, lower-frequency components of the target are fitted preferentially. Fourier feature mapping composes the input coordinates x with a fixed feature map γ(x) whose components are sin(2π bᵀ x) and cos(2π bᵀ x) for a chosen set of frequency vectors b. The MLP then approximates the target as a function of γ(x) rather than of x.
Because γ is not trained, the method is a positional encoding rather than a learned embedding layer. The distribution used to select b (for example isotropic Gaussian sampling with a scale hyperparameter) sets the effective bandwidth available to the model. Under-scaled frequencies preserve smoothness; over-scaled frequencies admit finer variation and can also admit noise. The same MLP width and depth therefore implement a different function class solely through the spectrum of γ.
Mechanically this helps because high-frequency structure need not be constructed by deep composition of weak nonlinearities applied to raw coordinates; it is already linearly present in the features. The approach is most clearly motivated when the native input dimension is small and the desired signal spectrum is broad. It is not automatically the right tool when inputs are already high-dimensional embeddings, nor does it replace the need for a loss and sampling scheme that expose high-frequency error during training.
Limitations follow from the fixed band-limit and from the usual approximation caveats: stationary frequency sampling does not adapt to local bandwidth; very high frequency settings can overfit; and comparisons across encodings must hold architecture and optimisation fixed if the effect of γ is to be isolated.
Look closer
The map is not learned
The Fourier feature layer is a fixed transform. Each input coordinate vector is projected against a set of frequency vectors, then passed through sine and cosine. Those frequencies are chosen once — often by sampling from a simple distribution — and then held constant while the multilayer perceptron trains on the resulting features. Nothing in the sinusoidal stage is updated by gradient descent.
Dimension goes up on purpose
A point that began as two or three numbers leaves the mapping as a long vector of paired sines and cosines. The network never sees the original coordinates directly. It sees only this lifted representation, so every weight it learns is a weight on a sinusoidal feature rather than on raw position.
Frequency content becomes a dial
Which frequencies appear in the mapping sets an effective band-limit on what the model can express with ease. A narrow band of low frequencies keeps the fit smooth. A broader band, or a different sampling of frequency vectors, lets finer variation through. The architecture of the MLP can stay the same; the encoding changes the function class it reaches first.
The story
Feed a multilayer perceptron a low-dimensional coordinate — a point on a line, a pixel location, a sample in a volume — and ask it to match a target that varies sharply across that domain. Training often progresses unevenly. Broad, smooth structure appears early. Fine detail arrives late, if it arrives at all. This behaviour is sometimes described as a spectral bias: standard networks, given raw coordinates, preferentially represent lower-frequency functions of their input.
Fourier feature mapping attacks that bias before the first learned layer. The coordinate is not passed in as-is. It is multiplied by a bank of frequency vectors and replaced by the sines and cosines of those products. The multilayer perceptron then receives a high-dimensional vector whose axes are pure sinusoids of the original position. Learning still happens in the usual way, but the basis on which it happens has changed.
The mapping itself is simple and fixed. No parameters inside the sine–cosine stage are trained. The only design choices are how many frequency vectors to use and how those vectors are chosen. A common practical pattern is to draw them from a Gaussian or to place them on a regular grid in frequency space; either way, the choice is made up front and then frozen. After that, ordinary gradient descent on the MLP is free to combine those sinusoids into whatever target the loss demands.
Why this helps is less mysterious than it first sounds. A network that must synthesise a high-frequency oscillation from raw coordinates has to arrange its nonlinearities carefully across depth. A network that is handed that oscillation as an input feature can approximate it with a shallow combination of weights. In low-dimensional domains — image coordinates, three-dimensional positions, parameterised curves — the cost of the lift is small and the change in what can be fitted is large.
The same idea sits behind several positional encodings used when a model must condition on location. The Fourier feature formulation makes the frequency-domain intent explicit: you are choosing, in advance, which wavelengths of variation the network should be able to represent without a struggle. Under-shoot the band and the fit stays too smooth. Over-shoot it and the model may fit noise or oscillate between samples. The encoding is therefore not a neutral preprocessing step; it is part of the inductive bias.
None of this removes the need for data or for a loss that actually demands high-frequency structure. It only changes how readily a standard multilayer perceptron can answer that demand when its inputs live in a low-dimensional coordinate space. In higher-dimensional or already-rich feature regimes the same trick is less often the bottleneck, which is why the method is discussed most sharply for coordinate-based networks and related positional tasks.
Why it mattered then
Coordinate-based multilayer perceptrons had become a practical tool for representing images, scenes and other signals directly as functions of position. Their failure modes were equally practical: reconstructions that looked right from a distance and collapsed into blur or soft edges up close. The Fourier feature result gave a clear diagnosis — spectral bias on low-dimensional inputs — and a cheap remedy that did not require redesigning the network. It turned an opaque training frustration into a controllable choice about frequency content, which is why it spread quickly through work on neural representations of spatial signals.
Why it matters now
Any pipeline that still maps coordinates through an MLP — neural fields, some forms of positional conditioning, grid-free signal fits — meets the same bias. Fourier features remain a small, architecture-agnostic lever: change the encoding, keep the network. They also sharpen a broader design habit. When a model struggles with fine structure, it is worth asking whether the difficulty is capacity in the weights or the spectrum of the features those weights see. That question applies well beyond the original setting, whenever low-dimensional positions are asked to carry high-frequency meaning.
The surprising detail
The powerful step is not a new layer type or a clever training schedule. It is a frozen trigonometric lift applied before learning begins. The network architecture can be ordinary; the frequencies in the mapping do much of the work that people might otherwise expect from depth or width. That separation — choose the spectrum by hand, learn only the combination — is what makes the method easy to ablate and easy to get wrong. A single scale parameter in how the frequency vectors are sampled can decide whether the fit stays smooth or suddenly resolves detail.
What is disputed
The strength of spectral bias, and how large a gain Fourier features bring, depends on architecture, width, depth and the spectrum of the target. The method is well supported for low-dimensional coordinate inputs; it is not a universal fix for every failure to capture fine detail. Frequency sampling schemes also differ across implementations, and evidence for one sampling distribution over another is often empirical rather than settled.
Remember this
Fourier features do not teach a network new mathematics. They hand it sinusoids of the input so high-frequency targets stop being an uphill fight against spectral bias.
Test yourself
You train two identical multilayer perceptrons to fit a sharply varying function of 2D coordinates. One receives the raw coordinates; the other receives a Fourier feature mapping of those coordinates. The second fits fine detail far sooner. What, precisely, has changed about the learning problem — and what has not?
What has changed is the representation of the input: the second network never sees position directly, only fixed sines and cosines of linear projections of position, so high-frequency variation is already present as features it can weight. What has not changed is the MLP architecture, the loss, and the fact that the sinusoidal stage itself is not learned. Capacity in the usual sense is similar; the inductive bias over frequency content is not.
Go deeper
- [2006.10739] Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains · arxiv.org
- [2108.12490] VisGraphNet: a complex network interpretation of convolutional neural features · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.