II · THE IDEA · ARTIFICIAL INTELLIGENCE
NeRF: Neural Radiance Fields
▶ Listen · narrated
Given a handful of photographs of a scene, the method recovers a continuous volumetric field from which any new viewpoint can be synthesised — without storing an explicit mesh or voxel grid.
At a glance
- What it is
- A continuous 5D function from position and view direction to density and colour
- Representation
- Weights of a multilayer perceptron, not a mesh or voxel grid
- Training input
- A set of 2D images with known camera poses
- Rendering
- Differentiable classical volume rendering along camera rays
- Key trick
- Positional encoding so the network can fit high-frequency detail
Think of the scene as a block of coloured mist. At every tiny location inside the block, a small calculator can tell you two things: how thick the mist is there, and what colour it glows, including a tint that may change if you look from another angle.
To make a picture, you send a straight line out through each pixel of an imaginary camera, ask the calculator for thickness and colour at many stops along that line, and blend the answers so thick mist hides whatever sits behind it. That blend is an old technique from computer graphics; what is new is that the calculator is a neural network whose internal numbers are adjusted until every training photograph is reproduced.
Nothing like a digital clay model is ever built. The whole scene lives only as those adjustable numbers. When you want a view from a new spot, you aim new lines and ask the same calculator again. A fixed preprocessing step turns plain coordinates into a richer pattern of waves so the network can capture fine edges instead of only soft blobs.
NeRF represents a static scene as an MLP approximating a continuous 5D radiance field F_θ: (x, d) → (c, σ), where x ∈ ℝ³ is position, d is a unit viewing direction, σ ≥ 0 is volume density (view-independent), and c ∈ [0,1]³ is emitted RGB (view-dependent). Camera rays r(t) = o + t d are marched between near and far bounds. Densities and colours at sample depths are composited with the classical volume-rendering quadrature:
Ĉ(r) = Σ_i T_i (1 − exp(−σ_i δ_i)) c_i, T_i = exp(−Σ_{j<i} σ_j δ_j),
which is fully differentiable in θ. Training minimises squared error between Ĉ(r) and ground-truth pixel colours over rays from posed input images.
Raw coordinates are lifted by a fixed positional encoding γ(·) of sines and cosines at dyadic frequencies before entering the MLP; without γ, spectral bias yields low-frequency fits and blurred geometry. Optimisation uses hierarchical sampling: a coarse network proposes a piecewise-constant PDF along each ray; a fine network allocates additional samples to high-density regions.
The stored artefact is θ, not a mesh or regular grid. Novel-view synthesis reuses the same integral from new poses. Limitations of the original method include slow per-scene optimisation, the static-scene assumption, dependence on accurate extrinsics, and expensive ray sampling at inference. HyperNeRF extends the ambient domain so topologically varying deformations can be read as slices of a higher-dimensional continuous field, preserving the radiance-field rendering loop.
Look closer
Five dimensions, not three
At each 3D point the field returns a volume density that does not depend on where the viewer stands. Colour, however, is allowed to change with viewing direction. That split is deliberate: density describes where matter is; view-dependent colour is what lets specular highlights and glossy surfaces move correctly as the camera circles the scene. The network is therefore a map from (x, y, z, viewing direction) to (density, RGB), and both outputs are continuous.
Rays, samples, and a classical integral
To make an image, the method does not paint surfaces. It casts a ray through each pixel, samples the continuous field at many points along that ray, and composites the returned colours using the same transmittance integral used in classical volume rendering. Because every step is differentiable, the mismatch between the rendered pixel and the real photograph becomes a gradient that can update the network weights. The geometry is never written down as polygons; it is whatever density arrangement makes the photographs reappear.
Why raw coordinates are not enough
A plain multilayer perceptron fed with raw spatial coordinates tends to fit low-frequency patterns and smear fine detail. NeRF therefore lifts each coordinate through a fixed positional encoding — a bank of sines and cosines at increasing frequencies — before the network sees it. High-frequency structure in the scene becomes easier to represent as variation in those encoded features. Without that encoding, the same architecture produces noticeably blurrier geometry and texture.
The story
NeRF begins from a deceptively plain question: if you already have several photographs of a static scene, and you know where the camera was for each one, can you invent the photograph that would have been taken from a place you never stood? Earlier answers often built an explicit 3D object — a mesh, a voxel grid, a point cloud — and then textured it. NeRF refuses that intermediate object. It treats the scene as a continuous volumetric function and stores that function in the weights of a neural network.
The function is five-dimensional. Three coordinates locate a point in space; two more describe the direction from which that point is being observed. From those five numbers the network emits two things: a volume density, and an emitted colour. Density is view-independent. Colour may change with direction, which is how the representation captures specular reflection without a separate material model. Query the network densely enough and you have, in principle, every optical property the method needs to redraw the scene.
Drawing a pixel is an act of integration, not of ray–triangle intersection. A camera ray is stepped through the volume; at each sample point the network is asked for density and colour; those samples are composited with the classical volume-rendering integral so that opaque regions occlude what lies behind them. The entire pipeline is differentiable. When the synthesised image disagrees with a real training photograph, gradients flow back through the integral and into the network weights. Over many such comparisons the field settles into an arrangement of density and colour that explains every training view at once.
Two practical choices make the fit work. First, positional encoding: raw (x, y, z) coordinates are mapped through a fixed set of sinusoids at multiple frequencies before they enter the multilayer perceptron. Without that lift, the network prefers smooth, low-frequency solutions and cannot recover sharp edges or fine texture. Second, hierarchical sampling: a coarse network proposes where along each ray the interesting density is likely to lie, and a fine network then concentrates its samples there. Compute is spent near surfaces rather than in empty space.
What you store at the end is not a mesh. It is a few megabytes of network weights. Novel views are produced by the same ray-marching procedure used in training, aimed from any camera pose you choose. The method assumes a static scene and known camera parameters; when those assumptions hold, the continuous field often reproduces fine geometry — thin structures, intricate foliage, glossy materials — that discrete grids at comparable memory budgets tend to lose.
HyperNeRF later pushed the same idea into higher-dimensional domains so that topologically varying phenomena — a mouth opening, an object deforming — could be represented as slices through a continuous field rather than as a single static volume. The core bargain remains the one NeRF struck: trade an explicit 3D data structure for a continuous function that is only ever observed through differentiable rendering.
Why it mattered then
At the time of the original paper, neural view synthesis was active but still often tethered to discrete scene representations or to methods that struggled with high-frequency detail. NeRF showed that a plain multilayer perceptron, given positional encoding and a classical volume-rendering integral, could serve as the scene itself. The result mattered less as a claim about networks being magical than as a demonstration that continuous, differentiable volumetric rendering could recover photorealistic novel views from ordinary posed images — and that the geometry need never be extracted as an intermediate mesh. It reframed the output of multi-view reconstruction as a queryable field rather than as a surface to be meshed and textured.
Why it matters now
The continuous-field idea has become a default vocabulary across neural rendering, 3D generation, and robotics perception. Later systems replaced or accelerated the original multilayer perceptron, but they still lean on the same bargain: represent space as something you can sample, render by integrating along rays, and train by comparing pixels. Understanding NeRF is less about one architecture than about why density-plus-view-dependent colour, positional encoding, and differentiable volume rendering form a workable loop. When modern pipelines speak of radiance fields, Gaussian approximations, or higher-dimensional deformations in the spirit of HyperNeRF, they are still negotiating the trade-offs this paper made explicit.
The surprising detail
The scene is never assembled as geometry. There is no marching-cubes step required for rendering, no voxel grid that must be kept in memory at the resolution of the finest detail. Sharp structure emerges only because the positional encoding lets the network oscillate quickly in space, and because the rendering loss punishes any density arrangement that fails to match the input photographs. The representation is almost embarrassingly indirect: photographs supervise a function, and the function, queried along rays, becomes the photographs again — plus views that were never taken.
What is disputed
NeRF assumes known camera poses and a static scene; quality depends heavily on view coverage and calibration. HyperNeRF addresses some topological variation but still relies on the same continuous-field premise. Claims about which scenes reconstruct cleanly are empirical and method-dependent, not guaranteed by the formulation alone.
Remember this
NeRF stores a scene as a continuous function from position and viewing direction to density and colour, trained so that differentiable volume rendering reproduces the input photographs.
Test yourself
A colleague suggests skipping positional encoding and feeding raw xyz coordinates straight into the multilayer perceptron, arguing that a deep enough network can learn any mapping. What specific failure should you expect in the recovered scene, and why does the rendering loss alone not prevent it?
You should expect overly smooth geometry and blurred texture: multilayer perceptrons have a well-known bias toward low-frequency functions when given raw coordinates. The rendering loss still pushes rendered pixels toward the training images, but without a high-frequency basis the network lacks an easy way to represent sharp variation in space, so it settles for a blurry compromise that reduces average error without recovering fine structure. Positional encoding supplies those frequencies up front.
Go deeper
- [2003.08934] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis · arxiv.org
- [2106.13228] HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.