II · THE IDEA · ARTIFICIAL INTELLIGENCE
Neural Radiance Fields for 3D Reconstruction
▶ Listen · narrated
From a handful of ordinary photographs and their camera poses, a multilayer perceptron can be optimised until it renders the scene from angles nobody ever photographed.
At a glance
- Representation
- MLP mapping 5D coordinates to density and view-dependent radiance
- Inputs
- 3D position plus 2D viewing direction
- Outputs
- Volume density σ and emitted RGB colour
- Supervision
- 2D images only; no explicit 3D geometry labels
- Rendering
- Classical volume rendering along camera rays, made differentiable
Think of the scene as a fog that is thick in some places and thin in others, and that glows different colours depending on where you stand. Instead of listing every droplet, you keep a small calculator: you point at a spot and a direction, and it tells you how thick the fog is there and what colour it glows toward you.
To make a picture, you send a straight line out from the camera through each pixel. Along that line you ask the calculator many times, then blend the answers the way photographers already blend layers of semi-transparent mist—thick glowing patches hide what is behind them. You adjust the calculator until the blended pictures look like your real photographs. After that, you can point the camera anywhere and ask for a new picture. You never built a doll’s-house model of the room; you only taught the calculator to answer density and colour.
NeRF fits a scene-specific MLP fθ: (x, d) → (σ, c) where x ∈ ℝ³ is position, d is a unit viewing direction, σ ≥ 0 is volume density, and c ∈ [0,1]³ is view-dependent emitted radiance. Direction is usually encoded as a unit vector; both x and d are lifted with a fixed sinusoidal positional encoding γ(·) so that the network can fit high-frequency detail.
Pixel colour is the quadrature of the volume-rendering integral C(r) = ∫ T(t) σ(r(t)) c(r(t), d) dt with transmittance T(t) = exp(−∫ σ ds). In practice a stratified hierarchical sampler draws a coarse set of t-values, evaluates a coarse MLP, then draws additional fine samples from the normalised weights of that coarse pass and evaluates a fine MLP. The L2 (or similar) error between C(r) and the observed pixel is back-propagated through the quadrature into θ. No depth, mesh, or multi-view stereo labels are required—only posed RGB images.
Limitations of the original setup include per-scene optimisation (no cross-scene weight sharing), sensitivity to pose noise, expensive stochastic sampling at training and inference time, and the fact that the learned field is not an explicit surface. Sample allocation remains a first-order concern: later work on learning to sample shows that the coarse-to-fine heuristic can still waste capacity and blur thin structures when the weight distribution is poorly estimated.
Look closer
Five dimensions, not three
The network is queried with a location (x, y, z) and a viewing direction. Density is a property of position alone; colour is allowed to change with direction. That split is what lets the field reproduce specular highlights and other view-dependent appearance without storing a mesh or a texture atlas.
Rays, not surfaces
To colour a pixel, a ray is cast through the camera, points are sampled along it, and the network returns density and radiance at each sample. Those values are composited with the standard volume-rendering integral. The integral is differentiable, so the only training signal required is the difference between the rendered pixel and the photograph.
Where the samples fall matters
A coarse network first estimates where density lies along each ray; a fine network is then evaluated at additional samples concentrated in those regions. Most of the capacity is spent near visible surfaces rather than in empty space. Later analysis shows that sample placement itself is a central design choice, because poor placement wastes evaluations and softens thin structure.
The story
A neural radiance field treats a scene as a continuous function. Give the function a point in space and a direction of view; it returns how densely that point occupies space and what colour of light leaves it toward the viewer. The function is implemented as a multilayer perceptron. Nothing in the representation is a triangle, a voxel grid, or a depth map. The geometry and appearance are whatever the weights have encoded.
Training uses only ordinary images and known camera poses. For each pixel in each training view, a ray is marched into the volume. At a set of sample positions along that ray the network is evaluated, producing densities and view-dependent colours. Classical volume rendering composites those samples into a single RGB value. Because every step is differentiable, gradient descent on a photometric loss adjusts the network weights until rendered views match the input photographs. No three-dimensional ground truth is supplied.
Two design choices make the optimisation practical. First, positional encoding lifts the raw coordinates into a higher-frequency Fourier basis before they enter the MLP, so the network can fit sharp edges and fine texture rather than only smooth, low-frequency variation. Second, hierarchical sampling runs a coarse pass to locate occupied regions, then allocates a fine pass’s samples where they contribute to the image. Empty space is skipped; surfaces receive denser queries.
At inference the same rendering procedure produces images from camera poses that never appeared in training. The field is continuous, so the viewpoint can move freely. What the method does not produce directly is an editable mesh or a semantically labelled scene graph; it produces a renderable volumetric function whose weights happen to store the scene. Photorealism here means that novel views match the lighting and fine detail of the captured photographs closely enough that the gaps between training angles are hard to see, not that the representation is a general graphics asset.
The original formulation optimises one network per scene. Generalisation across scenes, speed of training, and explicit geometry extraction are left to later work. The core claim is narrower and still striking: sparse calibrated images plus a differentiable volume renderer are enough to recover a continuous 5D field that synthesises new views.
Why it mattered then
Before this approach, high-quality novel-view synthesis from sparse photographs typically relied on explicit multi-view stereo geometry, mesh texturing, or discrete volumetric grids. Those pipelines either demanded dense capture and careful reconstruction or paid a resolution cost for storing empty space. Representing the scene as weights of a continuous function removed the need to choose a voxel resolution up front and removed the need for direct 3D supervision. Differentiable volume rendering turned a classical integral into a training objective, so photometric error on 2D images could shape both geometry-like density and view-dependent appearance in one optimisation. That combination—continuous 5D field, positional encoding, hierarchical sampling—made photorealistic view synthesis from ordinary posed images a practical research baseline rather than a heavyweight graphics production process.
Why it matters now
The same idea—encode a signal as a neural field and supervise it through a differentiable renderer—now appears in variants for dynamic scenes, relighting, and compression, and it sits behind many modern neural rendering stacks. Even when later systems replace the pure MLP with grids, hash encodings, or Gaussian primitives, they still owe the framing: scene as continuous field, images as integrals through that field, optimise until the integrals match. For anyone working on multimodality, NeRF is a clean example of lifting 2D observations into a 3D-consistent representation without ever labelling 3D structure by hand. It also remains a reminder that the sampling pattern along each ray is part of the model; capacity spent in empty space is capacity not spent on surfaces.
The surprising detail
Density is deliberately independent of viewing direction, while colour is not. The network is architected so that the same point cannot be transparent from one angle and opaque from another, yet it can look mirror-bright from one direction and dull from another. That asymmetry is a small inductive bias with large visual consequences: it stabilises geometry while still allowing specular effects that a view-independent colour field would smear away.
What is disputed
Reported visual quality depends on pose accuracy, sampling schedule, and the frequency design of the positional encoding. Comparisons across papers are not always matched on capture density or compute, so claims of strict superiority over a given classical baseline should be read in light of those experimental choices.
Remember this
A NeRF stores a scene as an MLP that maps position and view direction to density and colour, trained only by matching volume-rendered rays to photographs.
Test yourself
A colleague proposes storing colour as a function of position alone, dropping the viewing direction from the network input to save computation. Which class of appearance effects would you expect to degrade first, and why does the original design keep direction as an input?
View-dependent effects such as specular highlights and glossy reflections would degrade first, because they change with the angle between surface, light, and camera. The original design feeds direction into the colour head precisely so the same spatial point can emit different radiance toward different rays, while density stays view-independent and therefore geometrically consistent.
Go deeper
- [2003.08934] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis · arxiv.org
- [2106.05264] NeRF in detail: Learning to sample for view synthesis · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.