II · THE IDEA · ARTIFICIAL INTELLIGENCE
Volumetric Rendering with Neural Graphics Primitives
▶ Listen · narrated
Neural radiance fields made novel views from a network alone, but training and rendering stayed slow. Pairing a multiresolution hash grid with a tiny MLP changed the practical calculus.
At a glance
- What it is
- A scene as neural feature grids decoded by a small MLP
- Key encoding
- Multiresolution hash tables mapping 3D coordinates to features
- Practical gain
- Real-time rendering quality earlier methods needed hours per frame for
- MLP role
- A compact network turns concatenated features into density and colour
Think of the scene as stored in a set of labelled drawers rather than in one enormous recipe book. When you ask “what is here?” at a point in space, you open one drawer for the rough neighbourhood, one for medium detail, and one for fine detail. Each drawer hands you a short code. You tie those codes together and give them to a very small calculator that answers with how opaque and what colour that point should be. March many such points along a camera ray, blend them, and you get a pixel.
Earlier neural methods tried to make the calculator alone remember the whole scene from raw coordinates. That worked, but every question was expensive. Instant-NGP moves almost all of the remembering into the drawers (hash tables at several scales) and leaves the calculator tiny. Opening drawers is fast; running a small network is fast; so the image can update at interactive rates while still being learned from data.
Instant-NGP represents a scene as a multiresolution hash encoding feeding a compact MLP for volumetric rendering. For a continuous coordinate x, each resolution level ℓ scales x into that level’s grid, hashes the resulting integer cell indices into a fixed-size table, and retrieves trainable feature vectors (with interpolation across cell corners as implemented). Features from all levels are concatenated—often with a view-direction encoding for colour—and passed through a small MLP that predicts volume density and view-dependent radiance. Pixels are formed by standard quadrature along camera rays.
Because table size is capped per level, memory does not scale as O(R³) with finest resolution R the way a dense grid does. Hash collisions map distinct spatial locations to the same parameters; the MLP is trained to decode the resulting features anyway, and independence across levels limits simultaneous collision at every scale. The bulk of trainable capacity lives in the tables, so MLP depth and width stay small enough for dense sample evaluation. Training jointly optimises table entries and MLP weights. Limitations include collision-induced interference in highly detailed or periodic regions, scene extent versus table budget trade-offs, and the usual dependence of image quality on sample count, ray scheduling, and occupancy structures used to skip empty space.
Look closer
Coordinates never enter the MLP raw
A query point is looked up at several spatial resolutions at once. At each level the coordinates are hashed into a fixed-size table of trainable feature vectors. The vectors from every level are concatenated and only then passed to the network. Coarse levels carry large-scale structure; fine levels carry detail. The MLP never has to invent spatial frequency from a bare xyz triple.
Hash collisions are tolerated, not avoided
Different spatial locations can land in the same table entry. Instant-NGP does not resolve those collisions with perfect hashing. Instead the small MLP learns to disentangle the mixed signal, and the multiresolution arrangement means a collision at one level is rarely a collision at all levels. Storage stays bounded while effective capacity remains high.
The network is deliberately tiny
Most of the scene’s degrees of freedom sit in the feature grids, not in the weights. The MLP is kept small enough to evaluate cheaply at every sample along a ray. That division of labour is what lets volumetric integration approach interactive rates: the expensive representation is a table lookup; the learned part is a shallow decode.
The story
Volumetric neural rendering asks a simple question of every camera ray: what colour accumulates as the ray steps through space? Early neural radiance fields answered by feeding continuous coordinates straight into a large multilayer perceptron, which returned density and view-dependent colour at each sample. The results could be striking, but the cost was severe. Every sample was a full network evaluation, training ran for hours, and a finished frame was nowhere near real time.
Instant Neural Graphics Primitives reframes the representation. Instead of asking one large network to memorise the entire scene from raw coordinates, the method stores most of the information in a hierarchy of trainable feature grids and leaves only a thin decoder network. The grids are not dense voxel arrays. They are hash tables. A 3D position is scaled to each resolution level, hashed into an index, and used to retrieve a short feature vector. Features from all levels are concatenated and handed to a small MLP, which predicts the quantities needed for volumetric integration.
The multiresolution structure matters. Low-resolution levels capture broad layout—walls, large surfaces, the overall distribution of density. High-resolution levels capture edges, texture, and fine geometry. Because each level has its own fixed table size, memory does not explode with resolution the way a dense grid would. Collisions in the hash are accepted; the decoder learns to live with them, and the presence of several independent levels reduces the chance that every scale is corrupted at once.
Rendering still follows the familiar volumetric integral along each ray, but the cost per sample collapses. Table lookups and a shallow network replace repeated evaluations of a large MLP. Training benefits from the same split: gradients update both the hash-table entries and the small decoder, and the scene fits into a representation that modern GPUs can keep hot in memory.
The editorial point is not that neural rendering suddenly became free. It is that moving spatial detail out of network weights and into an engineered encoding—multiresolution hash features—recovered interactive rates while keeping the flexible, optimisable character of a learned scene. Quality that had been associated with offline path-tracing budgets for neural methods became something you could iterate on in real time.
Why it mattered then
When neural radiance fields first showed that a network could stand in for a scene, the field faced a practical bottleneck. Training was slow, and novel-view synthesis was far from interactive. Instant-NGP arrived as a concrete engineering answer: keep the learned volumetric integral, but stop asking a large MLP to be both the spatial index and the appearance model. Multiresolution hash encoding gave practitioners a representation that trained quickly and rendered at rates useful for inspection, capture loops, and demos—without abandoning the neural formulation that made the approach attractive in the first place.
Why it matters now
The same pattern—structured spatial encodings plus small decoders—still shapes how learned 3D representations are built. Whenever a system must query continuous space many times per pixel, the cost of the encoding dominates. Hash grids, feature volumes, and their relatives remain relevant wherever real-time or near-real-time neural rendering is required, and wherever memory bounds forbid naïve dense grids at high resolution. The lesson also generalises beyond graphics: put the high-frequency structure in a lookup that hardware already executes well, and reserve the network for the residual mapping you actually need to learn.
The surprising detail
The method does not try to prevent hash collisions. Two unrelated points in the scene can share a table entry at a given resolution, and the system relies on the small MLP—and on the fact that other resolution levels will not collide the same way—to sort out the ambiguity. That choice keeps table sizes fixed and implementations simple, at the price of a decoder that must be robust to mixed features. It is an unusually frank trade: controlled aliasing in the encoding, cleaned up by learning.
What is disputed
Reported speedups and absolute frame rates depend on scene, hardware, implementation, and sampling budget. The qualitative claim—hash-grid encodings with tiny MLPs reach interactive neural volumetric rendering where large coordinate-MLPs did not—is what the sources support; treat any specific milliseconds-per-frame figure as configuration-dependent.
Remember this
Real-time neural volumes here come from multiresolution hash features plus a tiny MLP—not from a large network alone.
Test yourself
A dense voxel grid at the finest resolution would avoid hash collisions entirely. Why might Instant-NGP still prefer fixed-size hash tables at multiple resolutions, even though collisions remain?
A dense fine grid grows with the cube of resolution and quickly exhausts memory; hash tables keep storage bounded at every level. Multiresolution features still supply coarse structure and fine detail, and the small MLP can absorb residual collision noise. The trade is bounded memory and fast lookup against imperfect spatial isolation—acceptable when the decoder is trained end-to-end on the same representation.
Go deeper
- [2106.05963] Learning to See by Looking at Noise · arxiv.org
- [2201.05989] Instant Neural Graphics Primitives with a Multiresolution Hash Encoding · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.