II · THE IDEA · ARTIFICIAL INTELLIGENCE
Compositional Visual Reasoning with Neural Module Networks
▶ Listen · narrated
Most vision models answer every question with the same fixed network. Neural module networks build a fresh layout each time, shaped by the structure of the words.
At a glance
- Core idea
- Compose a network from specialised modules using the question’s structure
- Input
- An image plus a natural-language question about it
- Output
- An answer from executing the assembled module layout
- Modules
- Reusable neural pieces with typed roles, not one fixed net
- Structure
- Taken from a parse of the question, not handwritten per image
Think of a kitchen drawer of tools: a finder, a comparer, a counter, a colour-reader. When someone asks a question about a picture, you do not rebuild the whole kitchen. You pull out only the tools that question needs and use them in order. “What colour is the ball next to the cat?” might mean: find the cat, find the ball near it, read the colour. Neural module networks work in that spirit. The question is turned into a small plan. Each step in the plan is a small neural network with a job. Those pieces are joined only for that question, run on the image, and produce an answer. The same finder can be reused in many plans; what changes is the wiring.
Neural module networks address visual question answering by factoring prediction into (1) a structure-predicting stage that maps a question to a layout over a catalogue of module types, and (2) an execution stage in which that layout is instantiated as a directed computation graph and evaluated on image features. Modules are specialised networks with typed inputs and outputs — commonly attention maps, region features, or answer distributions — so that outputs of one module are valid inputs to the next. Parameters are shared across instances of the same type and trained over many layouts, which is what makes composition useful rather than a collection of one-off subnetworks. The approach is neurosymbolic in a narrow sense: the layout is discrete and linguistically driven; the module internals are differentiable. Limitations follow from that split. Layout errors are hard to recover from inside execution. The catalogue must be expressive enough for the question distribution. Generalisation depends on modules learning roles that remain stable under novel wirings, which is not guaranteed if training layouts are narrow. Compared with a monolithic encoder–decoder VQA model, compute and implementation cost shift toward dynamic graph construction and batching heterogeneous layouts, while inspection gains intermediate maps along an explicit path.
Look closer
The layout is not fixed in advance
A conventional visual question-answering model runs every query through one shared architecture. Here the connectivity changes with the question. A query that asks where something is, and another that asks how two things relate, can produce different graphs of modules even when they share some of the same building blocks. The composition is the reasoning plan; the weights inside the modules are what get learned.
Modules are specialised by role
Rather than one undifferentiated stack of layers, the approach defines a catalogue of module types with distinct jobs — for example locating a mentioned object, transforming an attention pattern, combining two intermediate results, or mapping a final representation to an answer. Instances of those types are selected and wired according to the parse. The same find-type module can be reused for different nouns; what changes is which instance is called and what it receives.
Language supplies the scaffold
The question is not merely encoded as a bag of features. A linguistic analysis of it determines which modules appear and how their inputs and outputs connect. In that sense the model does not invent its own control flow from scratch for each sample; it inherits a skeleton from the structure of the query, then fills that skeleton with learned neural computation over the image.
The story
Visual question answering asks a model to look at an image and reply to a free-form question about it. The difficulty is not only recognition. Questions differ in structure: some request a colour, some a count, some a relation between two regions. A single fixed network must absorb all of those patterns in one set of weights and one unchanging data path.
Neural module networks take a different route. They start from the observation that the question already contains a kind of program. If a linguistic analysis can expose that structure — which phrases name objects, which words signal relations, which clause is the thing being asked — then a network can be assembled to mirror it. The pieces of that network are modules: small neural components, each trained for a family of operations, with interfaces that let them be stacked and branched.
In outline the pipeline is straightforward. The question is analysed into a symbolic layout. That layout is instantiated as a graph of modules drawn from a fixed inventory. The image is processed through that graph. Intermediate results are typically spatial representations, such as attention over image regions, that one module can hand to the next. A final stage maps whatever representation remains into an answer. Training adjusts the module parameters, often across many different layouts, so that a module learns a role that can transfer when it is plugged into a new configuration.
Composition is the point. A module that learns to attend to a named kind of object is useful in many questions, not only the ones seen during training in a particular order. A module that combines two attentions can serve both a question about whether two things touch and a question about which of two things is larger, if the surrounding layout asks for those readings. The hope is that the inventory stays small while the set of expressible reasoning patterns grows with the ways modules can be wired.
This is still a neural method end to end once the layout is chosen: the modules are differentiable, and gradients can flow through the assembled graph. What is symbolic is the choice of which graph to build. That choice is the explicit compositional step. It is also a point of brittleness. If the analysis of the question is wrong, the wrong modules are joined, and no amount of smooth optimisation inside those modules fully repairs a mistaken plan. Later work in the same broad family has explored softer ways to select and mix modules; the original framing keeps the layout discrete and readable.
The contrast with a monolithic encoder is practical as well as conceptual. In a fixed network, inspecting why an answer appeared means probing distributed activations. In a modular layout, one can at least ask which sub-computation was supposed to run — which find, which combine — and inspect intermediate maps along that path. The modules do not by themselves guarantee correct reasoning, but they make the intended decomposition visible.
Nothing in the approach requires that every visual question factor cleanly into a neat expression tree. Many real questions are vague, elliptical, or depend on world knowledge that no small catalogue of visual operators will capture. The method is strongest where the question’s structure really does line up with a sequence of localisable visual operations. That is a narrower slice of language than free conversation, and the paper’s setting is visual question answering rather than open-ended dialogue. Within that slice, the claim is modest and sharp: structure in the question can be structure in the network, if the network is allowed to change shape.
Why it mattered then
At the time, strong results on vision-and-language tasks were increasingly coming from large, uniform networks trained end to end. Those models worked, but they offered little grip on how a particular answer was assembled, and they treated every question as the same kind of mapping from pixels and words to a label. Neural module networks argued for an older idea in a new form: that reasoning is partly the assembly of reusable parts, and that language can name those parts. Making the assembly explicit was a way to bring compositionality back into neural visual reasoning without giving up learned features inside each part.
Why it matters now
Compositional generalisation remains a live concern. Models still fail when a familiar relation appears with unfamiliar objects, or when a question nests operations in an order rarely seen in training. The modular pattern — parse or plan first, then execute with specialised pieces — reappears in program-of-thought methods, tool-using agents, and neurosymbolic pipelines. Even where modern systems keep a single backbone, the instinct to separate “what should be computed” from “how each step is computed” owes a debt to this line of work. Readability of the intermediate plan is still one of the few practical handles we have on visual reasoning errors.
The surprising detail
The network that runs is not stored as one static graph. It is built on the fly from the question, so two prompts that look similar in plain text can compile into different circuits, while two very different sentences that share a parse shape can share a layout and only swap which module instances are filled in. The model’s “architecture” is partly a function of linguistic analysis at inference time.
What is disputed
Public discussion of this work often blurs the original discrete layouts with later soft or learned routing variants. The core paper frames modules as composable pieces selected from a catalogue using linguistic structure; details of inventory design, parser quality, and how far composition helps on free-form questions remain dependent on the specific setup and are easy to over-generalise.
Remember this
The question is compiled into a layout; the modules only execute that layout. Structure comes from language, computation from the neural pieces.
Test yourself
A monolithic VQA network and a neural module network both answer questions about the same image. For a new question whose linguistic structure never appeared in training, why might the modular system still have an advantage even if every individual module has been seen before in other layouts?
Because the modular system can recombine familiar module types into a new graph that mirrors the new structure, whereas the monolithic net must rely on whatever fixed path and shared weights it learned, without an explicit way to assemble a fresh sequence of operations. The advantage holds only if the parse is right and the modules truly carry transferable roles; a wrong layout removes the benefit.
Go deeper
- [1511.02799] Neural Module Networks · arxiv.org
- [2004.09553] Structure theorems for idempotent residuated lattices · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.