II · THE IDEA · ARTIFICIAL INTELLIGENCE
Recurrent Independent Mechanisms
▶ Listen · narrated
When distinct processes only sometimes interact, one tangled hidden state is a poor match. Keeping them separate—and letting them talk sparingly—can change what the network generalises to.
At a glance
- What it is
- A recurrent net as many independent modules, not one shared state
- Activation
- Only a sparse subset of modules updates on each step
- Communication
- Active modules exchange information; inactive ones stay frozen
- Design aim
- Compositionality and more systematic generalisation
Think of a workshop with several specialists, each at their own bench. When a new job arrives, only the two or three specialists who recognise it get up; the others leave their work exactly as it stands. The active few glance at the job, update their own notes, and may lean over to exchange a quick word with one another. Then the next job arrives and a different subset may rise.
A RIM network works in a similar way. Instead of one shared memory that is rewritten on every step, it keeps many small recurrent modules. A competition decides which modules the current input is most relevant to. Only those winners read the input and update. They may also share information among themselves. Everyone else stays frozen until a later step selects them. The point is to encourage each module to become good at one kind of subprocess, so familiar pieces can be reused in new combinations without tangling the whole memory.
RIMs factor a recurrent hidden state into n modules h_1,…,h_n, each with its own parameters. At time t, input attention scores are computed between each module state and a per-step representation of x_t (or encoder outputs). The top-k modules by score are marked active. Each active module performs an attention-based read from the input and applies a recurrent update to its own state; inactive modules set h_i^{t} ← h_i^{t−1} with no gradient path through an update on that step.
A second attention stage allows active modules to communicate: queries from each active module attend over keys and values from the active set (sometimes including residual links to the module’s own pre-communication state). Inactive modules are excluded from this exchange. The result is sparse, conditional interaction rather than full all-to-all mixing.
Hyperparameters that matter in practice include n, k, module dimensionality, and whether communication is symmetric. Limitations follow from the same sparsity: if selection collapses to a fixed subset, unused modules contribute little; if k is too large, the independence bias weakens toward a standard multi-headed recurrent net. Specialisation is encouraged, not enforced, and evaluation of systematic generalisation remains task-dependent.
Look closer
Competition for the input
At each time step the modules do not all read the input equally. An attention-like competition ranks them by how relevant the current input appears to each module’s stored state. Only the top-k winners are allowed to update. The rest keep their hidden states exactly as they were. The sparsity is hard, not soft: losers are idle for that step.
Two kinds of attention
RIMs separate input attention from inter-module attention. First, selected modules attend to the input (or encoder outputs) to write a new proposal for their own state. Second, those same active modules may attend to one another so information can pass between mechanisms that are co-active. Modules that lost the competition neither write nor speak; their parameters and states are untouched on that step.
Independence as an inductive bias
The architecture does not merely add capacity. By freezing most modules most of the time, it encourages each one to specialise on patterns that recur independently of others. When a familiar subprocess reappears in a new combination, the module that already handles it can be re-selected without rewriting the rest of the network’s memory. That is the intended route to compositional reuse.
The story
A standard recurrent network carries a single hidden vector through time. Every new observation is mixed into that same vector, and every parameter that touches the hidden state can, in principle, affect every aspect of the past. That design is flexible, but it offers no reason for the network to keep distinct processes distinct. If the training distribution always presents two factors together, their representations tend to entangle. At test time, a novel combination can then fail even when each factor alone was frequent in training.
Recurrent Independent Mechanisms start from a different premise: many systems worth modelling are better described as a collection of mechanisms that run largely on their own and only occasionally interact. The network is therefore built as a set of recurrent modules, each with its own parameters and its own hidden state. The modules are not a committee that always votes. On each time step a sparse subset is chosen to be active, and only that subset may update.
Selection is competitive. Queries derived from each module’s current state are compared with keys derived from the input (or from a per-step encoding of it). The modules with the strongest match win; a fixed number k are kept. Winning modules then use attention over the input to produce an update to their own state, in the style of a recurrent cell that can focus on particular parts of what just arrived. Losing modules copy their previous state forward unchanged.
After the input-driven update, the active modules are allowed a second, narrower stage of communication: they may attend to one another’s states. Information can therefore move between mechanisms that are co-active on the same step, but it does not flood through the whole set. Inactive modules remain silent and unmodified. The next step repeats the competition from the new states, so the cast of active mechanisms can change as the sequence unfolds.
The intended effect is compositional. A module that has specialised on one kind of subprocess—tracking an object, keeping a count, modelling a physical law—can be re-awakened whenever that subprocess is relevant, without forcing every other module to rewrite its memory. When the test distribution recombines familiar factors in unfamiliar ways, the hope is that the right subset of modules lights up and the others stay out of the way. The paper presents this sparse, modular recurrence as an architectural inductive bias toward systematic generalisation, rather than as a guarantee that any particular dataset will yield clean specialisation.
Nothing in the design requires the modules to be semantically labelled in advance. Specialisation, when it appears, is an outcome of the learning dynamics under the sparsity constraint. How cleanly that happens depends on the data, the number of modules, the value of k, and the rest of the training setup. The architecture supplies a structural preference for independence and sparse interaction; it does not hand the modeller a named catalogue of mechanisms.
Why it mattered then
By 2019, recurrent and attention-based sequence models were already strong on many benchmarks, yet systematic generalisation—recombining known parts into new wholes—remained fragile. A single shared hidden state offers no structural reason to keep distinct causal factors apart, so co-occurrence in training often becomes entanglement in the representation. RIMs proposed a concrete alternative: keep many recurrent states, activate few of them, and restrict communication to the active set. That was a direct architectural response to the compositionality problem as it was then framed, rather than only a capacity or optimisation tweak.
Why it matters now
Modular and sparsely activated networks remain a live design theme, from mixture-of-experts routing to object-centric and mechanism-based world models. RIMs sit in that lineage as an early, explicit argument that recurrence itself can be factored into independent processes with gated interaction. Anyone building systems that must reuse skills across changing combinations of objects, rules or sub-tasks still faces the same tension between a convenient shared state and a factored one. The paper’s vocabulary—competition for activation, frozen idle modules, sparse inter-module attention—remains a clear reference point for that trade-off.
The surprising detail
The idle modules are not softly down-weighted; they are strictly copied forward. On a given step, most of the network’s recurrent state is deliberately stale by design. Learning still reaches those modules, but only on steps when they win the competition, so each module’s effective training signal is already a filtered subset of the sequence. That makes specialisation possible, and it also means a poorly tuned k or a weak selection signal can leave some modules under-used rather than elegantly independent.
What is disputed
The second listed source addresses theta and eta correspondences in representation theory and is unrelated to this architecture; all technical claims here follow the Recurrent Independent Mechanisms paper alone. How reliably modules specialise in practice is empirical and setup-dependent, not settled by the architectural description.
Remember this
RIMs replace one shared recurrent state with many modules, activate only a few per step, and let only those few read the input and talk to each other.
Test yourself
A RIM model is trained on sequences where object motion and background lighting always change together. At test time they vary independently. Why might the architecture still help relative to a monolithic RNN, and what would still have to go right during training?
Sparsity and separate states give the model a structural chance to park motion and lighting in different modules, so that at test time only the relevant module need update. But the architecture does not force that split: if selection never separates the two factors during training, both can still entangle inside the same module or remain co-activated. The inductive bias is real; successful factorisation is still an outcome of learning.
Go deeper
- [1909.10893] Recurrent Independent Mechanisms · arxiv.org
- [2006.06241] On Theta and Eta Correspondences for Finite Symplectic/Orthogonal Dual Pairs · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.