II · THE IDEA · ARTIFICIAL INTELLIGENCE
Weight Agnostic Neural Architecture Search
▶ Listen · narrated
A network that solves a control task with one shared random weight forces a question most training runs never ask: how much of the work was already done by the wiring?
At a glance
- What it is
- Architecture search that scores topologies under random weights, not trained ones
- Weight regime
- Often one shared scalar reused on every connection
- How scored
- Performance averaged across several random weight samples
- Editorial aim
- Expose inductive bias carried by connectivity alone
Think of a radio built from a fixed circuit board and a single volume knob. If someone searches for circuit boards that already pick up a clear station no matter where the knob sits—within reason—they are doing something like weight-agnostic architecture search. The board is the topology; the knob position is the weight.
In ordinary deep learning we design or borrow a board and then spend most of our effort tuning thousands of knobs. Here the knobs are deliberately unhelpful: they are set at random, and sometimes every connection is forced to share one setting. A candidate network is kept only if it still behaves well across several of those random settings. Whatever success remains must come from how the parts are wired—the paths, loops and bottlenecks—not from carefully chosen numbers.
The point is not that weights are useless. It is that good wiring does real computational work, and you can expose that work by refusing to let training cover for a weak diagram.
Weight-agnostic neural architecture search evaluates candidate graphs by performance under weight distributions that carry little or no task-specific information. In the representative formulation, a topology is instantiated with a single shared parameter value w drawn from a simple distribution; the network is rolled out on the task; and the draw is repeated so that selection depends on expected behaviour over w rather than on one lucky sample. Variants allow independent random weights per edge, which is a weaker probe because magnitude diversity can still encode accidental structure.
Search (typically evolutionary) mutates connectivity and activation choices while the inner loop refuses the usual inner optimisation over parameters. The outer objective therefore rewards graphs whose induced function class already intersects the task under severe parameter tying. Empirically this pressure yields compact controllers on tasks that admit small policies; it is not, by itself, a recipe for training frontier-scale perceptual models.
For a practitioner the useful residue is diagnostic. If a module remains competent under weight noise, heavy tying, or minimal fine-tuning, its topology is contributing inductive bias you may want to preserve under pruning, quantisation, or transfer. If competence collapses as soon as weights are not freely trained, the graph is mostly a scaffold for parameter fitting. Weight-agnostic evaluation is one way to tell those cases apart without trusting post-hoc stories about what training “found.”
Look closer
Weights are deliberately uninformative
In the usual training loop the architecture is fixed and the weights absorb the task. Here the procedure is inverted. Connections are varied while weights are drawn at random—and, in the strictest setting, every connection is forced to hold the same scalar. A topology only survives if it still produces useful behaviour across several of those draws. Whatever competence remains cannot be credited to carefully fitted parameters; it has to live in which units connect to which, and in the pattern of activation that wiring induces.
Shared weight is a harsher test than random weights
Allowing each connection its own random value still leaves room for accidental tuning: some edges get lucky magnitudes. Collapsing the network onto a single shared weight removes that escape. The same number must serve as a strong excitatory link in one place and a weak one in another, depending only on how information is routed. Architectures that keep working under that constraint are doing more of the computational labour in their graph structure than in their numerical settings.
Search pressure favours minimal competent graphs
Because a candidate is judged by behaviour under untrained weights, extra connections are not free. Unused or noisy pathways tend to hurt the average score rather than help it. The search therefore tends toward compact topologies that encode a usable policy or mapping with few parts. The interesting objects are not large nets that happen to tolerate randomness, but small ones whose wiring already sketches the right computation.
The story
Most neural network practice treats architecture as a container and learning as the filling of that container with numbers. Gradient descent, or some other optimiser, moves those numbers until the mapping from input to output matches the task. Weight-agnostic neural architecture search asks what happens if that division of labour is refused. Instead of asking which weights make a fixed graph succeed, it asks which graphs still succeed when the weights are essentially noise—or, more stringently, when every edge is forced to carry the same random scalar.
The procedure is a form of architecture search, but the inner evaluation loop is unfamiliar. A candidate topology is instantiated, a weight value (or a small set of values) is sampled, and the network is run on the task without a conventional training phase for those weights. The sample is repeated; performance is aggregated. Topologies that only work for a narrow lucky draw are discarded. Topologies that remain competent across draws are retained and varied. Over generations, the population is pushed toward structures whose behaviour is stable under weight uncertainty.
What this isolates is inductive bias in the purest wiring sense. A feed-forward chain, a particular pattern of recurrence, a sparse skip, a bottleneck—these choices constrain the functions the network can express even before any parameter is meaningful. When a sparse controller can balance a pole or steer a car under a single shared weight, the claim is not that weights never matter. It is that a surprising fraction of the policy was already implicit in the graph. The weights become a volume knob on a mechanism the topology has already assembled.
That finding cuts against a habit of thought in which architecture is a rough sketch and learning does all the real work. It also reframes failures of generalisation. If two networks with similar parameter counts behave differently out of distribution, part of the difference may be topological rather than a matter of which minimum the optimiser found. Weight-agnostic search is one way of making that topological contribution visible: strip the parameters of their usual authority and see what competence remains.
The method is demanding on the search process. Evaluating each candidate under multiple weight samples is expensive, and the tasks on which striking results appear are often ones where a compact policy exists—classic control and similar settings—rather than open-ended perception at web scale. None of that cancels the conceptual point. By refusing to let weights rescue a weak structure, the search surfaces structures that need less rescuing. Those structures are evidence that generalisation can be partly designed into connectivity, not only optimised into tensors.
Why it mattered then
At the time, neural architecture search was already busy enumerating cells, depths and connectivity patterns, but almost always with the assumption that a full weight-training run would follow and would carry the burden of performance. Weight-agnostic evaluation arrived as a deliberate counter-pressure. It borrowed the evolutionary and search machinery of that moment and removed the crutch those pipelines relied on. In doing so it joined a longer conversation—older than deep learning—about how much of a network’s behaviour is fixed by its diagram of connections. The work mattered in its own moment because it gave a concrete experimental handle on that question: not a philosophical claim that structure counts, but a search objective in which structure was almost the only thing allowed to count.
Why it matters now
Modern practice still pours most of its compute into fitting weights inside a handful of dominant templates. That works, and at scale it works dramatically, yet it leaves structural bias half-examined. When practitioners prune, distill, or transfer a model, they are often discovering after the fact which parts of the wiring carried the task. Weight-agnostic thinking pushes the question earlier: design or search for topologies that are competent before heavy optimisation, and the later training stage has less pathology to correct. The same instinct appears, in different clothes, whenever someone argues for stronger inductive biases, for tiny recurrent controllers, or for architectures whose behaviour degrades gracefully when parameters are noisy, quantised, or shared. The original search setup need not be reproduced wholesale for the lesson to apply. Any time a system remains useful under weight noise, parameter sharing, or minimal fine-tuning, some of the credit belongs to topology—and it is worth knowing how to look for that credit on purpose.
The surprising detail
The harshest and most memorable setup is not random weights per connection but one scalar reused everywhere. The network is asked to solve the task when it cannot even assign different strengths to different edges—only different patterns of use. That a search can still return controllers under that rule is a sharper result than “random nets sometimes work.” It suggests the graph is arranging a computation whose important distinctions are routing and nonlinearity, not a carefully tuned spectrum of weight magnitudes.
What is disputed
Public discussion of this line of work often leans on particular control demos and on the shared-weight setting. Those demos illustrate the claim; they do not by themselves establish how far weight-agnostic search extends to large perceptual models or to tasks where no compact policy exists. Treat the method as a probe of structural bias, not as a universal training replacement.
Remember this
When a network works with random shared weights, the topology is doing work we usually misattribute to training.
Test yourself
A team reports a small network that performs a control task well when every connection uses the same randomly drawn weight, averaged over several draws. A second network matches the score only after full gradient training with independent weights per edge. What different claims can honestly be made about the two topologies, and what claim cannot?
The first topology can be said to carry a strong task-relevant inductive bias in its connectivity: useful behaviour is already available before parameters are specialised. The second topology may still be an excellent substrate for learning, but the experiment has not shown that its wiring alone encodes the policy—the weights were allowed to absorb the difficulty. What cannot be claimed is that the second network is “worse” in general deployment; once trained, it might be more accurate or more stable. Weight-agnostic success measures structural prior, not ceiling performance after optimisation.
Go deeper
- [1906.04358] Weight Agnostic Neural Networks · arxiv.org
- [2106.05662] To The Point: Correspondence-driven monocular 3D category reconstruction · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.