II · THE IDEA · ARTIFICIAL INTELLIGENCE
Self-Play with Population-Based Training
▶ Listen · narrated
Gradient updates are not the only route to a strong policy. Evolving a population of deep networks can reach competitive returns without ever computing a gradient at all.
At a glance
- What it is
- Improving policies by evaluating, selecting and varying a population of candidates
- Neuroevolution
- Genetic algorithms treating network weights as genomes scored by return
- Claim tested
- GAs can be a competitive alternative to deep RL on control benchmarks
- Offline RL
- Learning from a fixed dataset without further environment interaction
Think of a sports club rather than a lone athlete. Each season every player on the squad plays matches and gets a score. The coach keeps the better players, drops the weakest, and brings in slightly altered copies of the keepers — new trainees who play almost like a veteran but not exactly. Over many seasons the squad as a whole gets stronger, even though nobody sat down and calculated the perfect tiny adjustment to one person’s technique.
That is the idea behind training a population of neural networks for reinforcement learning. Each network is a full candidate controller. You run it in the environment, read how much reward it earned, and let high scorers parent the next generation through small random changes to their weights. Deep neuroevolution showed this style of genetic algorithm can compete with popular gradient-based deep RL methods on standard control tasks.
A different situation is offline RL: you are not allowed new matches, only a box of old recorded games. You still might keep several candidate networks and pick among them using that fixed box, but you cannot send mutants out into a live world to prove themselves. The optimistic view is that careful learning from the box is still enough to improve; the caution is that the box may not cover the situations you care about.
Deep neuroevolution frames policy search as a genetic algorithm over neural network parameters. Individuals are deep weight vectors; fitness is episodic return (or a related performance statistic); variation is typically isotropic or structured mutation in parameter space; selection retains elites and refills the population from them. No back-propagation through a policy or value objective is required for the outer loop. Empirically, on the RL benchmarks studied, such GAs were reported as competitive with contemporary deep RL baselines, especially when parallel evaluation is abundant and the fitness signal, though sparse, is sufficient to rank individuals.
Mechanically this differs from policy gradient and Q-learning families: credit assignment is whole-episode and comparative, not local along a computational graph. Sample cost per generation scales with population size times rollouts per individual. Failure modes include premature convergence when diversity collapses, high variance of fitness estimates under stochastic environments, and wasteful mutation in extremely high-dimensional parameter spaces without sensible scaling of step sizes.
Offline RL removes interactive sampling. Algorithms learn from a fixed dataset and must limit exploitation of out-of-support actions. An optimistic line of work argues that strong policies are still attainable under those constraints when the data distribution is adequate and the update rule is conservative with respect to unseen state-action pairs. Population or ensemble structure may appear as multiple value heads or candidate policies evaluated on the same buffer, but that is not the same as online self-play against archived opponents. Combining the ideas carefully means using population search only within the support the logs justify, or using online population evaluation only when fresh interaction is allowed.
Look closer
Selection replaces the gradient step
In the neuroevolution framing, each individual is a full deep network. Candidates are scored by the return they achieve in the environment. Better scorers are kept as parents; poorer ones are discarded. Offspring are produced by copying a parent and applying random variation to its weights. No back-propagation is required for that loop. The population as a whole is the learner: progress is the shift in which genomes survive, not a single parameter trajectory descending a loss.
Diversity is part of the mechanism
A population that collapses to near-copies of one winner stops exploring. The deep neuroevolution line of work treats variation — and, in some setups, explicit novelty pressure — as a way to keep different behaviours alive long enough to be tested. That is a different bet from a lone agent following one noisy gradient estimate. Whether the diversity is accidental (random mutations) or encouraged, the observable fact is that several distinct policies are always under evaluation at once.
Offline data changes what competition can mean
An optimistic reading of offline reinforcement learning stresses that useful policies can be trained from logged experience alone, without fresh interaction. In that setting there is no live opponent and no live self-play loop: the fixed dataset is the world. Population or ensemble structure still appears, but as a way to stabilise estimates and hedge against distributional shift in the logged data, rather than as agents literally playing one another frame by frame.
The story
Most introductions to deep reinforcement learning centre on one agent, one set of weights, and a stream of gradient updates derived from trial and error. Population-based training offers a different picture. Instead of a single policy being nudged through parameter space, a collection of policies is maintained at once. Each is evaluated. Some are retained. Others are replaced by modified copies of the survivors. Over generations the distribution of behaviours in the pool shifts toward higher return.
Deep neuroevolution makes that picture concrete for large neural networks. Genetic algorithms, long used on smaller controllers, were shown to be a competitive alternative for training deep networks on reinforcement learning benchmarks. The genome is the weight vector. Fitness is episodic return. Variation is typically random perturbation of parameters. Selection is truncation or a similar keep-the-best rule. The method does not need differentiable policies or value functions; it only needs to run candidates and read scores. Parallel workers fit the scheme naturally, because evaluation of one individual does not depend on the back-prop graph of another.
That design choice matters for how improvement feels from the outside. A gradient method can look smooth on a learning curve even when the policy is brittle. A population method shows a cloud of returns: elites, middling contenders, and failures that will not reproduce. The editorial angle is competition against a diverse set of versions — often past survivors kept in the pool — rather than pure competition against a single moving target. Diversity is not decoration. If every member of the population becomes a near-copy of the current champion, the search collapses and the process stops discovering new behaviours.
A related but distinct setting is offline reinforcement learning, where the learner is denied further environment interaction and must work from a fixed dataset of logged experience. Here the optimistic claim is that strong policies remain attainable without on-policy rollouts, provided the algorithm respects the support of the data and does not wander into actions the logs never cover. Population structure can still help — as an ensemble of value estimates, or as multiple candidate policies scored against the same static buffer — but the “opponents” are no longer live copies of past selves in a shared simulator. They are alternative hypotheses about the same frozen evidence.
Read together, the two strands separate mechanism from romance. Self-improvement need not mean one hero network forever climbing its own loss. It can mean a pool of candidates, scored honestly, with selection pressure and controlled variation doing the work that gradients usually do. Where the environment is open and parallel evaluation is cheap, genetic algorithms on deep weights can be surprisingly strong. Where the environment is closed and only logs remain, the same instinct — keep several hypotheses alive, distrust a single overfit champion — reappears in offline methods that refuse to pretend the data cover more than they do.
Why it mattered then
When deep neuroevolution results appeared, the default assumption in much of the field was that scalable deep RL required back-propagation through a carefully shaped objective. Showing that a genetic algorithm could train deep networks to competitive returns on standard control benchmarks mattered because it widened the set of workable training loops. It also reframed compute: many independent evaluations can substitute for a sophisticated gradient estimator when workers are plentiful and communication is light. Offline RL work of the same broad period answered a different pressure. Real deployments often cannot afford endless online trial and error. An optimistic reading argued that logged data, used carefully, still support policy improvement — a claim that mattered for anyone sitting on historical interaction logs and unwilling to let an untrained agent explore live systems.
Why it matters now
Population thinking remains a practical counterweight to single-run training. When a lone seed looks brilliant and the next seed collapses, maintaining diversity across candidates is not nostalgia for evolutionary computation; it is insurance. Offline and hybrid pipelines are also ordinary now: many teams train from mixed logs first and only later allow limited online fine-tuning. The old tension — how hard to push beyond the data, how many alternative policies to keep alive — is still the engineering problem. For open-weight and local experimenters, the lesson is portable. If gradients are awkward, discontinuous, or expensive to route, a scored population with simple mutation is a legitimate baseline. If interaction is restricted, treat the buffer as a hard world and let several candidates compete on held-out slices of it rather than crowning one fit early.
The surprising detail
A result that still catches people out is how far a simple genetic algorithm can get on deep policies: random weight perturbations plus selection, without any gradient, proved competitive on deep RL benchmarks in the deep neuroevolution work. The sophistication sits in evaluation volume and selection discipline, not in the variation operator. Equally sobering is the offline counterpart: optimism is conditional. Strong offline performance depends on what the logs actually cover; a population of candidates cannot invent support the dataset never provided.
What is disputed
The deep neuroevolution paper argues genetic algorithms are a competitive alternative on the benchmarks it studies; that is not a universal claim across every RL domain, action space or compute budget. Offline RL optimism likewise depends on dataset coverage and algorithm choice — thin or biased logs remain a hard limit no population size removes.
Remember this
A population scored by return can improve deep policies by selection and variation alone — gradients are powerful, not mandatory.
Test yourself
You can either train one deep policy with gradient-based RL, or maintain a population of deep policies scored only by episodic return with mutation of winners. Name two concrete ways the failure modes differ when the reward is sparse and many seeds look identical for a long time.
The gradient run depends on a learning signal that may simply be zero or vanishingly small for long stretches, so the single trajectory can stall without a useful update direction. The population run can still rank candidates if any rare success appears, because selection needs only comparative scores, not a differentiable path to them — but it burns far more evaluations and can waste workers on near-clones unless diversity is actively preserved. Second, the gradient agent explores along one noisy path in parameter space; the population explores several lineages at once, so a brittle champion can be replaced by a cousin that failed earlier yet mutates into a working behaviour. The cost is bookkeeping and compute: you must store, evaluate and cull many full networks, not one.
Go deeper
- [1712.06567] Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning · arxiv.org
- [1907.04543] An Optimistic Perspective on Offline Reinforcement Learning · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.