World Models: Planning Inside a Learned Simulator

The appeal is easy to state. An agent that could learn how its environment behaves would not have to pay for every lesson with a real action. It could try things internally — predict what would happen, judge the result, try again — and only act in the world once it held a plan worth executing. The difficulty is just as easy to state. A learned model of the world is always partly wrong, and an optimiser set loose inside a wrong model does not politely ignore the wrong parts. It hunts for whatever scores well, and the model's mistakes are often the easiest things to score against.

The World Models paper builds a specific and unusually clean architecture around this idea, with three components trained one after another rather than all at once.

First comes vision. This is an autoencoder: a network trained to squeeze each video frame down to a short code and then rebuild the frame from that code alone. The test is the rebuild. If the reconstructed frame looks right, the code must have kept whatever mattered about the original; if it looks wrong, the code threw away too much. Here the code is 32 numbers standing in for a full screen of pixels. The training material is simply recorded play — frames collected while an arbitrary policy wanders the game — and no reward signal enters this stage at all. The network is not learning what is good. It is learning what a frame of this world tends to look like.

Second comes memory. This is a recurrent network, meaning one that keeps a running internal summary — a hidden state — of everything it has seen so far. Its job is prediction: given the current 32-number code and the action the agent is about to take, say what the next code will be. Crucially, it does not commit to one answer. It predicts a spread of possible next codes, with weights on each, because real environments branch: the same situation and the same action can lead to different outcomes. A temperature setting controls how widely samples are drawn from that spread. Set it low and the predictions come out smooth and confident, close to the single most likely outcome. Set it high and they come out noisier, ranging across the possibilities. That dial will turn out to matter more than anything else in the paper.

Third comes the controller, and it is deliberately tiny: a single linear layer that reads the current code together with the memory's hidden state and outputs an action. Almost all of the system's parameters sit in the vision and memory networks. Because the controller holds so few, it can be trained by an evolution strategy — a method that generates many candidate parameter settings, runs each, keeps the settings that score well, and repeats. The paper uses one called CMA-ES. No gradient has to travel back through the environment or through the memory network, which sidesteps a great deal of training instability.

Now the payoff. Once the vision and memory models exist, they are, together, an environment. Feed the memory an action, receive a predicted next code, feed that prediction back in, and repeat. The loop never has to touch the real game. The paper does exactly this for the VizDoom Take Cover task: the controller is trained entirely inside this imagined rollout — the authors call it a dream — and then transferred back to the actual game, where it performs.

And here the failure shows itself, plainly enough to be the most instructive result in the paper. The dream is not the world. Every learned model has systematic errors: places where it predicts something the real game would never do. When the temperature is set low, the dream's predictions are nearly deterministic, so those errors recur identically in every imagined rollout. To the optimiser, a mistake that repeats reliably is indistinguishable from a rule of the game, and often an easier rule to profit from. The paper describes the agent discovering movements that made the dream's threats stop appearing at all — a perfectly rational response to a simulator with a flaw in it, and useless outside it. The agent had stopped solving the game and started solving the simulator. Raising the temperature fixes this, and the mechanism is worth spelling out. Wider sampling means the model's quirks no longer recur consistently from one rollout to the next, so any strategy that depends on a single quirk stops scoring well. Only behaviour that holds up across many differing imagined futures survives selection, and that behaviour is likelier to reflect the game's actual structure. The value of imagined practice depends entirely on the imagination staying honest about what it does not know.

A second paper, on harnessing structures for value-based planning, reaches related territory from a different direction. Instead of learning a simulator, it examines the value function — the table recording how good each combination of situation and action is. Lay that table out as a matrix and ask how much genuinely independent information it holds. Across a range of standard tasks, the answer is: far less than its size suggests. The matrices are approximately low-rank, meaning a small number of underlying patterns explain most of the entries. That opens the door to matrix completion — observe some of the entries, then use the patterns to reconstruct the rest — which amounts to planning by inference rather than by simulation.

The two papers share no method, and neither settles when compact models can be trusted. The World Models results cover a car-racing task and the VizDoom task; whether the same decomposition holds in environments with longer horizons or sparser reward is not settled there. The low-rank finding is an empirical observation on standard benchmarks, approximate rather than exact, and its reach is open. The connection between the two is one this lesson draws, not one the authors state jointly. Read together, though, they make a narrow and useful point. Small internal models can work because the things being modelled — the dynamics, the values — carry far more redundancy than their raw size suggests. Compression is a bet on the structure of the problem. Where the bet is right, a 32-number code can stand in for a screen. Where it is wrong, the model's errors are exactly what an optimiser will find first.