Model-Based RL and the Limits of a Learned Simulator

A dynamics model is a learned function with one job: given the situation the system is in now, and the action it takes, predict the situation that comes next. Fit that function well and you own a simulator that runs for nothing. The agent can practise inside it over and over without touching the real robot or the real environment — nothing wears out and nothing breaks. That is the standing promise of model-based reinforcement learning, and it explains why the field keeps returning to the idea despite repeated disappointment.

The disappointment has a precise mechanism, and it is worth walking through slowly. Suppose the model's first prediction is slightly wrong — a joint angle off by a fraction of a degree. That slightly wrong state is fed back in as the input for the second prediction. But the model was trained on states the agent actually visited, and the slightly wrong state sits just outside that training data, where the model is less reliable. So the second prediction is wronger than the first. The second output becomes the third input, further from the data still. The error does not merely add up step by step; each error pushes the model into territory where it makes larger errors. Run the loop for 20 steps and the imagined trajectory can arrive somewhere real physics forbids — a limb passing through the floor, energy appearing from nowhere.

Now add the second ingredient of the failure. The policy — the decision-making rule the agent is training — is being optimised to score well inside this simulator. If the simulator has a region where it wrongly reports high rewards, optimisation will find that region, because finding high-scoring regions is exactly what optimisation does. The agent ends up expert at exploiting a bug in its own imagination, and useless in the world.

So the practical question is not whether to use a learned model but how far to run it before the imagined data stops helping the policy and starts poisoning it. The 2019 paper When to Trust Your Model, which introduced the method known as MBPO, gives one clean answer: keep the imagined stretches short, and start them from moments that really happened. The agent keeps a replay buffer — simply a stored log of everything it has experienced. The method picks a real state from that log, lets the current policy choose an action, asks the model what happens next, and repeats only a handful of times before stopping and starting afresh from another real state. Because every branch begins on solid ground, the model has only a short window in which to drift. And because the branches are cheap, the agent can grow a great many of them, producing a large volume of training data in which no single fabrication travels far.

That data feeds an off-policy learner — an algorithm, such as a soft actor-critic, that can learn from experience it did not generate itself. It trains on a mixture: real transitions from the environment, and a much larger quantity of these short imagined ones.

Short horizons alone are still a guess. What makes the discipline principled is the ensemble. Train not one dynamics model but several, each starting from different random initial weights and each shown a differently resampled slice of the same experience. Then compare their predictions. In situations the agent has visited often, the data pins all of them down, and they give nearly the same answer. In situations it has barely touched, the data constrains nothing, so each network falls back on its own inductive bias — the kind of pattern its structure prefers to fit — and the predictions fan out. The size of that fan measures how little the data said about this situation, and it costs nothing beyond the expense of training more than one model.

Each individual member is itself probabilistic: it outputs not a single next state but a spread of possibilities. That spread describes noise genuinely in the world — a slippery surface, a sensor that jitters — which no amount of exploring will remove. The disagreement between members describes something different: ignorance that more data would cure. Keeping the two apart matters, because only the second is a reason to shorten a rollout.

The disagreement signal can be used in at least three ways. Sample a different member at each imagined step, so a path through uncertain territory becomes visibly noisy rather than confidently wrong, and the policy learns not to rely on it. Subtract a penalty from the imagined reward wherever the members scatter, making unfamiliar states unattractive by construction. Or watch the scatter grow along a branch and cut the branch when it crosses a threshold. All three follow the same logic: disagreement among equally well-trained models stands in for the true error, which cannot be measured directly, because measuring it would require the ground truth the agent does not have.

Dynamics are not the only thing an agent can be wrong about. The second paper here, on fast Bayesian reward inference from preferences, applies the same style of reasoning to the goal itself. When a reward function is inferred from human comparisons — this attempt was better than that one — many different reward functions fit the same comparisons equally well. Rather than committing to a single best guess, the method keeps a whole distribution of plausible rewards, which allows a statement of the form: under most of the goals consistent with what the human said, this policy performs at least this well. That is a stronger and more useful claim than a good average, and it is the claim you want before deploying an imitation-trained system anywhere consequential.

Put the two papers together and the picture is symmetrical. An agent can be wrong about where its actions lead, and wrong about what it is being asked to achieve. In both cases the remedy is the same: hold several hypotheses at once, and let their disagreement decide how boldly to act.