Inverse Reinforcement Learning

Ordinary reinforcement learning starts with a number. A designer writes a reward function - a rule that hands the agent a score for each thing it does - and the agent then does whatever raises that score. When the agent misbehaves, the designer edits the rule and tries again. The trouble with this loop is well known: an agent will raise the written number by any route available, and the routes the designer never thought of are exactly the ones it tends to find.

Inverse reinforcement learning turns the order round. You begin with behaviour rather than a rule. Someone competent performs the task, and their run is recorded as a trajectory: the list of situations they passed through and the actions they took in each. The question is then asked backwards. Which scoring rule would make those particular actions the sensible ones to choose? Fit such a rule, and you have a reward you never had to write.

The fitting works by comparison. Take a candidate rule. Under that rule, score the actions the expert actually took, and score the actions they could have taken instead. If the expert's choices come out on top, the rule explains the behaviour; if not, adjust the rule and check again. Different methods carry out this comparison in different ways, but the shape is the same: the recorded behaviour is treated as evidence, and rules survive when they make that behaviour look like winning play.

Here the method meets its central difficulty, and it is worth going slowly. A recording says very little. A trajectory shows one route through the task. It does not say which other routes were considered, nor why they were dropped, nor how much worse a near miss would have been. So a great many different rules explain the same run equally well. Some of those rules capture what the expert cared about. Others merely reward some feature that happened to sit alongside good play on that route, and would fall apart anywhere else. Nothing in the demonstration itself tells the two kinds apart. This is not a snag to be tidied up later; it is the central difficulty of the whole approach.

One way to add information is to ask a person to judge. Show them two short clips of behaviour and ask which is better. Each answer supplies something a demonstration cannot: not just what was done, but which of two things should score lower. Every comparison rules out the candidate rewards that rank that pair the wrong way round, so the set of surviving rules shrinks with each label. Work on reward learning from human preferences and demonstrations in Atari sits exactly here, combining the two kinds of evidence. Games suit the job for a plain reason: both kinds of data can be collected, and once an agent has been trained on the recovered reward, you can watch it play and see on screen whether the reward captured anything real.

The second stage is ordinary reinforcement learning, only pointed at the inferred reward instead of a written one. This is where the method parts company with copying. The obvious alternative, behaviour cloning, trains an agent to reproduce the expert's actions directly, so the agent never strays far from what it saw. Reward learning trains an agent to raise a score, and the agent is free to find any route that raises it, including routes the expert never showed. If the recovered reward really does capture what success means, that freedom is the whole payoff. The agent can outdo its teacher, and it can cope with situations the teacher never faced, because it carries the goal rather than the moves.

If the reward captured something else, the same freedom becomes the danger, and the mechanism deserves stating as cause and effect. Suppose the recovered rule rewards a feature that merely accompanied good play in the demonstrations. On the demonstrated routes the rule looks fine, because there the feature and the good play arrive together. But the optimiser - the training process searching for high scores - does not stay on the demonstrated routes. It searches everywhere it can reach. Somewhere in that wider space are states where the feature appears without the good play, and because the rule pays for the feature, the optimiser steers straight to them. The error made in the first stage does not stay the size it was. The second stage hunts for it and enlarges it.

Preference labels help but do not settle the matter. They are sparse, because a person can only watch so many clips. They carry noise, because judges disagree or misread what they saw. And they only speak about the particular clips someone happened to watch, which leaves the rest of the task unconstrained. So evaluation cannot end with a reward that explains the recorded runs. It has to include watching how the trained agent behaves in states the demonstrations never reached. Atari is instructive partly because failure there is visible rather than statistical: an agent chasing a badly recovered reward looks odd on screen, not just a little low on points.

What the sources here do not settle is also worth saying. They establish the shape of the approach - infer a reward from demonstrations and comparisons, then optimise it - but not a general verdict on when this beats plain imitation, nor precise numbers for any one system. Treat the mechanisms above as the standard anatomy of the method, not as a scorecard.

Read as a whole, the method moves human effort rather than removing it. People no longer write the objective in code; they supply behaviour and judgements, a model proposes an objective from those, and an optimiser stress-tests the proposal by trying its hardest to exploit it. The hope, for alignment work, is that intent is easier to show and to criticise than to state exactly. The matching caution is that the two stages are joined: whatever the first stage gets wrong, the second stage will look for.