Imitation Learning from Observation

Standard imitation learning is supervised learning in disguise. You gather a large set of pairs — the situation the expert was in, and the action the expert took in it — and you train a network to predict the action from the situation. This is called behavioural cloning, and it works whenever those pairs exist.

They usually do not. A recording of a surgeon, a driver, a dancer or a games player captures what a camera can see: positions, objects, outcomes. It does not capture hand pressure, wheel angle or which buttons were held down. So the half of the data that behavioural cloning trains on — the action — is exactly the half that ordinary recordings leave out. Imitation learning from observation is the attempt to learn anyway, and the two papers here fill the same missing column in different ways.

The first, Playing hard exploration games by watching YouTube, works on Atari games where the score changes so rarely that an agent learning by trial and error gets no feedback for thousands of frames at a stretch. Montezuma's Revenge, Pitfall! and Private Eye are the standard examples. The problem is mechanical. Reinforcement learning improves a policy by noticing which actions led to reward. If random play essentially never reaches a reward, there is nothing to notice and nothing to improve. The agent does not fail slowly; it fails to start.

Human videos of these games exist in quantity, but they are messy evidence. A clip pulled off YouTube was made by a different player, on unknown emulator settings, at a different resolution and frame rate, with compression smear and sometimes an overlay on top, and nothing in it lines up in time with the agent's own play. Compare such a frame with a frame from the agent's emulator pixel by pixel and the comparison says almost nothing about whether the agent has reached the same point in the game.

So the method's first job is to make the frames comparable. It learns an embedding — a short list of numbers describing each frame — trained so that frames from equivalent moments in different recordings end up with similar numbers, even when the raw pixels differ. No human labels are needed for this. The training signal comes from the videos themselves: from the order of frames within each video, and from the correspondence between several videos of the same game.

Once frames can be compared this way, one demonstration video becomes a route. Take checkpoints along it, in order, spaced through the run. The agent earns a small reward whenever its current frame lands close, in the learned embedding, to the next checkpoint it has not yet claimed. Notice what this does and does not require. The video never says which buttons the demonstrator pressed. It says where to be next. Which actions get the agent there is left for trial and error to discover — and now trial and error works, because reward arrives at every checkpoint rather than almost never.

The second paper, Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations, starts from a different problem. Copying has a ceiling. Copy a mediocre driver and the best you can hope for is a mediocre driver. But suppose you have several demonstrations, none of them good, and you can rank them from worst to best. The ranking carries something no single demonstration does. It points in the direction of improvement.

So instead of copying any demonstration, the method fits a reward function to the ranking. A reward function is a rule that scores each moment of behaviour; add up the scores along a whole run and you get that run's total. The network is trained so that better-ranked runs get higher totals than worse-ranked ones. Nothing in this needs the expert's actions — only the observed states, in order, and the judgement of which run beat which. The trained network is then handed to a standard reinforcement learning algorithm as the thing to maximise. Because the network has captured what tends to make a run better, rather than what any particular run did, maximising it can push the policy past the best demonstration it ever saw. The authors report exactly that: learned policies that outscore the demonstrations used to train them.

Both methods share a structure worth naming. Neither tries to work out what the expert's actions were, and neither has to. Both convert observations into a reward — a route of checkpoints in one case, a fitted scoring rule in the other — and then let reinforcement learning handle the part that needs to know which button produces which change on screen, because that part the agent can find out for itself by playing. The demonstration supplies the goal; interaction supplies the mechanics.

The same structure sets the limit. Both approaches assume the agent can practise freely in an environment resembling the one in the recordings. Neither is a way to learn from video alone, with no play at all. What they remove is the need for a matched log of expert actions — the requirement that keeps most real-world footage out of imitation learning entirely.

It is also worth saying what the record does not settle. The two papers used separate benchmarks, so nothing here shows how they compare against each other; the pairing is editorial. The checkpoint method was reported on a small number of hard-exploration Atari titles, which is a narrow base from which to generalise. And the ranking method depends on the rankings being reliable. How far a learned reward can be pushed beyond its demonstrations before it stops corresponding to the real task is a question neither paper closes.