II · THE IDEA · ARTIFICIAL INTELLIGENCE
Hindsight Experience Replay
▶ Listen · narrated
Sparse rewards starve off-policy learners of signal. HER turns every miss into a success for a different goal, so failed episodes still update the policy.
At a glance
- Core idea
- Replay failed trajectories as if the achieved outcome had been the goal
- Reward type
- Sparse binary success or failure on multi-goal tasks
- Requires
- Off-policy learning and a goal-conditioned value or policy
- Related
- Temporal Difference Models later connect HER-style ideas to model-based control
Think of practising free throws in an empty gym. You aim at the left hoop and miss, but the ball drops cleanly through the right hoop. A normal coach says “failed” and moves on. A hindsight coach says two things at once: “You missed the left hoop,” and also “If the target had been the right hoop, that shot was perfect—remember how that felt.”
Hindsight Experience Replay does the second kind of coaching for a learning agent. The agent tries for a goal, usually fails, and stores the attempt. Then it pretends the place it actually ended up was the goal all along, marks those steps as success for that pretend goal, and learns from them. The real goal is still practised; the pretend goals simply stop the training log from being full of pure failure. The agent must already be the sort that learns from a mixed memory of past attempts, not only from the attempt it is making right now.
HER assumes a multi-goal MDP in which the policy π(a|s,g) and critics are goal-conditioned, and the reward r(s,a,s',g) is a sparse function of the achieved state and the goal (commonly 0 on success and −1 otherwise). Episodes are generated under a behaviour goal g. Transitions (s,a,s') are stored in a replay buffer. For each episode, HER additionally samples goals g' from states visited in that episode (the future strategy samples g' from states after the transition’s timestep) and stores copies (s,a,s',g') with reward recomputed as r(s,a,s',g').
Because g' ≠ g in general, the relabelled data are off-policy with respect to the goal the behaviour policy conditioned on. Compatible base learners are therefore off-policy algorithms with replay (e.g. DDPG in the original paper). On-policy methods cannot consume the relabelled tuples without importance correction or equivalent machinery that HER does not provide.
Limitations follow from the mechanics: the reward must be recomputable from stored state and an arbitrary goal without further environment interaction; goals must live in a space where achieved states are valid goals; and relabeling densifies signal for goals near the behavioural distribution, not for arbitrary far-away goals the agent never approached. Temporal Difference Models (arxiv 1802.09081) later relate multi-goal value learning to model-based control; HER itself remains a model-free replay technique.
Look closer
The goal is part of the transition
In the multi-goal formulation HER uses, each transition is stored with a goal. The reward is computed from the achieved state and that goal, not baked into the environment forever. Once the goal is an argument rather than a fixed property of the MDP, it can be rewritten after the fact without replaying the robot or the simulator.
Failures become labelled successes
A rollout aimed at goal g that ends in some other state s' still contains a coherent sequence of states and actions. HER copies those transitions into the replay buffer a second time with a new goal taken from a state that was actually reached—often a future state in the same episode—and recomputes the sparse reward. Relative to that substituted goal the episode succeeded, so the buffer gains a positive sample it would otherwise never have seen.
Off-policy algorithms only
Because the stored goal no longer matches the one the behaviour policy was pursuing, the data are off-policy with respect to the original intention. HER therefore pairs with off-policy methods that already learn from a replay buffer. On-policy learners that require trajectories generated under the current goal cannot use the relabelled copies without further machinery.
The story
Reinforcement learning with a sparse binary reward is simple to specify and hard to optimise. The agent receives a success signal only when it reaches a designated goal; every other outcome is failure. In continuous control, and especially in multi-goal robotic tasks, random exploration almost never hits the target. The replay buffer fills with trajectories whose returns are uniformly zero, and gradient updates have almost nothing useful to say.
Hindsight Experience Replay attacks that sparsity at the data layer rather than inside the optimiser. The setting is multi-goal: the policy and the value function are conditioned on a goal supplied at the start of the episode. When an episode finishes, HER does not throw away a failed attempt. It samples one or more goals from the states that were actually visited, pretends those had been the intended goals all along, and recomputes the sparse reward for each transition under the new goals. A trajectory that missed the original target is almost always a clean success for the state it did reach.
The original transitions, still labelled with the true goal, remain in the buffer as well. The agent therefore keeps practising the task it was asked to solve, while the hindsight copies supply a dense stream of successful examples for nearby goals. Because the reward is a deterministic function of achieved state and goal, no extra environment interaction is required to relabel; the arithmetic is done offline on stored tuples.
The method is deliberately thin. It does not build a dynamics model, invent shaped rewards, or change the exploration policy. It only enlarges the training distribution that an off-policy learner already draws from. That thinness is why it composes cleanly with algorithms such as DDPG in the original work: the learner sees more positive samples; the update rule itself is unchanged.
A later line of work, Temporal Difference Models, takes a related stance on multi-goal data and connects model-free deep RL to model-based control. HER itself stays model-free. Its contribution is the observation that, once goals are arguments to the reward, every trajectory is informative about some goal—even the ones that looked like pure failure under the goal that was asked for.
Why it mattered then
At the time, multi-goal continuous control with sparse rewards was a practical bottleneck. Shaped rewards required careful design and often biased the policy toward the wrong behaviour. Pure sparse rewards left standard off-policy methods without gradient signal for long stretches of training. HER gave a general, goal-agnostic way to extract learning signal from the failures that dominate early exploration, without hand-crafting a dense reward for each new task. That mattered for robotic manipulation benchmarks where specifying “success” is easy and specifying a smooth reward is not.
Why it matters now
Sparse multi-goal problems have not gone away. Goal-conditioned policies, offline RL from mixed-quality logs, and any setting where success is rare relative to the state space still face the same starvation of positive examples. Relabeling achieved outcomes as intended goals remains a standard tool in that toolkit, and the broader habit—ask what a trajectory teaches if the objective is rewritten—shows up in hindsight goal generation, reverse curriculum methods, and some forms of goal-conditioned imitation. The original paper’s constraint also still applies: the trick needs an off-policy learner and a reward that can be recomputed from state and goal alone.
The surprising detail
The algorithm does not need the failed episode to be “almost” successful. Even a trajectory that wandered nowhere near the requested goal is treated as a perfect success for whatever state it did reach. The cleverness is not in judging near-misses; it is in refusing to treat the original goal label as sacred once the data are already collected.
What is disputed
How many hindsight goals to sample per episode, and whether to draw them from future states, final states, or the whole buffer, are design choices. The original work compares strategies; later practice often defaults to the future strategy, but the papers do not establish a single universally optimal rule across domains.
Remember this
HER keeps failed rollouts by rewriting the goal to an outcome that actually occurred, turning sparse misses into training signal for an off-policy learner.
Test yourself
A HER agent is trained with a sparse reward that is 0 only when the achieved state matches the goal and −1 otherwise. After an episode that never hit the commanded goal, which transitions enter the replay buffer, and which of them can carry a non-negative reward?
Both the original transitions (labelled with the commanded goal) and one or more relabelled copies (labelled with goals sampled from states actually visited, often future states in the same episode). Under the original goal the rewards stay negative if the goal was never reached. Under a hindsight goal taken from a visited state, transitions that lead to that state can receive reward 0—the success signal the buffer would otherwise lack. The behaviour policy need not have intended that goal; the off-policy learner only needs the recomputed tuples.
Go deeper
- [1707.01495] Hindsight Experience Replay · arxiv.org
- [1802.09081] Temporal Difference Models: Model-Free Deep RL for Model-Based Control · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.