Vision-Language Navigation

Vision-and-Language Navigation asks a single question with awkward precision: can an agent that sees only what a camera would see, and that moves only along a pre-defined graph of viewpoints, execute a free-form English route description in a real building?

The Room-to-Room task, introduced on top of Matterport3D scans, makes that question concrete. Each environment is a set of panoramic images captured at discrete positions indoors. Between some of those positions there are edges; the agent may step along an edge or issue a stop action. A human annotator has already walked a path and written an instruction that would get another person from the start to the goal — typically a short paragraph naming landmarks, turns and intermediate rooms. The agent receives the instruction and the starting panorama. It does not receive a map, a floor plan, or the goal coordinates.

That design forces joint reasoning. The words “the glass doors past the seating area” only become useful once the model can find glass doors and seating in the current view, decide which neighbouring viewpoint lies beyond them, and keep track of how much of the instruction has already been consumed. Early sequence-to-sequence baselines encoded the full instruction, attended over it while observing each panorama, and predicted the next action. They established that the task was learnable, and also that performance collapsed when the test buildings were truly unseen — a gap that later work treated as the central problem rather than a side effect.

Speaker–follower models attack the same setting from two directions at once. The follower is the navigator. The speaker is trained the other way round: given a route, produce an instruction. Because routes can be sampled automatically on the navigation graph, the speaker becomes a machine for minting extra training pairs. The follower trained on that mixture sees more linguistic variety and more visual contexts than the human annotations alone provide. At test time the two models can cooperate again. Several follower trajectories are proposed; the speaker scores how well each trajectory matches the original instruction; the highest-scoring trajectory is retained. The paper frames this as a pragmatic choice — prefer the path a speaker would actually describe with those words — rather than as simple ensemble voting.

Nothing in the setup assumes outdoor maps, GPS, or continuous control. The difficulty is almost entirely about binding phrases to appearance under partial observability, and about doing so in interiors whose layout was never in the training set. That is a narrower problem than full embodied intelligence, and a clearer one: the inputs, the action space and the success criterion are all fixed, so progress and failure modes can be compared directly.