Vision-Language-Action Models for Robotics

Robot learning has an awkward bottleneck: robots produce their own training data, and there is never much of it. A language model can learn from an almost unlimited supply of text and images scraped from the web. A robot arm learns only from the hours it has physically spent moving, usually with a person guiding it through demonstrations. For years the practical consequence was that every new object, and every new way of phrasing an instruction, tended to mean another round of data collection in a lab.

RT-2, described in a 2023 paper, starts from the other end. Its authors began with a vision-language model — a network already trained on web images and their accompanying text, so that it can look at a photograph and answer questions about it. They worked with two such model families, PaLI-X and PaLM-E. A model like this has some notion of what a banana looks like, what a flag is, and what people usually mean when they call something rubbish. None of that knowledge came from a robot. The question was how to connect it to an arm.

The answer is a choice about how to write an action down. At every control step — each moment the robot needs its next command — the arm requires eight quantities: a flag saying whether the task is finished, three numbers for how far to move the gripper through space, three for how to rotate it, and one for how wide to open or close it. These are naturally continuous values, ordinary decimals. RT-2 does not output decimals. Each quantity's possible range is chopped into 256 evenly spaced bins, and the value is replaced by the number of the bin it falls into — a whole number between 0 and 255. The eight bin numbers are then written out, separated by spaces, as a short string of text.

That string is the point. A language model already produces its answers one text token at a time, whatever the question. If the action is a string, the model can produce the action with exactly the machinery it already has; nothing about its architecture had to be redesigned. The output does have to be checked — a valid answer is precisely eight integers, so the decoded string is parsed before use — and then a short piece of code on the robot converts the bin numbers back into a movement command. The arm moves, the camera takes a fresh image, and the cycle repeats.

This is the design the paper's name for the category describes: a vision-language-action model, or VLA. One network takes in a camera image and a plain-language instruction and puts out an action, with no separately built perception module, no separate planner, and no parser turning the instruction into symbols for other components to look up. For years the standard robot system had been exactly that kind of modular stack, each part trained or hand-built on its own. Here the whole path from pixels and words to motor command runs through a single model.

Training is where a subtle problem appears. The obvious recipe — take the web-trained model, then fine-tune it on robot demonstrations alone — quietly destroys the thing you started with. Fine-tuning pulls a model towards whatever data it currently sees. Show it nothing but robot trajectories and it drifts towards being a robot-trajectory model, gradually losing the broad visual and linguistic knowledge that made a web-trained starting point worth having. RT-2's answer is to mix rather than sequence: robot episodes are interleaved in the same training batches with the original web tasks, describing images and answering questions about them. The authors call this co-fine-tuning, and they report that it matters — a model trained on robot data alone generalises less well to objects and phrasings it never saw during robot training. The web knowledge is not a foundation laid once. It has to be actively maintained while the robot skill is added.

What does the web knowledge actually buy, once preserved? Mostly comprehension. The model can be told to pick up an object that never appeared in any robot demonstration and still find it in the camera image, because it learned what that object looks like from the web. It can follow an instruction phrased in a way no demonstration used. It can handle requests with a small amount of reasoning folded in — choosing among the visible objects on the basis of some property, rather than matching a fixed template. What it does not gain is new physical skill. A motion the arm never practised does not appear because the model has read about it. The split is worth stating plainly: the web supplies recognition and vocabulary; the demonstrations supply the movement.

Two costs come with the design, and both follow directly from it. The first is precision. The arm can only express positions on the 256-step grid the encoding allows, so fine positioning is bounded by a representational choice made for convenience, not by the policy's competence. The second is speed. A very large model must run once per control step, and an arm wants its next command promptly, so model size and control frequency pull directly against each other. The largest, most capable version of such a system is not automatically the most useful one on real hardware.

A caution belongs at the end. The paper reports evaluations on real robots, but success rates in manipulation depend heavily on which objects, scenes and phrasings are tested, and the field's benchmarks are not standardised enough for easy comparison across papers. What transfers cleanly out of RT-2 is the mechanism — actions written as text, trained in a mixture — rather than any particular number.