MolmoBot trains robots using simulated data alone

The MolmoBot robot learns to pick up objects, open drawers, and manipulate doors without ever seeing real human demonstrations. Its creators trained it exclusively on data generated in simulation and say it can transfer those skills to physical robots without additional adjustments.

The project, developed by the Allen Institute for AI, tests an idea that has so far been difficult to support: that a robot can learn manipulation directly in virtual worlds and then work in real environments. The key is not to create a single simulation, but millions of scenarios with different objects, cameras, lighting conditions, and physical properties.

What is MolmoBot

MolmoBot is a suite of models for robotic manipulation. It includes policies, training data, code for generating that data, and the tools needed to reproduce the process.

The suite works with two different physical platforms:

• Rainbow Robotics RB-Y1, a mobile robot capable of moving around and manipulating objects.
• Franka FR3, a desktop robotic arm used for precision tasks.

The models can receive natural-language instructions, such as “pick up the object and place it in the tray,” or point-based commands, such as indicating where to pick up, place, or close something.

MolmoBot focuses on manipulation, not navigation. Its tasks include picking up and placing objects, opening and closing drawers or microwaves, and opening doors by pushing or pulling.

Millions of virtual trajectories

Training uses MolmoBot-Data, a dataset of millions of expert trajectories generated in simulation. A trajectory is the complete sequence of movements that enables a robot to finish a task.

To reduce the gap between what the robot sees in a simulation and what it encounters in the real world, the system constantly changes its training conditions:

• Object position and type.
• Camera angle and movement.
• Lighting, textures, and backgrounds.
• Physical properties and object placement.
• Types of environments and surfaces.

This process is known as domain randomization: exposing the model to many variations so it does not memorize a specific scene. Although the simulation can also produce depth and other internal data, MolmoBot learns its policies using RGB images, like those captured by conventional cameras.

How it performs against models trained with humans

The team evaluated MolmoBot both in simulation and on real robots. During testing, it also changed elements the model had not seen during training, such as the camera, lighting, and rendering engine.

According to their evaluations, the best model achieves direct sim-to-real transfer on both robots, with unseen objects and environments and without fine-tuning in the real world. In pick-and-place tests, it outperforms π0.5, a model trained on large quantities of real demonstrations. The results also compare competitively with methods such as π0 and π0.5 under similar evaluation protocols.

The suite includes three architectures, meaning three different model designs:

• MolmoBot, based on the Molmo2 vision-language model, with the best performance in the evaluation.
• MolmoBot-SPOC, a lighter version that requires less computing capacity.
• MolmoBot-Pi0, based on PaliGemma and designed to directly compare the effect of using synthetic data versus real data.

What changes for robotics

Training robots with people is expensive and slow. Earlier projects have collected hundreds of hours of teleoperation or more than one million real trajectories to teach models how to act. Much of that data is also not publicly available.

With an approach like MolmoBot, a lab could generate new tasks and scenarios using computers, without building a teleoperation infrastructure from scratch. That lowers the cost of experimentation and makes it easier for other teams to test what works and where it fails.

The problem does not disappear. It moves somewhere else. Instead of manually collecting every demonstration, researchers must build simulations that better represent the contact, weight, friction, and irregularities of real objects. A drawer that sticks or a cup that slips remains difficult to model.

MolmoBot does not show that simulation can solve all of robotics, but it does offer an open and reproducible demonstration that it can cover varied manipulation tasks without real data during training. The next step will be to find out how far that transfer reaches as complexity, uncertainty, and physical errors in the real world increase.