MolmoAct: the open AI model that reasons in 3D for robots

Ai2 has introduced MolmoAct, an artificial intelligence model that plans movements in three-dimensional space before telling a robot what to do. The project aims to address a common limitation: current models understand instructions and images, but often struggle to calculate distances, trajectories, and forces.

MolmoAct belongs to a new category created by Ai2: Action Reasoning Models, or models that reason about actions. According to the institute, it is the first model of its kind and is available as open source, along with its data, evaluations, and training tools.

How it turns an instruction into movement

Instead of relying only on words, MolmoAct builds a spatial representation of the scene. It uses depth information, meaning an estimate of the distance between objects and the camera, to better understand the geometry of its surroundings.

Its process has three steps:

• Understand the physical space: identifies objects, positions, and distances using perception tokens, pieces of information that encode the scene's geometric structure.
• Plan a trajectory: draws a sequence of intermediate points over the image to show how the task should unfold.
• Execute the action: turns those points into concrete commands for components such as grippers, robotic arms, or joints.

Planning happens first in image space and is then adapted to the type of robot being used. That allows the same model to be adapted with little additional training to humanoid robots or arms with grippers.

For example, when given the instruction "put the clothes in the closet," MolmoAct does not simply associate words with movements. It can identify the clothes, calculate where the closet is, map out a route to pick up a garment, and translate it into movements for the robot's arm.

Results and efficiency

Ai2 says MolmoAct-7B, the first version in the family, achieved a 72.1% success rate on out-of-distribution tasks in SimplerEnv, a set of simulated environments for testing robotic manipulation. That result surpassed models from Physical Intelligence, Google, Microsoft, NVIDIA, and other labs in that evaluation.

In LIBERO, another simulation environment focused on learning multiple tasks, it achieved an average success rate of 86.6% through parameter-efficient fine-tuning. Ai2 also compared the model in real-world tests with OpenVLA and π0 Fast, using the same demonstrations while changing the instructions, objects, and conditions. MolmoAct recorded the highest success rates across all the tested variants, according to the institute.

The model was also trained with far fewer resources than some comparable systems. Its pretraining used 26.3 million samples and 256 NVIDIA H100 GPUs, and finished in approximately one day. The subsequent fine-tuning required 64 H100s for about two hours.

For comparison, Ai2 notes that NVIDIA's GR00T-N1-2B used 600 million samples and 1,024 H100s, while Physical Intelligence trained π0 with 900 million samples and an undisclosed number of chips. These figures describe the training processes and do not by themselves guarantee better performance on every robot or in every environment.

A model that shows you its plan

One of MolmoAct's most practical features is that it displays its planned trajectory over the image before sending control commands. If the robot is going to move a cup toward a table, you can see the route the system intends to follow and correct it before it acts.

You can also guide it by drawing a route or a target pose on a phone, tablet, or computer. This visual input can indicate, for example, which side an object should be approached from or where a gripper should be placed.

For robotics, this matters because it makes it easier to spot errors and control unwanted behavior. It does not make the system infallible, but it does offer a more understandable way to review what it plans to do.

What the open version includes

MolmoAct-7B was trained with a selection from Open-X Embodiment, a dataset of robotic demonstrations, and a multimodal reasoning dataset. It then received fine-tuning with nearly 10,000 robot episodes in household environments, such as arranging cushions or putting away clothes.

Ai2 is also publishing that fine-tuning dataset, the training code, the model checkpoints, the evaluations, and the reference tools. Unlike many models that only provide their weights, this allows other teams to inspect how it was trained, modify it, and verify its results.

For you, the immediate effect will not be having a household robot capable of performing these tasks. MolmoAct still needs compatible hardware, demonstrations, and adjustments for each environment. What matters is that researchers and manufacturers can start with an open model that combines spatial perception, planning, and control in a single architecture.

The next step will be finding out how much of these simulated results holds up in homes, factories, and changing environments. The question is not only whether a robot can follow a route, but whether it can do so safely when new objects, ambiguous instructions, or moving people appear.