MolmoMotion predicts 3D motion for robots

MolmoMotion can anticipate how an object will move in 3D from a single image, several points marked on it, and a written instruction. Allen Institute's model turns phrases such as “move and rotate the wooden bowl” into a future trajectory that a robot or video generator can use.

The key distinction is between seeing motion that has already happened and predicting what comes next. A system watching a video can track a cup while someone lifts it. But a robot needs to know where the cup will move before touching it, so it can plan its grip without bumping into it or losing it.

How it works

MolmoMotion represents each object using several 3D points attached to its surface. It does not try to render a complete video or depend on specific templates for hands, bodies, or cars. These points can describe the motion of rigid and articulated objects and, with some limitations, deformable ones.

The model receives three main inputs:

• An RGB observation, meaning a standard image of the scene.
• Eight points selected on the object and their initial 3D positions.
• A description of the action to be anticipated.

It then identifies the object mentioned in the instruction and predicts where those points will be over the following seconds. This lets it describe actions such as rolling a lint remover over fabric, sliding and rotating a bowl, or following the path of a car taking a turn.

MolmoMotion uses Molmo 2 as its base and comes in two versions. MolmoMotion-AR calculates the trajectory step by step, which is useful when the future is well defined. MolmoMotion-FM generates trajectories in continuous 3D space and can better represent the fact that several possible futures may exist for the same instruction.

One million videos for learning motion

To train it, the team created MolmoMotion-1M, a collection of 3D trajectories associated with actions and objects. The dataset brings together data extracted from 1.16 million videos, covering 736 motion types and 5,600 distinct objects.

The process filters out points that move inconsistently, smooths the trajectories, and trims each video to focus on the moment when the object is actually moving. This matters because ordinary videos do not directly include precise depth or 3D motion annotations.

The team also released PointMotionBench, a benchmark validated by people with 2,700 clips, 111 object categories, and 61 motion types. Its goal is to measure whether the predicted trajectory matches the real motion, not merely whether the result seems reasonable.

What it improves in robots and video

On PointMotionBench, MolmoMotion outperformed the 3D prediction methods compared by the team, including video generators, specialized 3D models, and a baseline that assumes the object maintains a constant velocity.

The improvement also appears in robotics. In simulated pick-and-place tasks, a control policy based on MolmoMotion achieved a 76.3% success rate, compared with 56.0% for the same policy based on Molmo 2. It also reached a 51% success rate after 10,000 training steps, while the earlier version reached 19%.

The model can also guide video generators. Instead of asking the generator to guess the motion from a sentence, it receives a specific 3D trajectory. According to the team's tests, this improves all five motion-related metrics and outperforms a much larger video generator on four of them.

The main limitation is that eight points per object are not enough to represent complex surfaces in detail. As a result, the system still struggles with deformable motion, such as fabric changing shape.

MolmoMotion points to a simple but important idea: for a machine to act in the world, recognizing what is happening is not enough. It must also anticipate what will happen next. Its weights, training data, and benchmark have been released so other teams can evaluate them and adapt them to robots, video, and new physical tasks.