MolmoMotion uses AI to predict 3D motion

MolmoMotion, a new model from Ai2, predicts how objects will move in 3D from an image, points marked on the object, and a written instruction. For example, it can anticipate the path of a bowl that someone slides and spins across a table.

The difference matters because most current systems are good at describing movements that have already happened in a video. MolmoMotion tries to do the opposite: look at the current state and calculate what will happen next, over the following seconds.

From watching motion to anticipating it

The model does not first generate a complete video. It represents each object with several 3D points attached to its surface and predicts where each one will be over time. This is a more compact way to describe motion than rendering every future frame.

The technique can work with rigid, articulated, and, within certain limits, deformable objects. It also preserves the trajectory when the camera angle changes, because the points are expressed in a shared 3D coordinate system.

The written instruction connects language to action. Phrases such as “move and spin the wooden bowl with fruit” or “roll a roller over a blue fabric” indicate which object to track and what movement to anticipate.

MolmoMotion has two variants:

• MolmoMotion-AR predicts the coordinates one after another. It usually works best when the future is well defined and favors smooth trajectories.
• MolmoMotion-FM generates the trajectory in a continuous 3D space. It is designed for situations where the same instruction allows for several possible futures.

Data for teaching AI to predict

To train the system, Ai2 created MolmoMotion-1M, a collection built from 1.16 million videos. It includes 736 types of motion and 5,600 different objects, according to the team.

Internet videos did not come with ready-to-use 3D annotations. So Ai2 developed an automated process that identifies the object, tracks points on it, converts those positions into 3D coordinates, and removes unreliable trajectories. It also trims each video to focus on the moment when the object is actually moving.

The team also published PointMotionBench, a human-validated benchmark with 2,700 clips, 111 object categories, and 61 types of motion. It compares the model’s prediction with the actual movement instead of judging only whether the result looks plausible.

Results in robots and video

In PointMotionBench tests, MolmoMotion achieved lower error than the methods it was compared with across different scenarios, including objects manipulated indoors, interactions recorded from a person’s perspective, and outdoor scenes.

The idea also improves robotic planning. In a simulation of object-picking and placement tasks, a policy based on MolmoMotion achieved a 76.3% success rate, compared with 56% for the same policy using Molmo 2 as its base. With 10,000 training steps, it reached 51% success, while the previous version remained at 19%.

In video generation, the predicted trajectories guide another model so it knows how an object should move. This is especially useful for small or precise actions that text alone describes ambiguously, such as making a flamingo walk while lowering its beak toward the water.

The system guided by MolmoMotion outperformed the base model CogVideoX-5B on all five motion metrics analyzed. It also achieved better results than the larger video model Wan2.2-I2V-A14B on four of those five metrics.

What it could change for you

The technology is not yet a product for everyday use. Its value lies in serving as an intermediate component for systems that need to act in the physical world: a robot calculating how to move a lid, a camera anticipating a trajectory, or a video generator that follows a specific action more accurately.

There is one important limitation: training uses eight points per object. That is enough to estimate a general trajectory, but not to represent in much detail a surface that bends, stretches, or changes shape in complex ways.

Ai2 has released the model weights, the MolmoMotion-1M dataset, and PointMotionBench so other teams can study and adapt them. The question now is whether this ability to anticipate motion holds up beyond the videos and scenarios used during training, especially when a robot has to deal with new objects and unpredictable situations.