MolmoPoint improves how AI points to objects
Ai2 introduces MolmoPoint, an open family of models that points to objects by selecting parts of a visual representation instead of writing coordinates. The system uses fewer tokens and improves results across images, interfaces, video, and tracking.
MolmoPoint changes how AI models point to objects inside images, videos, and screens. Instead of writing coordinates as text, the model directly selects the visual parts that contain the answer.
The proposal comes from Ai2, the Allen Institute for AI, which is releasing three open models, their code, and their training data. The goal is to make AI do more than describe what it sees. It should also indicate exactly where it is.
From writing coordinates to selecting parts of an image
Most models that point to an object convert its position into numbers or tokens representing coordinate ranges. It is a functional solution, but it forces the model to learn an artificial system and can fail when the image resolution changes.
MolmoPoint uses three special tokens: <PATCH>, <SUBPATCH>, and <LOCATION>. First, it selects a broad area of the image or video. Then it narrows that down to a more precise subarea and finally chooses a specific location.
The process is closer to how a person searches for something visually: first they look at the general area, then zoom in, and finally point to the exact spot. Each point requires 3 tokens instead of 8, reducing the amount of information the model has to generate.
The system also includes a signal to indicate that no relevant objects remain, along with mechanisms for maintaining a consistent order when pointing to several elements. This prevents it from repeating points or continuing to point after the task is complete.
Three models and new training data
Ai2 is introducing three versions of MolmoPoint:
MolmoPoint-8B, for general tasks involving images and videos.MolmoPoint-GUI-8B, specialized in applications, websites, and other interfaces.MolmoPoint-Vid-4B, optimized for understanding videos.
To train the interface model, the team created MolmoPoint-GUISyn, a dataset of 36,000 high-resolution screenshots from desktop, mobile, and web environments. The screens were generated with HTML and analyzed with Playwright to identify every visible element.
The result includes more than 2 million annotated points, about 54 per image on average. They also published MolmoPoint-TrackData, with new trajectories marked by people and others generated artificially to cover partially hidden objects and complex movements.
Better results on images, interfaces, and video
On PointBench, a test that measures localization, spatial reasoning, and recognition of possible uses for objects, MolmoPoint-8B reaches 70.7% average accuracy, compared with 68.7% for Molmo 2 (8B).
On PixMo-Points, it achieves an F1 score of 89.2, compared with 85.2 for the previous model. The improvements are especially visible in reasoning and spatial localization tasks, with gains of about five points.
On graphical interfaces, MolmoPoint-GUI-8B scores 61.1 on ScreenSpot-Pro and 70.0 on OSWorldG. Ai2 ranks it as the best-performing fully open model on those tests. In a controlled comparison with Molmo 2 trained on the same data, the new method achieves an advantage of between 2 and 9 points.
In video, MolmoPoint-8B improves counting tasks and was preferred by human evaluators 59.1% of the time, excluding ties. MolmoPoint-Vid-4B reaches a final accuracy of 58.7 on Molmo2-VideoCount, which asks the model to count objects by pointing to each one across several frames.
For object tracking, the model achieves state-of-the-art results on MeViS and improves by 5.7 J&F points over Molmo2-Track. This metric combines localization accuracy with tracking quality across the video.
What changes for you
The ability to point matters for tasks that are already reaching real products:
- An agent that presses the right button in an application.
- A robot that identifies where to grasp a cup.
- A system that counts people or objects in a video.
- An AI that shows which part of an image supports its answer.
The method also appears easier to train. With just 8,192 examples, MolmoPoint beats the coordinate-based equivalent by about 20 F1 points, and during full training it reaches its best performance sooner.
The central idea is simple: if the model already represents an image using visual tokens, it may be more natural for it to select those tokens than to learn to write numbers. What remains to watch is whether this open approach is adopted in computer agents, robotics, and video analysis beyond laboratory tests.