Molmo takes visual AI from pointing to browsing

Molmo, Ai2’s family of visual AI models, is moving toward systems that not only understand images but can also act on them. Its new tools can point to objects precisely and operate web pages using screenshots alone.

The project is built around openness. Ai2 publishes the model weights, data and code, allowing researchers and developers to inspect them, adapt them and train their own versions without relying on a closed system.

MolmoPoint teaches AI where to look

Pointing may seem simple, but it is essential for an AI that needs to count objects, control a robot or click the right button on a screen. It also makes it possible to check what the model is actually seeing.

Models usually indicate a position by writing coordinates, such as a point defined by X and Y values. That method can fail, especially in large images or interfaces crowded with small elements.

MolmoPoint, introduced in March, takes a different approach: it directly selects an area of what it is viewing and then narrows it down to the exact point. According to Ai2, this technique works better on high-resolution images, crowded interfaces and video object-tracking tasks.

The model delivers the best results among open models of comparable size across several tests involving pointing, identifying screen elements and tracking objects, according to Ai2’s evaluation. The organization also says it needs fewer task-specific adjustments during training.

That could reduce the cost of building models capable of locating elements. For example, a company could adapt the system to identify buttons in an application, while a lab could train it to track an animal’s movement in a video without retraining the model from scratch.

Ai2 has also published variants for images, video, software interfaces and tracking, along with datasets containing thousands of annotated screenshots and object trajectories marked by people.

MolmoWeb uses screenshots to operate web pages

The second piece is MolmoWeb, a family of agents that can browse websites and complete tasks based on an instruction and a screenshot.

Instead of reading a page’s HTML code or accessibility trees directly, the agent observes the interface as a person would. It then decides what to do next: move the cursor, click, type or scroll.

“It is perception through screenshots and manipulation through the mouse and keyboard,” explains Tanmay Gupta, who leads MolmoWeb.

This approach has a practical advantage: screenshots can remain useful even when a website’s internal structure changes. Ai2 also argues that an image can replace processing thousands of lines of a page’s code, although real-world performance depends on the complexity of each task.

In the tests cited by Ai2, MolmoWeb outperforms comparable open models on several web-navigation benchmarks. The most capable version also outperforms agents based on larger proprietary models, such as GPT-4o, despite having fewer parameters and working only with screenshots. These are evaluation results, not a guarantee that the agent can complete every task on the internet.

Ai2 has released the models, training data and a common system for evaluating web agents. According to the organization, the dataset includes the largest public collection of human demonstrations of tasks on the web.

What this changes for you

The technology still needs supervision. In a web agent, an error at the start of a task can affect every step that follows. Buying something, filling out a form or changing an account also requires safeguards against incorrect actions.

But the approach points to a more direct way to automate digital tasks. Instead of creating a specific integration for every service, you could describe what you need and let an agent interact with the page as you would:

• Search for information across several sites.
• Complete repetitive tasks in an application.
• Identify specific elements in images or videos.
• Control software that does not offer an API, meaning a channel designed for other programs to operate it.

MolmoPoint and MolmoWeb are part of a broader ecosystem that includes open tools for robotics, 3D perception and augmented reality. Ai2’s bet is that these components can be combined and adapted in labs, companies and independent projects.

The important point is not that Molmo can already perform every task autonomously. It is that visual AI is beginning to move from describing what appears on a screen to pointing to it and acting on it. The next challenge will be making those actions accurate, safe and reliable enough for everyday work.