MolmoWeb opens up an AI agent for web tasks

Allen Institute for AI has introduced MolmoWeb, an AI agent that can browse web pages and carry out tasks in a browser. The system can click, type, scroll, open tabs and follow instructions using only screenshots, much like a person working at a screen.

The project includes open models with 4 billion and 8 billion parameters, along with the code, training data, evaluation tools and ready-to-run models. The goal is to let researchers and developers run it on their own machines or in the cloud, without depending on a proprietary service.

From looking at images to acting on the web

MolmoWeb works in a simple loop: it observes the screen, decides what to do next and carries out an action. It then looks at the result again and continues until it completes the task.

For example, you can ask it to find the cheapest direct flights between two cities. The agent identifies the search tool, enters the cities, interprets the results, applies filters and opens the relevant option. Every action is recorded so you can review what it did and stop it if necessary.

Unlike agents that depend on a page's internal structure, such as its HTML code or an accessibility tree, MolmoWeb works with screenshots. That reduces the amount of information it needs to process and lets the system see the same interface you see.

It can also operate on sites that do not offer a specific API, meaning a connection designed to let other programs access their functions directly. In theory, this makes it easier to automate searches, forms, product filters and workflows across several pages.

An open kit for building agents

The release includes MolmoWebMix, a dataset created to train multimodal web agents. It combines examples completed by people, automatically generated trajectories and data that teaches the model to locate elements on a screen.

Its main components include:

• 30,000 human web-task trajectories, with more than 590,000 subtasks demonstrations across more than 1,100 sites.
• Synthetic trajectories generated by agents that explore pages, break tasks into steps and check whether they were completed correctly.
• More than 2.2 million question-and-answer pairs about screenshots from nearly 400 websites.

According to Allen Institute for AI, the data was not obtained by copying the behavior of proprietary visual agents. The synthetic trajectories come from text-based agents that use accessibility trees, while the rest includes human demonstrations and data focused on visual perception.

The package also includes a library for running the system, tools for collecting new interactions and a training and evaluation pipeline. That lets you modify every part of the process, from data collection to final deployment.

What results does it achieve?

In tests conducted by its creators, the 8 billion parameter version scored 78.2% on WebVoyager, a benchmark that measures general browsing, 42.3% on DeepShop, focused on shopping, and 49.5% on WebTailBench, which evaluates instruction following.

Allen Institute for AI says MolmoWeb outperformed other open-weight agents, such as Fara-7B, on all four benchmarks evaluated. On DeepShop, even the 4 billion parameter model beat Fara-7B with the same step limit and also achieved better results using only 30 steps compared with the 100 allowed to its rival.

Reliability improves when you run several versions of the same attempt and keep the best result. With four runs, the 8 billion parameter model reached 94.7% pass@4 on WebVoyager, compared with 78.2% from a single run. On Online-Mind2Web, it rose from 35.3% to 60.5%.

That comes at a cost: more runs mean more time and computing capacity. It is not a guarantee that the agent will always get the answer right.

It still needs supervision

MolmoWeb can misread text in a screenshot, lose information if it scrolls too soon or go off course after an incorrect action. It also struggles with ambiguous instructions, tasks with many conditions, scrolling within specific elements and drag-and-drop operations.

The model was not trained to log in or carry out financial transactions. The demo hosted by its creators includes additional controls: it allows only certain sites, blocks password and card fields and rejects some queries considered dangerous. Those protections belong to the demo, not to the model itself.

For you, the significance is that browser automation is no longer limited to a feature offered by a few closed platforms. An open agent could collect information periodically, complete repetitive tasks or chain together several steps in a tool adapted to your needs.

But the more capable it is of acting on the web, the more important it becomes to limit what it can do without authorization. The next challenge is not just making these agents browse better, but making sure they know when to stop, respect privacy and avoid irreversible actions.