WebChain (CVPR 2026, First Author): Why GUI Agents Still Haven't Truly Learned to Use a Browser

The agent field has developed rapidly over the past two years. New computer-use agents keep appearing, seemingly capable of complex tasks: writing code, calling tools, automating workflows, and even controlling operating systems.

But put these systems in a browser environment, and a clear pattern emerges: most agents are not interacting with websites through the GUI.

They usually do something else: call an API directly, or parse the page's text and process it with program logic. In other words, they are not really using a browser the way a person does.

The gap between these two capabilities is substantial.

When people browse the web, we see a visual interface: buttons, menus, input fields, pop-ups, and layouts. We complete tasks by clicking, scrolling, and typing. Many existing agents instead read the page's underlying structure, such as the DOM, or work through APIs, then use program logic to determine the next action.

From an engineering perspective, that approach is certainly simpler. But building a GUI agent that can truly operate web interfaces is a very different problem.

Browser interfaces are highly visual. Elements change dynamically, page structures are often inconsistent, and some information exists only at the visual level. GUI agents therefore face a combined problem of visual understanding, action selection, and structural alignment. What this direction currently lacks most is not models, but data.

Why Is Browser Interaction Data So Scarce?

A look at the sources of AI training data reveals a stark contrast. Text is almost limitless: websites, forums, books, and code repositories produce new material every day. Images are similarly abundant, with enormous collections on social media, image-sharing sites, and video platforms.

Browser interaction traces are another matter entirely.

Real human web interactions are rarely recorded systematically. A person completing a task in a browser performs a sequence of actions: clicking, typing, scrolling, and navigating. Those actions occur in the GUI while also corresponding to specific elements in the DOM. Recording the full process requires capturing screenshots, DOM structure, user actions, and changes in page state together.

This is not especially difficult technically, but in practice, few people record such data over extended periods. The reason is straightforward: it is hard to obtain, and there is no ready-made source at scale. Text and images already exist on the internet; browser interaction traces must be actively collected.

As a result, open-source GUI agent datasets have remained small. Many benchmarks contain only a few thousand trajectories, or fewer. That is nowhere near enough for training large models, and the gap becomes particularly clear when compared with the scale of NLP and computer vision datasets.

This was the starting point for WebChain: continued progress in GUI agents requires addressing data scale first.

What Does WebChain Aim to Solve?

WebChain's core goal is to build a dataset of human interactions in real web environments.

We focused on two aspects of data collection. The first is realism: the data comes from actual websites rather than artificially constructed simulations, with tasks spanning e-commerce, online tools, information sites, and more. The second is cross-modal alignment. Browser interactions involve visual interfaces, DOM structure, and action sequences; every interaction needs precise correspondences between these sources of information.

In WebChain, each step includes a screenshot, DOM element information, the action performed by a person, and the specific element targeted by that action. During training, the model can therefore see the interface, understand its structure, and learn patterns of human behavior together.

The resulting dataset contains more than 30,000 complete task trajectories and over 300,000 interaction steps. This places it among the larger publicly available GUI agent datasets. More importantly, these trajectories come from real web tasks, rather than workflows generated by automated scripts.

Some Thoughts on Training GUI Agents

It is tempting to treat this as a straightforward imitation-learning problem: give the model a trajectory and teach it to predict the next action.

But browser tasks require two distinct capabilities. One is interface understanding: mapping screenshots to DOM elements, locating buttons, identifying which text belongs to which element, and determining what is interactive. This is closely related to visual grounding. The other is task strategy: browser tasks often involve searching, filtering, opening detail pages, filling out forms, and submitting requests. The model must understand the overall objective and plan a sequence of actions.

Training both capabilities together can leave the model struggling early on. It has neither learned to recognize interface elements nor developed a task strategy.

We therefore prefer a staged approach. Early training emphasizes UI grounding: learning to align the visual interface with the DOM and locate actionable elements. Once that capability is reasonably stable, the model learns from complete task trajectories. We have found this approach more stable than direct imitation learning.

Another interesting observation is that model size is not the most decisive factor. Many GUI agent failures happen because the training data is too narrow, rather than because the model is too small. Even a modest change in task type can cause the model to fail.

This is why data diversity matters so much. The web is highly heterogeneous, and interface styles vary considerably between websites. A model trained on only a handful of sites will struggle to generalize to new pages. Data scale and website coverage are therefore often crucial.

One Possible Direction for GUI Agents

Over the longer term, agent systems may increasingly develop along two lines.

API-based agents call tools or service interfaces directly, offering efficiency and greater stability. GUI agents operate software or websites through visual interfaces, more closely resembling human interaction.

These approaches serve different scenarios. Many real systems have no standard API, and some tasks can only be completed through the GUI. That makes GUI agents an important research direction.

The field is still at an early stage. Data infrastructure is a larger bottleneck than innovation in model architecture. WebChain is one attempt to provide a larger starting point for this work.

If more researchers begin recording and sharing real web interaction trajectories, we may eventually see a substantial leap in what GUI agents can do.

Paper

WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces

Conference: CVPR 2026 — First Author