OpenWebRL: Demystifying Online
Multi-turn Reinforcement Learning
for Visual Web Agents

Rui Yang^1*†, Qianhui Wu^2*‡, Yuxi Chen¹, Hao Bai¹, Wenlin Yao², Hao Cheng²,
Baolin Peng², Huan Zhang¹, Tong Zhang¹, Jianfeng Gao²

* Equal contribution · ‡ Project lead

¹ University of Illinois Urbana-Champaign

² Microsoft Research

Paper Code Data Models Developed with Orchard Env

Abstract

Building capable visual web agents requires long-horizon reasoning, precise grounding, and robust interaction with dynamic real-world websites. Despite rapid progress, the strongest systems remain largely proprietary, while open agents still depend heavily on supervised post-training over large collections of curated web trajectories. This dependence creates a major scalability bottleneck: high-quality demonstrations are expensive to collect, and static datasets offer limited coverage of the diverse, ever-changing open web.

We introduce OpenWebRL, an open framework for training visual web agents with online multi-turn RL on real websites. OpenWebRL covers the full training pipeline, including scalable live-browser infrastructure, supervised initialization, multimodal context management, trajectory-level success judging, and efficient multi-turn policy optimization.

Using this framework, we train OpenWebRL-4B, which establishes a new open-source state of the art on challenging live-web benchmarks. With only 0.4K initialization trajectories and 2.2K open-ended RL training tasks, OpenWebRL-4B achieves 67.0% success on Online-Mind2Web and 64.0% on DeepShop, outperforming prior open agents of similar or larger scale and remaining competitive with proprietary systems including OpenAI CUA and Gemini CUA. Beyond strong benchmark performance, we systematically study the key design choices that make online RL effective for visual web agents, and analyze how RL improves agentic reasoning during training.

Framework

Supervised Warm Start

Warm-starts the policy with only 0.4K high-quality trajectories collected from Qwen3-VL-235B, placing it in a productive exploration regime before online RL — eliminating cold-start instability with minimal data.

Agent Harness

A fault-tolerant live-browser environment with 13 atomic browser tools, a multi-tool-call interface, and textual environment feedback — making open-web rollouts reliable and efficient at scale.

Multimodal Context Management

Retains only the most recent screenshot (K=1) for visual grounding while preserving full reasoning traces as long-horizon textual memory — balancing performance and training cost.

Multimodal Multi-turn GRPO

MM-GRPO propagates a single trajectory-level reward to all assistant tokens across all turns, using the same context-management procedure for both rollout and optimization. Combined with asymmetric clipping and trajectory-level dynamic sampling for stable optimization.

OpenWebRL-Judge-8B

A distilled 8B judge trained on 12.5K rollouts achieves 89.8% accuracy / 92.1% F1 against GPT-4.1, matching proprietary quality while cutting ~$545 per training run.

Rollout-Length Curriculum

Trains first with 15-step horizons then extends to 30 steps — short horizons stabilize early exploration, longer horizons enable the policy to handle tasks requiring extended interactions.

OpenWebRL's infrastructure supports both lightweight local-process execution and isolated sandbox execution. The sandbox setup uses Orchard Env, a Kubernetes-native service that provides sandbox lifecycle management, command execution, file I/O, network policy, and a REST API, enabling hundreds of concurrent sandboxed browser environments in our setup. In Online-Mind2Web, running with the Orchard sandbox reduced the blocked-trajectory rate from 25.7% with local processes to 17.7%. This reliability advantage is expected to be even more important for online RL, where group-size rollouts amplify repeated access to the same websites.

Main Results

Official success rates (%) on three live-web benchmarks. * from FARA · † from MolmoWeb · Bold = best · Underline = second best.

System	# Steps	# Tasks	WebVoyager	Online-Mind2Web	DeepShop	Average
Proprietary Models
GPT-5 (Axtree) †	30	—	70.6	41.9	40.7	51.1
Gemini-3-Flash (Axtree) †	30	—	74.4	34.8	45.1	51.4
Gemini-3-Flash (Axtree) †	100	—	85.6	44.8	55.3	61.9
GPT-4o (SoM) *	100	—	65.1	34.6	16.0	38.6
o3 (SoM) *	100	—	79.3	55.4	49.7	61.5
GPT-5 (SoM) *	100	—	90.6	57.7	49.1	65.8
OpenAI CUA *	100	—	70.9	58.3	24.7	51.3
Gemini CUA †	100	—	88.6	57.3	62.0	69.3
Open-Source Models
Holo1-7B †	30	>15.6K	55.4	—	—	—
UI-TARS-1.5-7B *	100	—	66.4	31.3	11.6	36.4
GLM-4.1V-9B-Thinking *	100	—	66.8	33.9	32.0	44.2
FARA-7B *	100	>123.2K	73.5	34.1	26.2	44.6
MolmoWeb-4B †	100	>278.5K	75.2	31.3	35.6	47.4
MolmoWeb-8B †	100	>278.5K	78.2	35.3	42.3	51.9
Qwen3-VL-4B-Thinking	30	—	52.6	32.0	33.3	39.3
Qwen3-VL-235B-A22B-Thinking	30	—	66.4	63.7	56.7	62.3
Ours (Qwen3-VL-4B Backbone)
OpenWebRL-4B-SFT	30	0.4K	60.2	47.0	48.7	52.0
OpenWebRL-4B	30	2.2K	74.1	67.0	64.0	68.4
OpenWebRL-4B w/ Judge-8B	30	2.2K	68.9	67.3	68.7	68.3

Learning Dynamics

Does supervised warm-start help online RL?

MM-GRPO training curves: SFT vs base initialization

Comparison of MM-GRPO training from SFT and base-model initializations. (a) Average training reward. (b) Average response length. (c) Evaluation success rate (excl. aborted). (d) Success rate improvement by difficulty split on Online-Mind2Web.

Finding: SFT initialization maintains a consistent ~10% advantage throughout training and yields much larger gains on hard tasks (+22.3 pts vs. +2.3 pts from base init). Warm-starting places the policy in a better region of behavior space for effective online exploration.

How does RL reshape the agent's reasoning patterns?

Response length and reasoning pattern analysis during MM-GRPO training

Diagnosing response-length growth during MM-GRPO. (a) Trajectory length and average interaction steps. (b) Step-level proxy presence rates. (c) Average step-level response length conditioned on reasoning pattern type.

Finding: MM-GRPO increases both the frequency and the length of four recurring reasoning patterns: history summarization (presence rate 14.5%→21.4%, length 332→542 tokens), blocker diagnosis (14.2%→23.7%, 273→440 tokens), retry-plan reasoning, and condition-proof reasoning. By contrast, non-proxy steps remain nearly stable (282→325 tokens). The model learns when to reason deeply — allocating additional verbosity selectively to steps that matter, rather than simply producing longer outputs everywhere.

Ablations & Analysis

Does the judge model matter for RL stability?

RL training curves with GPT-4.1, OpenWebRL-Judge-8B, and the base VLM as judge.

Finding: The distilled OpenWebRL-Judge-8B matches GPT-4.1 training dynamics closely, while a naive base VLM judge causes reward hacking — high training reward but collapsing eval performance. Reliable reward signals are critical for stable online RL.

How much does the quality of SFT initialization matter?

MM-GRPO from different SFT initializations on Online-Mind2Web.

Finding: More SFT data or longer imitation training does not automatically yield better RL performance. The 0.4K/3-epoch default outperforms both a lightweight 0.4K/1-epoch init and a larger 1.9K/3-epoch init. Heavier imitation training on a less curated distribution may reduce policy plasticity and limit RL's room to improve. The key is a balanced warm start: strong enough for productive exploration, but not over-imitated.

What causes failures on live websites?

Distribution of failure modes based on manual inspection of 100 failed trajectories sampled from failed cases on Online-Mind2Web, evaluated without Browser-Use Stealth Browser service (raw open-web setting).

Finding: Among failed trajectories, 51% are attributable to live-web instability (access restrictions, CAPTCHAs, loading failures) — outside model control. Note that this analysis is conducted without Browser-Use Stealth Browsers and samples only from failures, so access-related issues are more prevalent than in the overall or official evaluation distribution. 27% reflect reasoning/knowledge limits, 13% visual grounding errors, and 9% task definition or judge issues. Further progress requires both stronger models and more robust infrastructure.

OpenWebRL: Demystifying Online
Multi-turn Reinforcement Learning
for Visual Web Agents

Performance Overview

Abstract