$\textsc{OpenForge RL}$: Train Harness-native Agents in Any Environment

Xiao Yu$^{1}$, Baolin Peng$^{3}$ $^{\dagger}$, Ruize Xu$^{2}$, Hao Zou$^{1}$, Qianhui Wu$^{3}$, Hao Cheng$^{3}$ bfWenlin Yao$^{3}$, bfNikhil Singh$^{2}$, bfZhou Yu$^{1}$ $^{}$, bfJianfeng Gao$^{3}$ $^{}$

$^{1}$Columbia University $^{2}$ Dartmouth College $^{3}$Microsoft Research

[email protected], [email protected]

$^{*}$ Equal Advisory Contribution; $^{\dagger}$ Project Lead

Abstract

Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems. While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference. To address this, we present $\textsc{OpenForge RL}$, an open-source framework for training harness-based agents end-to-end in diverse environments. $\textsc{OpenForge RL}$ achieves this with a lightweight proxy that serves the harness's model calls while recording them as training data for a standard RL codebase (e.g., veRL), and a Kubernetes orchestrator that runs each rollout in its own remote container, together enabling training on any harness in any environment at scale. By decoupling training and inference, $\textsc{OpenForge RL}$ allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with. We validate our framework across diverse, complex harnesses and environments, spanning tool/claw-based agents and multimodal GUI browser- and computer-use agents. Using only hundreds to a few thousand tasks, OpenForge-Claw reaches $31.7$ ($\mathrm{pass}^3$) and $55.9$ ($\mathrm{pass}@3$) on ClawEval and $33.7$ on QwenClawBench. OpenForge-GUI reaches $37.7$ on OSWorld-Verified, $63.0$ on Online-Mind2Web, and $72.3$ on WebVoyager. Both outperform open baselines of similar size on nearly all benchmarks, and in the GUI setting match or surpass models several times larger. Beyond benchmarks, we analyze how harness choice (e.g., ZeroClaw, OpenClaw, Codex) and RL shape agent behavior. We find that some harnesses are substantially harder to learn than others, and that RL improves agentic reliability, such as self-verification, tool coverage, and completing multi-step plans, though critical abilities such as error recovery remain weak. We will release our code, data, and models to facilitate research on harness-based agents.

Executive Summary: OpenForge RL addresses a growing barrier in AI agent development: state-of-the-art agents now depend on complex inference harnesses that manage multi-turn reasoning, tool use, and external systems, yet these harnesses cannot be expressed natively in open reinforcement-learning frameworks. The resulting train-deploy mismatch forces researchers either to simplify harnesses for training or to forgo end-to-end optimization, widening the gap between open models and proprietary systems.

The work introduces an open framework that decouples harness inference from training. A lightweight proxy intercepts model calls from any harness and records them as standard RL trajectories, while a Kubernetes orchestrator launches each rollout in its own remote container on cloud infrastructure. This design lets existing RL codebases such as veRL train agents directly inside the harnesses and environments where they will later be deployed. The authors synthesized a few hundred to a few thousand tasks, performed supervised fine-tuning followed by GRPO reinforcement learning, and evaluated the resulting Claw and GUI agents across six benchmarks.

OpenForge-Claw (30B-A3B) reached 31.7 pass³ and 55.9 pass@3 on ClawEval, 33.7 on QwenClawBench, and 28.1 on MCPAtlas, surpassing open baselines of comparable size. OpenForge-GUI (8B) scored 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager, matching or exceeding models several times larger despite using far less data. Certain harnesses proved substantially harder to learn than others; training on multiple harnesses improved generalization to unseen ones; and reinforcement learning strengthened self-verification, tool coverage, and multi-step completion, although error recovery remained weak.

These outcomes show that modest amounts of targeted interaction data, collected inside real deployment harnesses, can close much of the performance gap for open agents. The approach removes a major infrastructure obstacle, enables systematic study of how harness design affects learnability, and reduces reliance on simplified training environments that diverge from production use.

The authors plan to release code, data, and models. Practitioners should adopt the framework when training agents intended for production harnesses and should allocate additional effort to error-recovery data or algorithms. The main limitations are the modest number of training tasks and the persistent weakness in recovery behavior, both of which warrant further data collection before broad deployment decisions. Overall, the evidence supports using OpenForge RL as a practical foundation for scalable, harness-native agent training.

1. Introduction

Section Summary: Modern AI agents increasingly rely on complex inference harnesses that wrap base models to manage tools, browsers, and multi-step interactions in real-world environments, yet these harnesses are difficult for open research to improve through end-to-end training due to mismatches with existing reinforcement learning tools and the need for separate scaled computing environments. To address this, OpenForge RL introduces a lightweight proxy that decouples any harness from training while automatically converting its interactions into standard data, paired with a cloud-based orchestrator that runs rollouts in remote containers. This enables researchers to train capable agents across tool-use and GUI settings, yielding strong benchmark results while revealing how harness design influences learning.

Modern AI agents are increasingly deployed in complex, open-ended environments, from software engineering ([2, 3, 4]) and tool-use ([5, 6]) to controlling real web browsers and desktops ([7, 8, 9]). To operate effectively in these settings, state-of-the-art agents are rarely a bare language or vision-language model; instead, they are wrapped in increasingly sophisticated inference harnesses: orchestration scaffolds such as Claude Code ([10]), Codex ([11]), and OpenClaw ([12]) that manage multi-turn interaction, tool use, and context while linking the model to external systems such as MCP servers ([13]), browsers, and GUIs. This harness layer is now central to agentic capability: recent progress on agentic benchmarks comes as much from harnesses as from stronger base models ([3, 14]).

While harness-equipped agents are remarkably capable, improving them end-to-end remains out of reach for much of the open research community for two reasons. First, a harness turns inference into a stateful, multi-process procedure, with nested tool calls, subagents, and long-horizon context, that open training stacks ([15, 16, 17]) cannot natively express. Open efforts often need to reimplement a simplified harness for training ([18, 19]), creating a train–deploy mismatch. Second, running harnesses at scale requires dedicated, containerized environments that cannot be co-located on training nodes, yet most open RL frameworks assume rollouts run locally inside the trainer. Together, these gaps leave proprietary harness-based systems increasingly ahead of what the research community can train and study.

To close these gaps, we introduce $\textsc{OpenForge RL}$, an open framework that makes it accessible to train harness-based agents end-to-end. At its core, $\textsc{OpenForge RL}$ addresses these obstacles with two lightweight components (see Figure 1 left or Figure 2). First, a lightweight proxy abstracts the harness's inference process and decouples it from training, so that any harness can run its own arbitrary inference. Paired with automatic trajectory reconstruction, the recorded prompt-response pairs become standard samples compatible with any RL codebase (e.g., veRL ([15])). Second, a Kubernetes orchestrator, following [1], launches each rollout as a remote container on cloud providers such as Microsoft Azure, scaling elastically to many concurrent environments. Together, this reuses the rich harness ecosystem directly, spans environments from tool use to GUIs, and remains agnostic to the underlying RL algorithm.

We validate $\textsc{OpenForge RL}$ across a broad spectrum of complex agentic settings, ranging from daily tool-use and claw-based agents to multimodal GUI browser- and computer-use agents. In the daily tool-use setting, OpenForge-Claw (30B-A3B MoE) trains on 3 popular harnesses (ZeroClaw, OpenClaw, and Codex) in addition to the standard ReACT loop, and reaches $31.7$ ($\mathrm{pass}^3$) and $55.9$ ($\mathrm{pass}@3$) on ClawEval ([20]), $33.7$ on QwenClawBench ([21]), and $28.1$ on MCPAtlas ([5]). In the GUI setting, OpenForge-GUI (8B) trains on (a modified version of) Kimi-Agent ([9]) and Molmo-Web ([22]), and attains $37.7$ on OSWorld-Verified, $63.0$ on Online-Mind2Web, and $72.3$ on WebVoyager, outperforming open baselines of similar scale and matching or surpassing models several times larger. Crucially, because $\textsc{OpenForge RL}$ trains agents in their real deployment harnesses, it also lets us study how harness choice and RL training shape agent behavior, a question prior open work could not easily ask (Section 5): some harnesses prove far harder to learn than others, and RL broadly improves agentic reliability though abilities such as error recovery remain weak.

In summary, our contributions are threefold:

• We introduce $\textsc{OpenForge RL}$, a scalable training infrastructure that flexibly pairs any harness with any environment, connecting real harness ecosystems and remote sandboxes in cloud-service providers to powerful RL codebases (e.g., veRL)
• We conduct a broad empirical study across tool-use/claw and GUI (browser and computer-use) agents, improving over open models of similar size on nearly all benchmarks and, in several cases, over models several times larger.
• Enabled by training in real deployment harnesses, we analyze how harness choice and RL shape agents: simpler, better-aligned harnesses are easier to learn; training generalizes across similar harnesses; and RL overall improves agentic reliability (e.g., self-verification, tool coverage, and task completion), though error recovery remains challenging to learn.

2. Related Work

Section Summary: Recent research on LLM-based agents has emphasized the value of well-designed inference harnesses that let models interact with computers, code, and graphical interfaces, as shown by tools like SWE-Agent and its successors. At the same time, efforts to train such agents end-to-end with reinforcement learning have relied on frameworks that work well only for simple, single-turn or lightweight interactions, forcing researchers to create custom modifications for more realistic, containerized environments. This paper aims to close that gap by developing a general training approach that supports complex harnesses at scale while remaining compatible with existing RL methods and covering a wide range of text and vision tasks.

Agents with inference harnesses.

With the advent of LLM-based coding agents, early work such as SWE-Agent ([3]) showed that a carefully designed agent-computer interface substantially improves task success ([2, 23, 24]). Subsequent harnesses such as Claude Code ([10]) and Codex ([11]) refined this recipe for software engineering, and open-source efforts such as OpenClaw ([12]) extended it to everyday tasks as well as to GUI tasks ([25, 26, 9, 27]). Rather than developing harnesses, we study how to train LLM- and VLM-based agents end-to-end using these harnesses, aligning agent training with real-world usage so that models can learn in the same setting in which they are deployed.

Training harness-based agents.

As agents take on increasingly complex environments, training them end-to-end has become a central goal, with reinforcement learning (RL) emerging as the primary tool. Several open-source RL frameworks, such as veRL ([15]), Slime ([16]), and more ([28, 17, 29]), support advanced algorithms (e.g., PPO and GRPO) and asynchronous distributed training. However, they assume relatively simple rollouts: single-turn generation, or multi-turn interaction with lightweight tool calls such as sandboxed code execution. These assumptions break for complex inference harnesses, which manage multi-turn interaction, tool use, and context internally, and whose rollouts require containerized environments with dedicated CPU and memory that cannot be co-located on the training nodes at scale. As a result, initial attempts training agents in these complex environments requires heavily modifing the training loop to fit specific task case-by-case ([30, 31, 32, 33]) — making the codebase hard to extend and maintain. Our work presents a first step towards designing a training flow that natively supports complex harnesses and environments at scale, while remaining agnostic to the underlying RL framework[^1]

[^1]: Concurrent work such as Polar ([34]) proposes a similar approach but focuses on software-engineering tasks (SWE-Bench-Verified). Our work covers a substantially broader setting, spanning text-based claw tasks and vision-based GUI tasks across six benchmarks, and also further analyzes how harness choice and RL shape agent behavior.

3. Methods

Section Summary: The Methods section models complex agent tasks as Markov Decision Processes in which an LLM agent, wrapped inside a specialized harness of tools and control flows, generates actions from observations and produces training trajectories as collections of harness-processed prompt-response pairs rather than raw histories. It then presents OpenForge RL, a plug-and-play system that links standard reinforcement-learning frameworks to many remote, containerized harness environments by means of a Kubernetes orchestrator and an inference proxy that intercepts calls, records data, and reconstructs complete trajectories with discounted terminal rewards. The approach also introduces practical mechanisms for managing asynchronous rollouts, enforcing wall-clock timeouts, and discarding error-contaminated trajectories to enable stable, large-scale end-to-end training.

3.1 Notation

Completing tasks in complex, long-horizon environments is commonly formulated as a Markov Decision Process (MDP) $\left\langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \right\rangle$. At each step $t$ of a multi-step task, an LLM-based agent $\pi_\theta$ receives a task instruction together with an observation[^2] $s_{t} \sim \mathcal{S}$, generates an action $a_{t} \sim \pi(\cdot | s_t)$, and transitions to the next observation $s_{t+1} \sim \mathcal{S}$. Under naive inference (e.g., ReACT [35]), the agent is given the raw interaction history $\left\langle s_{t-H}, a_{t-H}, \ldots, s_t\right\rangle$ and reasons before emitting the next action $a_{t}$. This loop repeats until the task is completed or a maximum number of steps is reached, at which point a terminal reward $r_{T} \sim \mathcal{R}(a_T, s_T)$ indicates success or failure. However, for complex tasks such as coding and GUI control, prior work has shown that carefully designed tools and advanced control flows—subagents, skills, and planning modes—substantially improve agent performance ([36, 37, 27]). In practice, the agent is therefore wrapped in a harness $\mathcal{H}(\pi)$ that supplies these tools and control flows internally, so that its effective context is no longer the raw interaction history. To abstract away this complexity, we denote (i) each prompt-response pair produced during harness inference as $(\mathcal{H}(s_t), a_t)$, and (ii) the full trajectory as an unordered collection $\tau = \left\langle (\mathcal{H}(s_0), a_0), (\mathcal{H}(s_1), a_1), \ldots, (\mathcal{H}(s_T), a_T) \right\rangle$.

[^2]: Technically, any input to the agent from our environments is an observation (as in a POMDP); to simplify notation, we use $s$ to denote input data received from the environment.

3.2 OpenForge RL

While the emerging agent domains beyond software engineering are increasingly well benchmarked ([6, 38, 5]), little open-source infrastructure supports training harness-based agents end-to-end in them. This is challenging for two reasons. First, existing RL frameworks assume the trainer has direct access to the model's prompts and full control over generation throughout a rollout; harnesses break this assumption, since their control flows, subagents, and context management are not exposed to the trainer. Second, harness rollouts need containerized environments with dedicated CPU and memory that cannot be co-located on the training nodes at scale. More broadly, open models are rarely trained in the harnesses they are deployed with, widening the gap to closed-source frontier models.

We address both with $\textsc{OpenForge RL}$, a plug-and-play rollout interface connecting popular RL frameworks (e.g., veRL) to distributed harness rollouts (Figure 2). Given an inference server (e.g., vLLM [39]), $\textsc{OpenForge RL}$ launches (1) a Kubernetes orchestrator that creates, manages, and deletes rollout container pods on cloud providers such as Microsoft Azure, and (2) a proxy that wraps the inference server and intercepts all generation requests issued by those containers. When a rollout finishes, the proxy collects the terminal reward $r_T$ (typically task success) and the prompt-response pairs $(s^{\mathcal{H}}, a)$ from the container, and reconstructs training samples as:

$ \tau = (s^{\mathcal{H}}_0, a_0, r_0), (s^{\mathcal{H}}_1, a_1, r_1), \ldots, (s^{\mathcal{H}}T, a_T, r_T);\quad r{t} = \gamma^{T-t} \cdot r_T\tag{1} $

typically with $\gamma=1.0$. When optimizing with group-based algorithms such as GRPO, we follow [40] and compute advantage by comparing the average sample rewards in different trajectories in the same group. While offloading rollouts to remote containers enables scaling to many concurrent environments, it raises three practical challenges, which we address below.

Rollout orchestration.

To manage the lifecycle of many rollout containers, we build on Orchard ([1]) and implement a Kubernetes orchestrator that creates, deletes, and allocates resources for rollout containers on cloud providers such as Microsoft Azure. This allows us to easily manage and scale the number of concurrent rollouts elastically, without overloading training nodes.

Asynchronous rollout and timeouts.

Because each rollout runs remotely and outside the trainer's control, a single unresponsive rollout will block the collection of an entire training batch and subsequently stall training. Capping the number of turns per rollout ([31, 33]), a common safeguard, is unreliable in this setting: some harnesses (e.g., Codex) do not expose a turn limit, and a "turn" is defined inconsistently across harnesses. We instead impose a (generous) wall-clock timeout on each rollout job. When a job exceeds its timeout, we terminate it and return an error signal to the proxy and the trajectory-reconstruction module, so training continues collecting from the remaining rollouts rather than blocking by the terminated one.

Error handling.

Rollouts in these complex environments can fail for reasons unrelated to the policy model, such as network issues, harness crashes, or timeouts. Following DAPO ([41]), we consider a simple strategy to discard all samples from a trajectory that ended in such an error, since a partial rollout can inject misleading training signal (e.g., a correct prefix that receives a negative reward). Designing better credit assignment for these partial rollouts is a promising direction in this setting, which we leave to future work.

3.3 Data Synthesis

This work aims to train harness-based agents end-to-end across diverse environments beyond coding, such as browser- and computer-use. Unlike coding ([42]), these domains offer far fewer training tasks, harnesses, and RL-ready environments. To help experiment our framework in diverse environments, we therefore also built a simple pipeline to synthesize SFT and RL tasks for such data-scarce domains (e.g., claw/daily tool use and computer use). We give a high-level overview below and defer full details to Appendix B.1.

The pipeline is built to mimic how a human would curate a task ([43, 44]). Given a target domain and a number of tasks, it spawns agents in parallel that (1) propose candidate instructions grounded in realistic scenarios, drawn from the web/X API or a pool of reference assets and instructions; (2) prune low-quality and duplicate tasks; (3) build an executable environment and a verifier script for each task; (4) test the task by rolling out a separate open LLM/VLM; and (5) refine it by patching defects until it passes all checks. The test and refine stages are essential: they validate each task end-to-end before it enters the dataset. Figure 3 illustrates the pipeline, and Appendix B.1 provides full details. Because each environment is defined by a custom Dockerfile, the pipeline extends naturally from Linux/CLI tasks (e.g., claw) to GUI and computer-use tasks (e.g., rendering virtual displays with Xvfb), and can pre-install arbitrary harnesses such as OpenClaw and Codex. The resulting tasks and environments support both SFT (via distillation from a stronger model's rollouts) and RL. We will release these data and this pipeline for future use.

: Table 1: Statistics of SFT and RL data used for Claw and GUI agent training. For GUI, we used a slightly modified version of the Kimi-Agent and Molmo-Web harness (see Appendix C.3 and Appendix C.4).

	Claw	GUI(Computer)	GUI(Browser)
Harnesses	ReACT*, ZeroClaw, OpenClaw, Codex	Kimi-Agent*	MolmoWeb*
No. SFT Trajectories	892	795	1, 496
No. RL tasks	343	252	900

4. Experiments

Section Summary: In the experiments, researchers applied OpenForge RL to train language-model agents on both text-based tool-use tasks and multimodal GUI environments, automatically generating large sets of executable tasks and environments for supervised fine-tuning followed by reinforcement learning. Claw-focused models trained this way, using backbones like Qwen3-30B and harnesses such as OpenClaw and Codex, outperformed similar-sized baselines and their own SFT-only versions on benchmarks including ClawEval, QwenClawBench, and MCPAtlas, with RL adding gains in robustness and success rate. Parallel training on computer- and browser-use GUI tasks was evaluated on OSWorld, Mind2Web, and WebVoyager, demonstrating the framework’s effectiveness across diverse interaction settings.

In this section, we use $\textsc{OpenForge RL}$ to train agents across a range of harnesses and environments. We then evaluate the trained agents across six popular benchmarks, spanning both text-based tool-use (claw) and multimodal GUI (browser and computer use) domains. In the next section, we then study how $\textsc{OpenForge RL}$ training reshapes agent behavior, examining tool-use patterns, generalization to unseen harnesses, and the behavioral changes that RL introduces on top of SFT.

4.1 Claw Agents

We first use $\textsc{OpenForge RL}$ to train agents across diverse harnesses. Here we hold the environment type fixed to text-based tasks, and train and evaluate LLM-based agents on everyday tasks that require diverse tool use, such as reading email, searching a knowledge base, or updating a helpdesk ticket, inside harnesses such as ZeroClaw, OpenClaw, and Codex.

SFT and RL Data

We use our pipeline in Section 3.3 to automatically generate a large pool of tasks and executable environments for both SFT and RL. To seed the proposal stage, we give the pipeline an assets and reference pool drawn from (1) ClawHub access[^3] and (2) tasks from ZClawBench ([45]), which together supply examples that reflect how people daily usage of claw-based agents. Table 1 and Figure 4 report the statistics and category distribution of the resulting tasks. To study the effect of training on diverse harnesses, we pair each task with one of four harnesses for its rollouts: ReACT (solving the task via a simple loop), ZeroClaw, OpenClaw, and Codex. Training dataset statistics is shown in Table 1 and Figure 4. For more details, please see Appendix B.2.

[^3]: The agent can use command lines to lookup popular skills/use cases from https://clawhub.ai/

Training Details

We use Qwen3-30B-A3B-Thinking ([46]) as the backbone model for training. For SFT, we distill trajectories from a stronger teacher model (MiniMax-M2.5, [47]): we sample $N=3$ rollouts per task and keep only the successful ones for training. For RL, we continue from the SFT checkpoint and train with GRPO ([48, 49]). To instantiate $\textsc{OpenForge RL}$, we use veRL ([15]) as the training backend and Microsoft Azure as the cloud provider for rollout containers. We use a batch size of $8$ and a group size of $8$, and train on 8 $\times$ B200 GPUs. For additional details, including training hyperparameters and training curves, see Appendix A.1.

Evaluation Benchmarks

We evaluate on two popular claw-based benchmarks, ClawEval ([20]) and QwenClawBench ([21]), and one broader tool-call benchmark, MCPAtlas ([5]). For ClawEval, we use the 2026-04-08 release and follow the official protocol, reporting $pass^{3}$ and $pass@3$ with the benchmark's default ReACT loop as the harness. For QwenClawBench, we follow the official implementation and solve the tasks with OpenClaw ([12]). We report $pass@1$ as the main metric. For MCPAtlas, we use the benchmark's default 20-server configuration, which excludes optional servers requiring third-party credentials or service-specific data initialization for more reproducible results. In total, this includes 89 tasks whose ground-truth expected tool calls are fully supported by this server set. We then follow the official setting and use the benchmark's LLM-as-judge claim-coverage evaluator, counting a task as successful when its coverage score is at least 0.75. We report $pass@1$ as the main metric. For more details on evaluation, please see Appendix C.

::: {caption="Table 2: Claw-agent performance on Claw-Eval, QwenClawBench, and MCPAtlas. We use the general domain (0408) for Claw-Eval. *marks numbers from the Claw-Eval leaderboard. †marks numbers from the QwenClawBench leaderboard. MCPAtlas^spadesuit results are pass@1 over the 89-task without credential-server configuration."}

:::

Results

We present our results in Table 2. Compared to models of similar size (around 30B, or MoE models with $\sim$ 3B active parameters) and to the untrained Qwen3-30B-A3B-Thinking backbone, our OpenForge-Claw models achieve superior results across all three benchmarks. This indicates the effectiveness of our curated tasks and environments, which provide useful learning signal for the model to solve everyday tasks through harnesses such as OpenClaw. Next, compared to OpenForge-Claw(SFT), our OpenForge-Claw(SFT+RL) shows a significant improvement in both robustness ($pass^{3}$ on ClawEval) and average success rate ($pass@1$ on QwenClawBench and MCPAtlas). This indicates the effectiveness of our $\textsc{OpenForge RL}$ training infrastructure, which allows the model to explore and learn from online interaction with the environments and harnesses. For evaluation across different harnesses and generalization to unseen harnesses, please refer to Section 5.1 and Section 5.2, respectively.

4.2 GUI agents

Beyond diverse harnesses, we also explore $\textsc{OpenForge RL}$ training in diverse environments, such as multimodal GUI environments that require visual perception and low-level mouse and keyboard control in computer-use and browser-use tasks.

::: {caption="Table 3: GUI-agent performance on OSWorld-verified, OnlineMind2Web, and WebVoyager. All results are pass@1. *marks numbers reported by model's official technical report; †marks numbers reported in [50, 22]. Best in shown in bold, second shown in gray."}

:::

SFT and RL Data

Following the same procedure as in Section 4.1, we use our pipeline in Section 3.3 to automatically generate a large pool of tasks and environments. To seed the proposal stage for computer-use tasks, we give the pipeline a reference and artifact pool consisting of (1) X (formerly Twitter) search API access[^4], (2) 22k instructions from AgentNet ([51]), and (3) synthetic files and data from Synthetic-Computer-Use ([52]) that populate each environment with realistic assets. To build containerized computer-use environments, we render a virtual display with Xvfb and pre-install a GUI harness (e.g., Kimi-Agent), so that a VLM can control the machine through simulated mouse clicks (e.g., left_click(x, y)) and keyboard actions (e.g., type("text")). For browser-use, since synthesizing realistic websites and their databases is impractical, we instead follow OpenWebRL ([50]) and draw real-website tasks from existing datasets. Specifically, we (1) start with tasks from WebGym ([53]); (2) filter tasks that overlap the evaluation benchmarks and also restricting to popular websites; (3) performed deduplication. This results in an SFT pool of 2500 tasks and 900 RL tasks. As in OpenWebRL, we prompt GPT-4.1 ([54]) as the evaluator to determine task success during both SFT data construction and RL training. To build containerized browser-use environments, we pre-install a GUI-browser harness (i.e., Molmo-Web) and used remote browser service from Browser-Use ([55]) to interact with real websites. Training dataset statistics is shown in Table 1 and Figure 4. For more details on our GUI training data, please see Appendix B.3.

[^4]: The agent can use the X search API to find real-world use cases of computer-use agents.

Training Details

We use Qwen3-VL-8B-Thinking ([46]) as the backbone model for all training. For SFT, we distill trajectories from a stronger teacher model (Kimi-K2.5, [56]): we sample $N=3$ rollouts per task and keep only the successful ones for training. For RL, we also follow the previous section - we instantiate $\textsc{OpenForge RL}$ with veRL as the training backend and Microsoft Azure as the cloud provider for rollout containers. We use GRPO with a batch size of $8$ and a group size of $8$, and train on 8 $\times$ B200 GPUs. All models use screenshots as their visual input for both training and evaluation. For additional details, such as training hyperparameters and training curves, please see Appendix A.2.

Evaluation Benchmarks

We evaluate on three popular GUI benchmarks: OSWorld-Verified ([9]) for computer-use environments, and Online-Mind2Web ([8]) and WebVoyager ([57]) for browser-use. For OSWorld-Verified, we follow the official protocol and report the average success rate. For browser-use, we follow [50, 58] and report average success rate using the AgentTrek protocal ([59]) with o4-mini ([60]) for Online-Mind2Web and the official protocal with GPT-4o ([61]) for WebVoyager. More details about each benchmark can be found in Appendix C.3 and Appendix C.4.

Results

We present our results in Table 3. Compared to similar-size (around 8B) models that were specifically fine-tuned for computer-use or browser-use (e.g., OpenCUA, UI-TARS, and MolmoWeb), our OpenForge-GUI models achieve superior results on nearly all benchmarks. Notably, while MolmoWeb is trained on over 200k tasks, OpenForge-GUI uses only 2.5k tasks yet outperforms it on Online-Mind2Web and stays competitive on WebVoyager. More importantly, OpenForge-GUI(SFT+RL) improves substantially over OpenForge-GUI(SFT) on all three benchmarks. This is a considerably harder test of $\textsc{OpenForge RL}$ than the text-only setting: every GUI rollout runs a full VLM harness inside a containerized virtual display, perceives the screen visually, and issues long sequences of low-level mouse and keyboard actions against real applications and websites. These consistent gains show that $\textsc{OpenForge RL}$ generalizes beyond diverse harnesses to diverse, highly complex environments, training agents end-to-end even under demanding multi-turn, multimodal GUI tasks.

5. Discussion

Section Summary: The discussion examines how different tool-use harnesses influence what language models learn during supervised fine-tuning and reinforcement learning on agent tasks. Simpler harnesses that let models add custom tools are easiest to master and benefit most from training, while training on one harness transfers modestly to unseen ones and works best when models see multiple harnesses during training because of greater variety in tool calls. On top of supervised fine-tuning, reinforcement learning further improves performance by steering models toward specialized tools instead of generic shells and by strengthening skills such as error recovery and self-verification.

In this section, we analyze how the choice of harness affects learning and what our models learn from harness-based SFT and RL training. We organize the analysis around three questions. (1) Are some harnesses harder to master than others? (Section 5.1). (2) Does training on one harness transfer to unseen harnesses, and does training on several at once help further? (Section 5.2). (3) What capabilities does RL add on top of SFT? (Section 5.3). For simplicity, we focus on our study with OpenForge-Claw models evaluated on the ClawEval benchmark.

5.1 Evaluation across Different Harnesses

::: {caption="Table 4: Comparing OpenForge-Claw on ClawEval with different harnesses."}

:::

First, we study whether some harnesses are harder to learn than others. In principle, any task is solvable by any harness; in practice, prior work ([3, 62]) also find tools and control flows are better aligned with the model's capabilities can often reach higher performance more easily.

We therefore evaluate our trained OpenForge-Claw models on ClawEval using four harnesses of increasing sophistication: ReACT*, ZeroClaw, OpenClaw, and Codex. ReACT* is the benchmark's original harness, a ReACT-like loop that repeatedly prompts the model to reason and call tools, and ZeroClaw ([63]) is a lightweight version of OpenClaw that adds an easy way to register new tools. OpenClaw ([12]) and Codex ([11]) are far more advanced, offering a rich set of built-in tools and control flows, but neither easily supports custom tools; we therefore expose each ClawEval-specific tool to them through SKILL.md files (see Appendix C.1).

Table 4 reports the results, from which two trends stand out. First, the harnesses that support adding custom tools directly (ReACT and ZeroClaw) reach the highest performance. Second, SFT+RL bring large gains on every harness except OpenClaw, which only has moderate gains while consuming far longer prompts and contexts than the others. This corroborates prior findings that simpler, better-engineered tools and control flows are essential for agentic performance, and further shows that training on such harnesses advances the model's ability to solve tasks in complex environments.

5.2 Generalization to Unseen Harness

\begin{tabular}{@lccc@}
\toprule
  {} & \multicolumn{3}{c}{ClawEval($pass@1$)} \\
\cmidrule(lr){2-4}
\textbf{Training (SFT+RL)} & ZeroClaw & OpenClaw & Codex \\
\midrule
None (base)
   {} & 32.5 & 11.4 & 12.2 \\

ZeroClaw
   {} & 46.0 {\tiny(+13.5)} & 14.7 {\tiny(+3.3)} & 16.8 {\tiny(+4.6)} \\

\textbf{ZeroClaw+OpenClaw+Codex}
   {} & \textbf{48.5} {\tiny(+16.0)} & \textbf{20.9} {\tiny(+9.5)} & \textbf{32.5} {\tiny(+20.3)} \\
\bottomrule
\end{tabular}

Next, we ask whether training on one harness transfers to unseen harnesses in evaluation, and whether training on several harnesses at once improves over training on a single one. We compare two OpenForge-Claw models built from the same tasks and the same training recipe, differing only in which harnesses generate their rollouts (for both SFT distillation and RL): one trained on ZeroClaw alone, and one trained on ZeroClaw, OpenClaw, and Codex together. We evaluate both on all three harnesses (Table 5). For the ZeroClaw-only model, OpenClaw and Codex are unseen.

We observe two effects. First, training on a single harness already generalizes to the others: the ZeroClaw-only model improves over the untrained base on the unseen OpenClaw (+3.3) and Codex (+4.6). Second, training on all three harnesses is best across the board, with the largest gains over the base on the more complex harnesses (+9.5 on OpenClaw and +20.3 on Codex), and it even lifts ZeroClaw itself beyond ZeroClaw-only training (48.5 vs. 46.0). We attribute this to the greater diversity of tool calls and control flows the model sees when trained on multiple harnesses, which makes it more robust across different tools and scenarios.

5.3 Capability Learned by RL

Finally, we investigate what RL learns on top of SFT. We run the SFT and SFT+RL OpenForge-Claw checkpoints on ClawEval and compare 100 trajectories from each, examining tool-call statistics under the ZeroClaw harness (Figure 5, left) and higher-level behavioral capabilities under the Codex harness (Figure 5, right).

On ZeroClaw, we find RL changes how the model uses tools. It reduces generic shell calls from 22.6% to 13.9% of all tool calls and redistributes them toward dedicated service tools, while also slightly shortening trajectories (Figure 10 gives the full breakdown). This suggests that RL teaches the model to reach for the right specialized tool instead of falling back on a general-purpose shell.

On Codex, we find RL improves several agentic capabilities (see Appendix D.1 for detailed definitions) that matter for long-horizon tool use. In particular, it strengthens error recovery (correctly solving a task after a failed command) and self-verification (reading back its own writes to confirm them), and, echoing the ZeroClaw analysis, it exercises a wider set of the tools each task requires. These are the behaviors that keep an agent reliable across many steps. Error recovery, however, remains the weakest capability even after RL. We hypothesize that such capabilities are difficult to acquire from RL alone, and may require dedicated data or training methods to strengthen further.

6. Conclusion

Section Summary: The OpenForge RL framework enables researchers to train language and vision-language models as agents directly inside the actual software environments and harnesses where they will later be used, rather than relying on simplified approximations. With only modest amounts of curated tasks, the approach produced models that outperform other open models of similar size on tool-use and graphical-interface benchmarks, while also revealing that reinforcement learning mainly boosts an agent’s ability to verify its own steps and follow multi-step plans, though error recovery stays limited. The authors release the full code, data, and trained models to make such real-world agent training more accessible to others.

We present $\textsc{OpenForge RL}$, an open framework for training LLM- and VLM-based agents end-to-end, directly inside the inference harnesses they are deployed with. $\textsc{OpenForge RL}$ makes any harness $\times$ any environment trainable with standard RL codebases such as veRL, so agents can be optimized in their real deployment settings rather than in simplified reimplementations. Using only hundreds to a few thousand automatically curated tasks, we train OpenForge-Claw and OpenForge-GUI models that surpass open models of similar size on nearly all of our tool-use and GUI benchmarks, and in the GUI setting match or exceed models several times larger. Concretely, OpenForge-Claw reaches $31.7$ ($\mathrm{pass}^3$) on ClawEval, $33.7$ on QwenClawBench, and $28.1$ on MCPAtlas, while OpenForge-GUI reaches $37.7$ on OSWorld-Verified, $63.0$ on Online-Mind2Web, and $72.3$ on WebVoyager. Beyond training, $\textsc{OpenForge RL}$ lets us analyze how harness choice and RL shape agent behavior, a study prior open work could not easily conduct. We find that some harnesses are substantially harder to learn than others, that training gains transfer to harnesses unseen during training, and that RL primarily improves agentic reliability: the model verifies its own actions, covers more of the tools each task needs, and completes multi-step plans. Error recovery, however, remains weak even after RL, suggesting that some capabilities may need dedicated data. We release our code, data, and models, and hope $\textsc{OpenForge RL}$ lowers the barrier to training and studying agents in the real harnesses and environments where they are deployed.

Appendix

Section Summary: The appendix describes the technical setup for training Claw and GUI agents via reinforcement learning, including the use of cloud-based containers on Azure Kubernetes pods for running agent rollouts, separate GPU nodes for model updates, and specific resource limits, timeouts, and hyperparameters tailored to each agent type such as Claw, computer-use, and browser-use. It also outlines a multi-stage data synthesis pipeline that employs LLM agents to propose, prune, build, test, and refine executable tasks and environments, along with cost-saving adjustments for supervised fine-tuning data on Claw tasks. Supporting details include training curves and a table of key parameters.

A. Training Details

A.1 Claw Agent Training Details

For OpenForge-Claw RL training, we use veRL as the training backend and Microsoft Azure as the cloud provider for rollout containers. Each rollout runs in its own container, built from a task-specific Dockerfile with the target harness (e.g., OpenClaw, Codex, or ZeroClaw) pre-installed, and is scheduled onto a Kubernetes pod capped at 2 CPUs and 2GB RAM. These pods are packed onto Azure D128ads v5 nodes, while policy training runs on a single node of 8 $\times$ B200 GPUs. We report the key training hyperparameters in Table 6 and the training curves in Figure 6.

A.2 GUI Agent Training Details

For OpenForge-GUI RL training, we similarly use veRL as the training backend and Microsoft Azure as the cloud provider for rollout containers.

Computer-Use

For computer-use agent training, each rollout runs in its own container built from a task-specific Dockerfile with the target harness (a modified version of Kimi-Agent, see Appendix C.3) pre-installed, and is scheduled onto a Kubernetes pod capped at 4 CPUs and 4GB RAM. These pods are packed onto Azure D64ads v5 nodes, while policy training runs on a single node of 8 $\times$ B200 GPUs. We report the key training hyperparameters in Table 6 and the training curves in Figure 7.

Browser-Use

For browser-use agent training, each rollout drives a dedicated remote browser session on the Browser-Use Cloud service (hosted Chromium) over the Chrome DevTools Protocol. The per-rollout environment, which handles session management, action execution, screenshot capture, and the reward judge, runs in a lightweight Kubernetes sandbox pod capped at 2 CPUs and 6 GB RAM; the browser itself runs remotely rather than inside the pod. These pods are packed onto Azure D128ads v7 nodes, and up to 48 rollouts run concurrently, while policy training runs on a single node of 8 $\times$ B200 GPUs. We report the key training hyperparameters in Table 6.

: Table 6: Key RL training hyperparameters for OpenForge-Claw and OpenForge-GUI.

Hyperparameter	Claw	Computer-Use	Browser-Use
Learning rate	1e-6	5e-7	1e-6
Batch size	$8$	$8$	$12$
Group size	$8$	$8$	$5$
KL coefficient	0.001	0.01	0.0
Entropy coefficient	0.0	0.0	0.001
Max rollout time / step	900s	600s	900s
Training steps	100	80	100
Rollout VM CPU / pod	2	4	2
Rollout VM Memory / pod	2GiB	4GiB	6GiB
Total Training Time	48H	36H	32H

B. Data Details

B.1 More Details on Data Synthesis

We implement our data synthesis pipeline (Section 3.3) on the Claude Agent SDK, using Opus 4.6 ([64]) as the backbone for each of the five agent modules. Given a target domain (as a natural-language prompt) and a target number of tasks $N$, the pipeline spawns many agents in parallel that run the following five stages.

• Propose. Each agent drafts candidate instructions by browsing the assets and reference pool we construct for the target domain, as we find that directly prompting a model to invent tasks from scratch tends to yield infeasible or unoriginal instructions. For Claw agents, this pool comprises (1) skills from ClawHub and (2) tasks from ZClawBench ([45]); for computer-use agents, it comprises (1) X search API access, (2) 22k instructions from AgentNet, and (3) synthetic files and data from Synthetic-Computer-Use. To record drafted instructions and avoid duplicates, we maintain a shared SQLite database and instruct each agent to append its proposals to it.
• Prune. From the proposed but unimplemented instructions in the database, these agents select the $N$ highest-quality and most diverse instructions, and mark the remainder as discarded.
• Build. For each selected instruction, an agent constructs a fully executable environment: tool servers, mock websites, mock data and files, a Dockerfile that packages everything into an image, and a verifier script that scores task success.
• Test. Many environment errors and instruction ambiguities are hard to detect without actually running the task. We therefore provide the agent with a script that invokes a separate open LLM/VLM to attempt the task in the built environment: MiniMax-M2.5 for Claw tasks and Kimi-K2.5 for computer-use tasks.
• Refine. Finally, the refine agent inspects the rollout trajectory and the verifier's score and decides whether the environment has any defects or the instruction is ambiguous. If so, it patches the environment and/or revises the instruction, and repeats the test-and-refine loop until the task passes all checks.

B.2 More Details on Claw Data

At the time of the project, we did not find any large scale public dataset suitbale for training Claw-based agents, especially for RL training. As a result, we primarily rely on our data synthesis pipeline (Appendix B.1) to generate tasks for SFT and RL training. While powerful, we find synthesizing a task with a thoroughly tested verifier is costly in both time and money, as it involves multiple rounds of real rollouts, refinement, and re-rollout. To save cost, we instead consider for SFT tasks, we skip the test and refinement stage, and directly prompt GPT-5.4 as a judge to determine task success. This is much afforadable and reasonable since for SFT trajectories the success signals are often only used for onetime data filtering. For RL tasks, we maintain the full test-and-refine loop to ensure the verifier is robust and reliable, as RL training requires repeated rollouts and the verifier must be able to consistently judge task success. On average, synthesizing an RL task (with verifier and refining) takes 16.1 minutes and 4.36 USD, while synthesizing an SFT task (without verifier) takes 5.2 minutes and 0.86 USD. A full breakdown of our task distribution is shown in Figure 11.

For harnesses, in addition to the default ReACT loop implemented by most tool-use agents, we also include three popular harnesses: ZeroClaw, OpenClaw, and Codex, to experiment with the capabilities of our $\textsc{OpenForge RL}$ infrastructure.

B.3 More Details on GUI Data

Computer-use tasks

At the time of this work, there was no large-scale public dataset for training computer-use agents, especially with RL. One reason is that running a computer-use GUI in a lightweight container is challenging: such environments are commonly run as full Ubuntu virtual machines under QEMU ([9]). We find, however, that Xvfb can instead render a virtual display in memory, which is far more lightweight and lets us run many containers in parallel for RL training. With computer-use GUIs now runnable in lightweight containers, we follow our Claw setup and use the same data synthesis pipeline (Appendix B.1) to generate SFT and RL tasks. For SFT tasks, we again skip the test-and-refine stages and prompt GPT-5.4 as a judge to determine task success. On average, each SFT task takes 4.0 minutes and 1.37 USD to synthesize. For RL tasks, we keep the full test-and-refine loop so that the verifier is robust and reliable. On average, synthesizing an RL task (with verification and refinement) takes 21.3 minutes and 6.12 USD.

As the harness, we primarily use Kimi-Agent ([56]), following the implementation of [9], with slight modifications to support additional tools such as bash, following Anthropic's computer-use approach. For the full action space and tool list, see Appendix C.3.

Browser-use tasks

For browser-use, we adapt MolmoWeb ([22])'s codebase as the harness and our main modifications involve (1) using json-formatted action space other than MolmoWeb's tool-call format to avoid extra training stages for action alignment, and (2) integrating Browser-Use Stealth Browsers ([55]) following OpenWebRL to solve CAPTCHA and website blockings, which we found reducing the ratio of IP and CAPTCHA block from 40% to nearly zero. We provide a more detailed description and examples of the observation and action spaces for both environments in Appendix C.4.

For SFT data, we follow a similar data curation pipeline to OpenWebRL including task-filtering and trajectory collection. We start from subsampling the PAE-WebVoyager ([44]) split from WebGym ([53]) by removing tasks that overlap with evaluation benchmarks and subtasks decomposed from parent intents tasks. Then, we keep only the most popular websites that exists in SimilarWeb Top100 and MOZ Top 500 to remove the distrations of unusual websites in the long-tail distribution. Finally, to remain the diversity of tasks, we embed task instructions with Qwen3-Embedding-8B ([65]) and apply greedy similarity-based deduplication with a predefined threshold 0.55. The eventual SFT candidate pool contains 2500 tasks.

We apply a stronger teacher model (Kimi-K2.5) to infer on the candidate pool with max turns 30 and 4 repeats. We collect all successful trajectories, and keep only the shortest trajectory among multiple success of a single task. We found that repeated actions commonly appear for complex websites, and we reserve only the last turn for more than 3 consecutive identical actions anf remove the entire trajectory if more than 5 identical actions occur in the process. Following the above operations, we curate 1496 trajectories for distillation.

During RL stage, we apply the same popular website filter as SFT and randomly sample 600 tasks from the Insta-v3 ([66]) and 300 tasks from the PAE-WebVoyager ([44]) split that are distinct from the SFT pool. The maximum turn is restricted to 20 to balance the performance and training cost.

C. Evaluation Details

C.1 ClawEval and QwenClawBench Evaluation Details

We evaluate our OpenForge-Claw models on popular claw- and harness-related benchmarks that measure how well an agent can use a harness to solve tasks. Specifically, we use ClawEval ([20]), QwenClawBench ([21]), and MCPAtlas ([5]), the last serving as a related but "held-out" test set for novel tool use. All results in our main experiments (Section 4.1) use each benchmark's official evaluation protocol: for ClawEval, the official ReACT-like loop repeatedly prompts the model to reason and call tools; for QwenClawBench, the OpenClaw harness is used.

To study the effect of harness choice on model performance, we additionally evaluate on ClawEval under three other harnesses: ZeroClaw, OpenClaw, and Codex (Section 5.1). Because ClawEval requires the model to call custom tool servers, ZeroClaw was straightforward to support, as it natively allows registering new tools. To adapt OpenClaw and Codex, which is non-trivial for adding custom tools, we expose each ClawEval-specific tool to them through SKILL.md files. Specifically, for each custom tool server, we prompt an LLM (Claude Opus 4.6) with its API signature and tool descriptions to generate a SKILL.md file that explains how to call the tool from a bash command line. At inference time, these files are loaded into the harness, and the model invokes the tools through the bash interface. Evaluation protocols are unchanged from the official ClawEval benchmarks.

C.2 MCP-Atlas Evaluation Details

For reproducibility, we evaluate MCPAtlas ([5]) under the benchmark's default 20-server configuration, which excludes optional servers that require third-party credentials or service-specific data initialization. This configuration fixes the evaluation set without any manual selection on our part: of the 500 public tasks, exactly 89 have ground-truth expected tool calls that are fully supported by these default servers. We evaluate every model on this same 89-task set, holding the task identifiers and environment configuration fixed.

To evaluate, we use the MCPAtlas official harness which is based on a ReACT-like loop: the harness exposes the task-specific MCP tools to the policy model, executes its tool calls in the MCPAtlas sandbox, and returns the resulting observations until the model terminates. We use the default task prompts and tool configurations, and add no benchmark-specific demonstrations or fine-tuning.

Following the MCPAtlas claim-coverage protocol, we score each final response against the ground-truth factual claims using Gemini 2.5 Pro as the judge. A task counts as successful when its claim-coverage score is at least $0.75$, and we report the fraction of successful tasks as pass@1. Every model is evaluated under the same task subset, harness, judge, and threshold.

C.3 Computer-Use Evaluation Details

For computer-use environments, we evaluate on OSWorld-Verified ([9]). OSWorld is a popular benchmark that evaluates how well a multimodal agent can complete open-ended, real-world computer tasks on a real Ubuntu desktop, operating applications through the screen with mouse and keyboard and being scored by task-specific, execution-based verifiers. OSWorld-Verified is an in-place upgrade of OSWorld with enhanced infrastructure and improved task quality built from 300+ pieces of feedback from the community.

Evaluating on OSWorld-Verified first requires choosing a harness. Because our computer-use data pipeline is built around Kimi-K2.5 ([56]), we use Kimi-Agent, as implemented in the official OSWorld repository, as our main training and evaluation harness. To make the harness more efficient, we add a lightweight modification inspired by Anthropic's computer-use approach: we additionally expose bash and a str_replace_editor tool, letting the model directly manipulate files and data (e.g., editing text, CSV, and JSON files) without opening a GUI application. We run Kimi-Agent in screenshot-only mode: every observation is a screenshot of the GUI, with no additional text or metadata. We show an example trajectory in Figure 8 and the full action space and tool list in Table 7.

```latextable {caption="Table 7: Action space of our computer-use agent. The agent controls an Ubuntu desktop through three tools: computer (mouse, keyboard, scrolling, and episode control), bash (shell access), and str_replace_editor (file editing). Coordinates are normalized to [0, 1] relative to the screenshot (x=0 left, x=1 right, y=0 top, y=1 bottom), backed by a 1920×1080 screen."} \begin{tabular}{@p{0.30\textwidth}p{0.21\textwidth}p{0.41\textwidth}@} \toprule \textbf{Action / command} & \textbf{Arguments} & \textbf{Description} \ \midrule \multicolumn{3}{@l}{\textit{\quad\texttt{computer} --- desktop mouse, keyboard, scrolling, and control}}\ \cmidrule(lr){1-2} \addlinespace[2pt] \raggedright \texttt{left_click}, \texttt{right_click}, \texttt{middle_click}, \texttt{double_click}, \texttt{triple_click}, \texttt{left_press} {} & \raggedright \texttt{coordinate}, \texttt{keys} {} & \raggedright Click at \texttt{coordinate} with the given button and click count; \texttt{left_press} presses and holds briefly. \texttt{keys} adds modifiers. \tabularnewline \raggedright \texttt{mouse_move}, \texttt{left_click_drag}, \texttt{left_mouse_down}, \texttt{left_mouse_up} {} & \raggedright \texttt{coordinate}, \texttt{start_coordinate}, \texttt{coordinate2} {} & \raggedright Move the cursor, drag from \texttt{start_coordinate} to \texttt{coordinate2}, or press/release the left button for manual drags. \tabularnewline \raggedright \texttt{key}, \texttt{type}, \texttt{hold_key}, \texttt{key_down}, \texttt{key_up} {} & \raggedright \texttt{text}, \texttt{keys}, \texttt{duration} {} & \raggedright Type \texttt{text}, press a key or hotkey (e.g., \texttt{ctrl+l}), or hold / press / release specific \texttt{keys}. \tabularnewline \raggedright \texttt{scroll}, \texttt{hscroll} {} & \raggedright \texttt{scroll_direction}, \texttt{scroll_amount}, \texttt{coordinate} {} & \raggedright Vertical or horizontal mouse-wheel scroll by \texttt{scroll_amount} in \texttt{scroll_direction}, optionally anchored at \texttt{coordinate}. \tabularnewline \raggedright \texttt{screenshot}, \texttt{wait}, \texttt{terminate}, \texttt{done}, \texttt{fail} {} & \raggedright \texttt{duration}, \texttt{status} {} & \raggedright Capture the screen, wait, or end the episode as success / failure (\texttt{status}). \tabularnewline \midrule \multicolumn{3}{@l}{\textit{\quad\texttt{bash} --- shell access}}\ \cmidrule(lr){1-2} \addlinespace[2pt] \raggedright \texttt{bash} {} & \raggedright \texttt{command}, \texttt{timeout}, \texttt{working_dir} {} & \raggedright Run a shell \texttt{command} inside the desktop environment (not the host). \tabularnewline \midrule \multicolumn{3}{@l}{\textit{\quad\texttt{str_replace_editor} --- file viewing and editing}}\ \cmidrule(lr){1-2} \addlinespace[2pt] \raggedright \texttt{view}, \texttt{create}, \texttt{str_replace}, \texttt{insert}, \texttt{undo_edit} {} & \raggedright \texttt{path}, \texttt{file_text}, \texttt{old_str}, \texttt{new_str}, \texttt{insert_line}, \texttt{view_range} {} & \raggedright View, create, and edit files with precise text operations at absolute \texttt{path}s. \tabularnewline \bottomrule \end{tabular}

#### C.4 Browser-Use Evaluation Details

For browser-use, we evaluate on Online-Mind2Web ([8]) (300 tasks) and WebVoyager ([57]), the latter on the 595-task "Fara-595" subset released with Fara-7B ([58]). We use our modified MolmoWeb harness (Appendix B.3), which exposes a single `computer_use` tool; the full action space is listed in Table 8. Observations are screenshot-only: each step provides the current $1280\times720$ viewport screenshot, with no DOM or accessibility tree, together with a short text block giving the previous action's result, the page title and URL, and the step count. Click and drag coordinates are normalized to $[0, 1]$ relative to the screenshot, the same convention as our computer-use agent. At inference, we cap each episode at 30 steps, with a 900 s per-task and 90 s per-action timeout, and score Online-Mind2Web with the AgentTrek protocol (o4-mini) and WebVoyager with the official protocol (GPT-4o). We show an example step in Figure 9.

![**Figure 9:** A representative browser-use step on an Online-Mind2Web task. Given the task instruction and the current screenshot, the model emits a single generation that interleaves its reasoning (`<think>`) and a tool call (`<tool_call>`). We overlay the predicted normalized click coordinate with a <span style="color:teal">green marker</span> on the screenshot, which correctly targets the Amazon search bar.](https://ittowtnkqtyixxjxrhou.supabase.co/storage/v1/object/public/public-images/s97mujev/complex_fig_e9a0801635f7.png)

```latextable {caption="Table 8: Action space of our browser-use agent. The model controls a Chromium browser through a single `computer_use` tool. Click and drag coordinates are normalized to [0, 1] relative to the screenshot, which is pinned to a 1280×720 viewport."}
\begin{tabular}{@p{0.30\textwidth}p{0.21\textwidth}p{0.41\textwidth}@}
\toprule
\textbf{Action} & \textbf{Arguments} & \textbf{Description} \\
\midrule
\multicolumn{3}{@l}{\textit{\quad\texttt{computer\_use} --- browser mouse, keyboard, scrolling, navigation, and control}}\\
\cmidrule(lr){1-2}
\addlinespace[2pt]
\raggedright \texttt{left\_click}, \texttt{right\_click}, \texttt{middle\_click}, \texttt{double\_click}, \texttt{triple\_click}
   {} & \raggedright \texttt{coordinate}
   {} & \raggedright Click at \texttt{coordinate} with the given button and click count (double/triple select a word/line). \tabularnewline
\raggedright \texttt{mouse\_move}, \texttt{left\_click\_drag}
   {} & \raggedright \texttt{coordinate}
   {} & \raggedright Move the cursor to \texttt{coordinate}, or drag from the current cursor position to it. \tabularnewline
\raggedright \texttt{type}, \texttt{key}
   {} & \raggedright \texttt{text}, \texttt{keys}
   {} & \raggedright Type \texttt{text} into the focused field, or press a key or chord (e.g., \texttt{[``ctrl'', ``a'']}). \tabularnewline
\raggedright \texttt{scroll}, \texttt{hscroll}
   {} & \raggedright \texttt{pixels}
   {} & \raggedright Vertical or horizontal scroll at the cursor by \texttt{pixels} (sign gives direction). \tabularnewline
\raggedright \texttt{goto}, \texttt{go\_back}, \texttt{new\_tab}, \texttt{tab\_focus}
   {} & \raggedright \texttt{url}, \texttt{index}
   {} & \raggedright Navigate to \texttt{url}, go back in history, open a new tab, or switch to the tab at \texttt{index}. \tabularnewline
\raggedright \texttt{wait}, \texttt{terminate}, \texttt{answer}
   {} & \raggedright \texttt{time}, \texttt{status}, \texttt{text}
   {} & \raggedright Wait for \texttt{time} seconds, end the episode (\texttt{terminate} with success / failure), or return the final \texttt{answer}. \tabularnewline
\bottomrule
\end{tabular}

D. Analysis Details

D.1 Behavioral Analysis Details

We analyze behavioral differences between the SFT and SFT+RL OpenForge-Claw checkpoints on ClawEval, running each checkpoint under the ZeroClaw and Codex harnesses.

Tool usage (ZeroClaw).

For each tool, we report the percentage of all tool calls that invoke it, aggregated over all rollouts. This captures which tools the model relies on, independent of whether a task is ultimately solved.

Behavioral capabilities (Codex).

We summarize five capabilities relevant to long-horizon tool use, shown as the radar plot in Figure 5 (right). Each is a percentage in $[0, 100]$ where higher is better, computed over rollouts from the two OpenForge-Claw checkpoints on the same set of tasks.

• Format robustness. The percentage of rollouts not terminated by a malformed tool call, i.e., $100$ minus the fraction of rollouts in which the harness fails to parse the model's function-call payload (typically a long structured output) and the session disconnects. It measures how reliably the model emits well-formed tool calls.
• Error recovery. Among rollouts that hit at least one failed command, the percentage that still solve the task. It measures whether the model can proceed after an error rather than crashing or giving up.
• Self-verification. The percentage of write (state-changing) tool calls that are followed by a read-back of the same service, e.g., listing tasks after creating one. It measures whether the model checks the effect of its own actions.
• Tool coverage. On tasks that require at least three distinct services, the percentage of rollouts that invoke every required service at least once. It measures whether the model carries a multi-part plan through to completion.
• Step efficiency. For each task, we define the best step count as the fewest steps among all successful runs (from 3 runs per task) of that task across both checkpoints. A checkpoint's score on a task it solves is $\text{best} / \text{(its own step count)}$, averaged over its solved tasks. A higher value means the model solves tasks in closer to the minimum number of steps.

References

Section Summary: This references section compiles dozens of academic papers, technical reports, and online resources centered on artificial intelligence systems. The entries cover benchmarks for testing AI agents on software engineering, web navigation, and computer interface tasks, along with frameworks for training such agents through reinforcement learning and tool use. They also include documentation from companies like Anthropic and OpenAI on related protocols, models, and platforms.

[1] Baolin Peng et al. (2026). Orchard: An Open-Source Agentic Modeling Framework. https://arxiv.org/abs/2605.15040. arXiv:2605.15040.

[2] Carlos E. Jimenez et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. https://arxiv.org/abs/2310.06770. arXiv:2310.06770.

[3] John Yang et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://arxiv.org/abs/2405.15793.

[4] Mike A. Merrill et al. (2026). Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. https://arxiv.org/abs/2601.11868. arXiv:2601.11868.

[5] Bandi et al. (2026). MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers. https://arxiv.org/abs/2602.00933. arXiv:2602.00933.

[6] Patil et al. (2025). The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In Forty-second International Conference on Machine Learning.

[7] Shuyan Zhou et al. (2024). WebArena: A Realistic Web Environment for Building Autonomous Agents. https://arxiv.org/abs/2307.13854. arXiv:2307.13854.

[8] Tianci Xue et al. (2025). An Illusion of Progress? Assessing the Current State of Web Agents. https://arxiv.org/abs/2504.01382. arXiv:2504.01382.

[9] Tianbao Xie et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. https://arxiv.org/abs/2404.07972. arXiv:2404.07972.

[10] Anthropic (2025). Overview - Claude Code Docs. https://code.claude.com/docs/en/overview. Accessed: 2026-07-20.

[11] OpenAI (2025). Introducing Codex. https://openai.com/index/introducing-codex/. Accessed: 2026-07-20.

[12] OpenClaw (2025). OpenClaw — Personal AI Assistant. https://openclaw.ai/. Accessed: 2026-07-20.

[13] Anthropic (2024). Introducing the Model Context Protocol. https://www.anthropic.com/news/model-context-protocol. Accessed: 2026-07-20.

[14] OpenAI (2026). Harness engineering: leveraging Codex in an agent-first world. https://openai.com/index/harness-engineering/. Accessed: 2026-07-20.

[15] Guangming Sheng et al. (2024). HybridFlow: A Flexible and Efficient RLHF Framework. arXiv preprint arXiv: 2409.19256.

[16] Zilin Zhu et al. (2025). slime: An LLM post-training framework for RL Scaling. https://github.com/THUDM/slime. GitHub repository. Corresponding author: Xin Lv.

[17] Shiyi Cao et al. (2025). SkyRL-v0: Train Real-World Long-Horizon Agents via Reinforcement Learning.

[18] Yuxiang Wei et al. (2025). SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution. https://arxiv.org/abs/2502.18449. arXiv:2502.18449.

[19] Wang et al. (2026). OpenClaw-RL: Train Any Agent Simply by Talking. arXiv preprint arXiv:2603.10165.

[20] Bowen Ye et al. (2026). Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents. https://arxiv.org/abs/2604.06132. arXiv:2604.06132.

[21] Qwen Team and Data Team, Alibaba Group (2026). QwenClawBench: Real-user-distribution benchmark for OpenClaw agents. github.com/SKYLENAGE-AI/QwenClawBench.

[22] Tanmay Gupta et al. (2026). MolmoWeb: Open Visual Web Agent and Open Data for the Open Web. https://arxiv.org/abs/2604.08516. arXiv:2604.08516.

[23] Chunqiu Steven Xia et al. (2024). Agentless: Demystifying LLM-based Software Engineering Agents. https://arxiv.org/abs/2407.01489. arXiv:2407.01489.

[24] Xingyao Wang et al. (2025). OpenHands: An Open Platform for AI Software Developers as Generalist Agents. https://arxiv.org/abs/2407.16741. arXiv:2407.16741.

[25] Wenyi Hong et al. (2024). CogAgent: A Visual Language Model for GUI Agents. https://arxiv.org/abs/2312.08914. arXiv:2312.08914.

[26] Yujia Qin et al. (2025). UI-TARS: Pioneering Automated GUI Interaction with Native Agents. https://arxiv.org/abs/2501.12326. arXiv:2501.12326.

[27] Saaket Agashe et al. (2024). Agent S: An Open Agentic Framework that Uses Computers Like a Human. https://arxiv.org/abs/2410.08164. arXiv:2410.08164.

[28] Jian Hu et al. (2024). OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. arXiv preprint arXiv:2405.11143.

[29] Wei Fu et al. (2026). AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. https://arxiv.org/abs/2505.24298. arXiv:2505.24298.

[30] Hao Bai et al. (2024). DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning. https://arxiv.org/abs/2406.11896. arXiv:2406.11896.

[31] Bowen Jin et al. (2025). Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. https://arxiv.org/abs/2503.09516. arXiv:2503.09516.

[32] Zehan Qi et al. (2025). WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. https://arxiv.org/abs/2411.02337. arXiv:2411.02337.

[33] Xiao Yu et al. (2025). Dyna-Mind: Learning to Simulate from Experience for Better AI Agents. https://arxiv.org/abs/2510.09577. arXiv:2510.09577.

[34] Xu et al. (2026). Polar: Agentic RL on Any Harness at Scale. arXiv preprint arXiv:2605.24220.

[35] Shunyu Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. https://arxiv.org/abs/2210.03629. arXiv:2210.03629.

[36] Guanzhi Wang et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. https://arxiv.org/abs/2305.16291. arXiv:2305.16291.

[37] Qingyun Wu et al. (2023). AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. https://arxiv.org/abs/2308.08155. arXiv:2308.08155.

[38] Zijian Wu et al. (2025). MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use. https://arxiv.org/abs/2509.24002. arXiv:2509.24002.

[39] Woosuk Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. https://arxiv.org/abs/2309.06180. arXiv:2309.06180.

[40] Feng et al. (2025). Group-in-Group Policy Optimization for LLM Agent Training. arXiv preprint arXiv:2505.10978.

[41] Qiying Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. https://arxiv.org/abs/2503.14476. arXiv:2503.14476.

[42] Ibragim Badertdinov et al. (2026). SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale. https://arxiv.org/abs/2602.23866. arXiv:2602.23866.

[43] Jingxu Xie et al. (2026). AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents. https://arxiv.org/abs/2506.14205. arXiv:2506.14205.

[44] Zhou et al. (2025). Proposer-Agent-Evaluator (PAE): Autonomous Skill Discovery for Foundation Model Internet Agents. In ICML. https://arxiv.org/abs/2412.13194.

[45] Zhipu AI (2026). ZClawBench. https://huggingface.co/datasets/zai-org/ZClawBench. Accessed: 2026-07-20.

[46] An Yang et al. (2025). Qwen3 Technical Report. https://arxiv.org/abs/2505.09388. arXiv:2505.09388.

[47] MiniMax (2026). MiniMax-M2.5. https://github.com/MiniMax-AI/MiniMax-M2.5. Accessed: 2026-07-20.

[48] Guo et al. (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature. 645(8081). pp. 633–638. doi:10.1038/s41586-025-09422-z. http://dx.doi.org/10.1038/s41586-025-09422-z.

[49] Zhihong Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. https://arxiv.org/abs/2402.03300. arXiv:2402.03300.

[50] Rui Yang et al. (2026). OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents. https://arxiv.org/abs/2606.02031. arXiv:2606.02031.

[51] Xinyuan Wang et al. (2025). OpenCUA: Open Foundations for Computer-Use Agents. https://arxiv.org/abs/2508.09123. arXiv:2508.09123.

[52] Tao Ge et al. (2026). Synthetic Computers at Scale for Long-Horizon Productivity Simulation. https://arxiv.org/abs/2604.28181. arXiv:2604.28181.

[53] Hao Bai et al. (2026). WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks. https://arxiv.org/abs/2601.02439. arXiv:2601.02439.

[54] OpenAI (2025). Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/. Accessed: 2025-09-17.

[55] Browser Use (2026). Stealth Browsers for AI Agents. https://browser-use.com/stealth-browsers.

[56] Kimi Team et al. (2026). Kimi K2.5: Visual Agentic Intelligence. https://arxiv.org/abs/2602.02276. arXiv:2602.02276.

[57] Hongliang He et al. (2024). WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. https://arxiv.org/abs/2401.13919. arXiv:2401.13919.

[58] Awadallah et al. (2025). Fara-7B: An Efficient Agentic Model for Computer Use. arXiv:2511.19663.

[59] Yiheng Xu et al. (2025). AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials. https://arxiv.org/abs/2412.09605. arXiv:2412.09605.

[60] OpenAI (2025). Introducing OpenAI o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/.

[61] OpenAI (2024). Hello GPT-4o. https://openai.com/index/hello-gpt-4o/. Accessed: 2024-09-28.

[62] Xingyao Wang et al. (2024). Executable Code Actions Elicit Better LLM Agents. https://arxiv.org/abs/2402.01030. arXiv:2402.01030.

[63] ZeroClaw (2025). ZeroClaw: The Ultra-Lightweight AI Agent Runtime. https://zeroclaw.net/. Accessed: 2026-07-20.

[64] Anthropic (2026). Introducing Claude 4.6. https://www.anthropic.com/news/claude-opus-4-6. Accessed: 2026-07-20.

[65] Yanzhao Zhang et al. (2025). Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. https://arxiv.org/abs/2506.05176. arXiv:2506.05176.

[66] Brandon Trabucco et al. (2025). InSTA: Towards Internet-Scale Training For Agents. arXiv:2502.06776.