CUA-SWE Benchmark Tests Computer-Use Agents on Code Tasks

A research team introduced CUA-SWE, a benchmark and evaluation framework designed to test whether computer-use agents can perform real software-engineering work arXiv. The paper, authored by Prince Zizhuang Wang, Chenhao Liang, Zelong Xu, and six collaborators, was submitted to arXiv on September 26, 2026, and describes CUA-SWE as the first systematic benchmark to measure agent performance across practical coding tasks.

What the Benchmark Measures

CUA-SWE contains 105 tasks spanning four software-engineering domains arXiv. Each task requires agents to demonstrate multiple capabilities: modifying code and configuration files, executing commands in terminals, interacting with running software through graphical interfaces, and inspecting visual feedback to verify that changes work correctly arXiv.

The benchmark's design reflects a key challenge in agent evaluation: unlike traditional machine-learning benchmarks that measure classification or generation, software-engineering tasks require agents to operate across multiple interfaces and verify their own work. A task may require an agent to read a bug report, locate the relevant code, make a fix, run tests, and confirm the output matches expectations—all through visual observation and command execution.

Verification and Task Design

Each of the 105 tasks includes deterministic, task-specific tests intended to verify whether the requested behavior was correctly implemented and whether existing functionality was preserved arXiv. This means evaluators can programmatically confirm whether an agent succeeded, rather than relying on subjective assessment.

The benchmark dataset is publicly available on Hugging Face, enabling researchers to evaluate their own agent implementations against the same standardized tasks Hugging Face.

Why This Matters for the Agent Economy

CUA-SWE addresses a concrete gap in agent research: while benchmarks exist for language understanding and code generation, few rigorously measure an agent's ability to *act* in real development environments. As enterprises consider deploying agents to handle code maintenance, testing, and bug fixing, benchmarks like CUA-SWE provide verifiable evidence of what agents can and cannot do.

The paper signals growing maturity in computer-use agent evaluation. Rather than asking "can agents understand code?" the benchmark asks "can agents successfully modify, test, and verify code changes using only visual interfaces and terminal commands?"—a more realistic assessment of production readiness.

Full paper and supplementary materials are available on arXiv.