SWE-CoWork Benchmark Exposes Coding Agent Gaps
A New Benchmark for Agents as Coworkers
Researchers introduced SWE-CoWork, a benchmark that places coding agents inside simulated software organizations, in a paper submitted to arXiv on 15 November 2026 (arXiv). The paper, authored by Xinping Lei, Minghao Liu, Ruijie Zhu and 23 co-authors, describes the project as a way "to benchmark coding agents as digital employees inside software organizations" (arXiv). The benchmark spans 101 development assignments across 101 open-source repositories, covering 11 programming languages and 12 domains, with each task embedding an agent among 26 simulated digital employees for a total of 2,626 employee profiles (agihunt.info).
Why This Matters for Builders and Buyers
For teams evaluating whether coding agents can be deployed as autonomous contributors rather than narrow code-completion tools, SWE-CoWork's design matters because it tests organizational context — bounded responsibilities, colleague handoffs, and fixed test contracts — rather than isolated coding puzzles. The gap the researchers measured, between high scores on narrow test passes and low scores on fully completed assignments, is the central signal for anyone deciding how much autonomy to grant an agent in a real engineering workflow.
The Results
Across the 2,699 F2P (fail-to-pass) test nodes and 562 P2P (pass-to-pass) test nodes in the benchmark, Claude-Opus-5.5 led with 63.84% of F2P tests passed, 93.24% of P2P tests preserved, but only 6.93% of full assignments completed (Hugging Face). GPT-6 Astra followed closely, passing 62.91% of F2P tests and preserving 89.15% of P2P tests, while completing 7.92% of assignments — the highest full-completion rate recorded. Lower in the ranking, Seed Evolving passed 39.09% of F2P tests but completed just 0.99% of assignments, and Qwen3.8-Max passed 35.70% of F2P tests while completing zero assignments outright. GLM 5.3 passed 31.27% of F2P tests with 0.99% assignment completion, and DeepSeek V4 Pro trailed with 13.04% of F2P tests passed and no completed assignments (agihunt.info).
Reading the Gap
The pattern across all six reported models is consistent: strong performance on isolated, requirement-focused tests does not translate into finishing an entire assignment end to end. Even the top two models, which passed roughly 63% of individual F2P tests, fully completed only 7-8% of the 101 assignments (arXiv). Discussion of the findings has circulated among AI researchers on social platforms since the paper's release (@rohanpaul_ai, @omarsar0), with the dataset itself published for public evaluation (Hugging Face).


