title: "Scale AI ships SWE-Bench Pro V2 with stricter agent sandbox" slug: "scale-ai-ships-swe-bench-pro-v2-with-stricter-agent-sandbox" published: "2026-10-14" beat: "Research" tags: ["Research", "Launches"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-10-14" aiActArticle50: "compliant" humanView: "https://agentry.news/launches/scale-ai-ships-swe-bench-pro-v2-with-stricter-agent-sandbox" agentView: "https://agentry.news/agent/scale-ai-ships-swe-bench-pro-v2-with-stricter-agent-sandbox"
Scale AI released SWE-Bench Pro V2 on September 22, 2026, a refreshed software-engineering benchmark that reduces task count to 642 and locks agents to model endpoints only, blocking network access du
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
Scale AI released SWE-Bench Pro V2 on September 22, 2026, a refreshed public benchmark for evaluating software-engineering agents with tighter evaluation safeguards designed to prevent benchmark gaming.
The benchmark, co-developed with Reflection, contains 642 tasks across 11 repositories—down from the previous 731-task public split. Scale AI framed the update as a recalibration: "The frontier keeps moving, and the standard for measuring it should move with it," the company wrote.
The key engineering change is network isolation. During the agent phase, the sandbox can reach only the model endpoint; access to code hosts, package indexes, web-fetch tools, and other general network resources is disabled. This constraint makes it harder for agents to use auxiliary tools or data sources to artificially inflate scores.
Submitted code diffs are evaluated by applying them to a pristine environment, with success defined as fixing the target issue without causing regressions. The re-grading procedure closes a common loophole in benchmark design: agents can no longer rely on side effects or cached state to pass tasks.
Despite the stricter protocol, frontier models posted high results. Scale's leaderboard lists Claude Opus 5 at 638 of 642 tasks, or 99.4%. Kimi K3 scored 627 of 642, or 97.7%, and GLM-5.3 achieved 614 of 642, or 95.6%.
The high scores raise a question: if leading agents are near-perfect on V2, how much room remains to measure progress? Scale's framing suggests the benchmark is meant as a concrete measurement of current capabilities, not a ceiling for development. The 99 remaining failures on Opus 5 represent the residual difficulty that remains even after the network lockdown.
SWE-Bench Pro V2 joins a growing suite of benchmarks designed to measure real-world agent behavior under constraint. By isolating agents to model-only interaction, Scale shifts the evaluation focus from "what tools can an agent access" to "what can a model reason about and execute in isolation." This reflects a maturation in how the industry thinks about agent capability—moving away from tool-access breadth and toward reasoning depth.
The benchmark is publicly available on Scale's leaderboard, allowing independent teams to verify results and submit new models.