---
title: "Scale AI ships SWE-Bench Pro V2 with stricter agent sandbox"
slug: "scale-ai-ships-swe-bench-pro-v2-with-stricter-agent-sandbox"
published: "2026-10-14"
beat: "Research"
tags: ["Research", "Launches"]
creator: "Agentry Newsroom"
editor: "Susanne Sperling, Editor — Human in the Loop"
tools: ["Claude (Anthropic)", "Perplexity Sonar"]
creativeWorkStatus: "verified"
dateReviewed: "2026-10-14"
aiActArticle50: "compliant"
humanView: "https://agentry.news/launches/scale-ai-ships-swe-bench-pro-v2-with-stricter-agent-sandbox"
agentView: "https://agentry.news/agent/scale-ai-ships-swe-bench-pro-v2-with-stricter-agent-sandbox"
---# Scale AI ships SWE-Bench Pro V2 with stricter agent sandbox

> Scale AI released SWE-Bench Pro V2 on September 22, 2026, a refreshed software-engineering benchmark that reduces task count to 642 and locks agents to model endpoints only, blocking network access du

*Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. [AI policy](/ai-policy).*

Scale AI released SWE-Bench Pro V2 on [September 22, 2026](https://x.com/scale_AI/status/2102451007041282353), a refreshed public benchmark for evaluating software-engineering agents with tighter evaluation safeguards designed to prevent benchmark gaming.

The benchmark, [co-developed with Reflection](https://labs.scale.com/leaderboard/swe_bench_pro_public_v2), contains [642 tasks across 11 repositories](https://labs.scale.com/blog/swe-bench-pro-v2)—down from the previous 731-task public split. Scale AI framed the update as a recalibration: "The frontier keeps moving, and the standard for measuring it should move with it," [the company wrote](https://x.com/scale_AI/status/2102451007041282353).

## Locked Sandbox Prevents Network Access

The key engineering change is network isolation. [During the agent phase, the sandbox can reach only the model endpoint](https://labs.scale.com/blog/swe-bench-pro-v2); access to code hosts, package indexes, web-fetch tools, and other general network resources is disabled. This constraint makes it harder for agents to use auxiliary tools or data sources to artificially inflate scores.

[Submitted code diffs are evaluated by applying them to a pristine environment](https://labs.scale.com/blog/swe-bench-pro-v2), with success defined as fixing the target issue without causing regressions. The re-grading procedure closes a common loophole in benchmark design: agents can no longer rely on side effects or cached state to pass tasks.

## Leading Models Report Near-Perfect Scores

Despite the stricter protocol, frontier models posted high results. [Scale's leaderboard lists Claude Opus 5 at 638 of 642 tasks, or 99.4%](https://labs.scale.com/leaderboard/swe_bench_pro_public_v2). [Kimi K3 scored 627 of 642, or 97.7%, and GLM-5.3 achieved 614 of 642, or 95.6%](https://labs.scale.com/leaderboard/swe_bench_pro_public_v2).

The high scores raise a question: if leading agents are near-perfect on V2, how much room remains to measure progress? Scale's framing suggests the benchmark is meant as a **concrete measurement of current capabilities**, not a ceiling for development. The 99 remaining failures on Opus 5 represent the residual difficulty that remains even after the network lockdown.

## Implications for Agent Evaluation

SWE-Bench Pro V2 joins a growing suite of benchmarks designed to measure real-world agent behavior under constraint. By isolating agents to model-only interaction, Scale shifts the evaluation focus from "what tools can an agent access" to "what can a model reason about and execute in isolation." This reflects a maturation in how the industry thinks about agent capability—moving away from tool-access breadth and toward reasoning depth.

The benchmark is [publicly available on Scale's leaderboard](https://labs.scale.com/leaderboard/swe_bench_pro_public_v2), allowing independent teams to verify results and submit new models.