GPT-5, Claude Opus 4.1 score 23% on harder coding benchmark
Scale AI published results from SWE-Bench Pro V2, a harder variant of its coding benchmark, showing that the industry's top two models score roughly one-quarter as well as they do on the original test Scale Labs.
The Gap: Easy vs. Hard
OpenAI's GPT-5 achieved 23.3% accuracy and Anthropic's Claude Opus 4.1 reached 23.1% on the public-set version of SWE-Bench Pro V2 Scale Labs. Both models routinely exceed 70% on SWE-Bench Verified, the original benchmark, indicating that the Pro V2 test introduces substantially harder constraints—longer code contexts, more complex dependencies, or edge cases that require deeper reasoning.
The public leaderboard contains 731 instances and is hosted at Scale Labs. Scale AI has separately described SWE-Bench Pro V2 as a refreshed split spanning 642 tasks across 11 repositories, though the exact alignment between the public leaderboard's 731-instance set and the 642-task full release remains distinct in Scale's documentation.
What It Means for Agent Coding
The stark drop-off between Verified and Pro V2 performance suggests that while frontier models excel at typical coding problems, they struggle with the kind of real-world software engineering tasks that demand integration across multiple files, understanding of unfamiliar codebases, or recovery from incomplete specifications. For developers building autonomous coding agents, the results underscore that relying on raw model accuracy alone risks deploying systems that fail on harder, more realistic engineering work.
Both GPT-5 and Claude Opus 4.1 remain the best-performing models on Pro V2, but the 23% range leaves substantial room for improvement across the field. Smaller or specialized models may show different trade-offs, and ensemble or iterative approaches—common in production agent systems—could close the gap by allowing models to refine solutions over multiple passes.
Benchmark Context
SWE-Bench Verified focuses on real GitHub issues with known fixes; Pro V2 raises the difficulty by introducing a larger, more diverse test set with variants designed to probe edge cases and catch overfitting to the original benchmark. This mirrors the evolution of other ML benchmarks, where harder versions routinely expose weaknesses in models that seemed strong on earlier tests.
Scale AI's decision to publish a harder benchmark reflects pressure within the industry to move beyond narrow metrics and test genuine capability on problems closer to production engineering. As agents increasingly handle code generation and debugging in live environments, benchmarks that separate hype from measurable performance become essential for enterprise adoption and safety.