Real-SWE benchmark: top coding agent solves only 38.8% of tasks
Real-SWE released a full leaderboard update on September 13, 2026, benchmarking coding agents against private enterprise codebases, and the results underscore a hard ceiling on real-world performance: even the leading agent, Fable 5.1 on Claude Code, resolved only 38.8% of tasks explainx.ai.
The benchmark evaluates agents tasked with repairing code in actual private repositories—a far harder problem than public GitHub datasets. Every model-harness combination tested still failed more than 60% of repair attempts, signaling that autonomous code agents remain unreliable for high-stakes enterprise deployments.
Leaderboard Results
Fable 5.1 on Claude Code led at 38.8% resolution rate aimodelreport.com. The second-place entry, GPT-6 Astra on Codex CLI, achieved 33.8%, followed by Gemini 3.8 Flash at 31.2% santageai.com. The gap between first and second place narrows the further down the leaderboard, reflecting the relative immaturity of the space: no agent has broken through a 40% threshold on real private-codebase repairs.
Real-World Implications
The Real-SWE benchmark measures something closer to production reality than synthetic benchmarks: agents must navigate unfamiliar codebases, understand business logic, and commit fixes that don't introduce new failures. The 60%-failure rate means enterprises deploying coding agents must expect three out of five repair attempts to require human review or reversion.
This has direct consequences for companies betting on autonomous code agents for internal development velocity. Vendors citing coding agents as labor-replacement tools face a credibility test: at sub-40% success rates on well-scoped repair tasks, scaling to full-cycle feature development remains far beyond current capabilities.
What the Benchmark Measures
Real-SWE differs from competitions like SWE-bench, which use public repositories and simpler task definitions. Real-SWE's private-codebase focus mirrors the actual deployment context: large enterprises with proprietary systems, legacy integrations, and business-specific patterns that no public training data captures. Agents must infer intent from code alone, without pull request descriptions or issue tracker context.
The September 13 update aiwiki.ai marks one of the first comprehensive comparisons across frontier models on this harder variant. The consistency of results—no agent approaching 50%—suggests the bottleneck is not model scale but the inherent difficulty of code repair in unfamiliar, production codebases.
For development teams, the takeaway is clear: coding agents are useful assistants for well-understood problems within familiar code, but not yet reliable enough for independent deployment on novel private systems.