title: "Coding agents fail to break 50% on scientific software benchmark" slug: "coding-agents-fail-to-break-50-on-scientific-software-benchmark" published: "2026-09-03" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-09-03" aiActArticle50: "compliant" humanView: "https://agentry.news/research/coding-agents-fail-to-break-50-on-scientific-software-benchmark" agentView: "https://agentry.news/agent/coding-agents-fail-to-break-50-on-scientific-software-benchmark"
Researchers released SWE-bench Science on August 20, 2026, revealing that even the best-performing agent, Claude Code with Opus-5, achieved only 47.90% pass@1 on a repository-level benchmark of scient
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
Researchers released SWE-bench Science on August 20, 2026, exposing a stark performance ceiling for autonomous coding agents on scientific software engineering tasks. The benchmark, published via arXiv, evaluated agents on 119 repository-level tasks drawn from 98 GitHub repositories spanning 20 scientific domains.
SWE-bench Science extends prior software engineering benchmarks into specialized scientific domains, targeting a critical gap: whether agents can handle the domain-specific invariants and requirements embedded in scientific codebases. The 119 tasks represent real-world challenges from active repositories in fields requiring rigorous numerical validation, mathematical correctness, and domain knowledge beyond generic programming patterns.
The best-performing agent, Claude Code with Opus-5 (max), reached a pass@1 score of 47.90%—below the 50% threshold arXiv and the official benchmark site. This means even state-of-the-art agentic systems failed to solve nearly half of the tasks on their first attempt, a significant limitation for any deployment scenario requiring high reliability.
Scientific software poses distinct challenges that general-purpose agents struggle to navigate. Tasks involve validating physical correctness, handling numerical precision requirements, and working within established scientific frameworks and libraries. An agent must not only write syntactically correct code but also understand the domain logic—whether in physics simulation, statistical analysis, or computational chemistry—that determines whether a solution is actually valid.
The gap between 47.90% and any production-grade reliability threshold suggests that autonomous agents still require close human oversight or hybrid workflows when deployed on scientific projects. This contrasts with narrower coding domains where agent performance has approached or exceeded 50% in prior benchmarks.
For organizations considering agentic workflows in scientific computing—pharmaceuticals, climate research, materials science—these results signal that agents function better as assistants than autonomous solvers. The benchmark outcome reinforces a broader pattern: agent capabilities remain uneven across domains, with performance degrading sharply when domain-specific correctness criteria dominate.
Further research into agent reasoning on scientific invariants, improved training on domain-specific repositories, and hybrid human-agent evaluation frameworks will likely be necessary before widespread adoption in scientific software development.