title: "TestJack audit finds 34% of coding benchmarks fail verification" slug: "testjack-audit-finds-34-of-coding-benchmarks-fail-verification" published: "2026-10-21" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-10-21" aiActArticle50: "compliant" humanView: "https://agentry.news/research/testjack-audit-finds-34-of-coding-benchmarks-fail-verification" agentView: "https://agentry.news/agent/testjack-audit-finds-34-of-coding-benchmarks-fail-verification"
Researchers at UC Berkeley and collaborating institutions released TestJack, an auditing framework that re-evaluated five coding benchmarks used to measure agent performance and found approximately 34
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
Researchers Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, and Dawn Song published TestJack, an auditing framework that systematically re-evaluates coding benchmarks used to measure agentic AI performance, finding widespread failures in how results are judged arXiv.
The team tested six frontier-model backends across five coding benchmarks—DeepSWE, SkillsBench, SWE-Marathon, SWE-bench Verified, and SWE-bench Pro—and discovered that approximately 34.4% of trials marked as correct violated the original task requirements arXiv. The audit caused measured resolution rates to collapse: across the benchmarks, overall performance fell from 50.6% to 33.2% when evaluator verdicts were re-examined arXiv.
The auditing method generates new test cases designed to detect violations of patch-level task requirements, retains only tests passed by ground-truth patches, and re-evaluates agent-generated solutions against this tightened standard. This adversarial approach surfaces cases where evaluators—often large language models themselves—incorrectly accepted solutions that departed from specification.
Coding benchmarks are primary tools for ranking agent models and justifying enterprise deployment decisions. If 34% of "passing" results are actually incorrect, the measured capabilities of frontier agents collapse substantially. The paper's October 2026 publication date indicates the issue is current across latest-generation systems, not a legacy problem.
The gap between 50.6% and 33.2% suggests that public claims about agent coding ability may overstate real performance by as much as 50%. For enterprises evaluating agentic coding tools—whether for software engineering, infrastructure automation, or vulnerability repair—these findings underscore the risk of relying on published benchmark scores without independent auditing.
TestJack does not name specific model providers whose agents failed auditing, nor does it report whether benchmark maintainers have acknowledged or disputed the findings. The study is a peer-reviewed arXiv preprint rather than a regulator report, court filing, or formal benchmark retraction. No corrected benchmark leaderboards or official statements from benchmark operators are documented in the available record.
The research signals a structural problem: evaluator design (especially when evaluators are themselves LLMs) can systematically overstate agent reliability. As enterprises scale agentic systems into production environments—autonomous trading, HR operations, legal research—the difference between 50% and 33% resolution accuracy translates directly to downstream errors, rework, and risk.