---
title: "TestJack audit finds 34% of coding benchmarks fail verification"
slug: "testjack-audit-finds-34-of-coding-benchmarks-fail-verification"
published: "2026-10-21"
beat: "Research"
tags: ["Research"]
creator: "Agentry Newsroom"
editor: "Susanne Sperling, Editor — Human in the Loop"
tools: ["Claude (Anthropic)", "Perplexity Sonar"]
creativeWorkStatus: "verified"
dateReviewed: "2026-10-21"
aiActArticle50: "compliant"
humanView: "https://agentry.news/research/testjack-audit-finds-34-of-coding-benchmarks-fail-verification"
agentView: "https://agentry.news/agent/testjack-audit-finds-34-of-coding-benchmarks-fail-verification"
---# TestJack audit finds 34% of coding benchmarks fail verification

> Researchers at UC Berkeley and collaborating institutions released TestJack, an auditing framework that re-evaluated five coding benchmarks used to measure agent performance and found approximately 34

*Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. [AI policy](/ai-policy).*

Researchers Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, and Dawn Song published TestJack, an auditing framework that systematically re-evaluates coding benchmarks used to measure agentic AI performance, finding widespread failures in how results are judged [arXiv](https://arxiv.org/abs/2610.10619).

The team tested six frontier-model backends across five coding benchmarks—DeepSWE, SkillsBench, SWE-Marathon, SWE-bench Verified, and SWE-bench Pro—and discovered that approximately **34.4% of trials marked as correct violated the original task requirements** [arXiv](https://arxiv.org/abs/2610.10619). The audit caused measured resolution rates to collapse: across the benchmarks, overall performance fell from 50.6% to 33.2% when evaluator verdicts were re-examined [arXiv](https://arxiv.org/abs/2610.10619).

## How TestJack Works

The auditing method generates new test cases designed to detect violations of patch-level task requirements, retains only tests passed by ground-truth patches, and re-evaluates agent-generated solutions against this tightened standard. This adversarial approach surfaces cases where evaluators—often large language models themselves—incorrectly accepted solutions that departed from specification.

## Why This Matters for the Agent Economy

Coding benchmarks are primary tools for ranking agent models and justifying enterprise deployment decisions. If 34% of "passing" results are actually incorrect, the measured capabilities of frontier agents collapse substantially. The paper's October 2026 publication date indicates the issue is current across latest-generation systems, not a legacy problem.

The gap between 50.6% and 33.2% suggests that public claims about agent coding ability may overstate real performance by as much as 50%. For enterprises evaluating agentic coding tools—whether for software engineering, infrastructure automation, or vulnerability repair—these findings underscore the risk of relying on published benchmark scores without independent auditing.

## Broader Implications

TestJack does not name specific model providers whose agents failed auditing, nor does it report whether benchmark maintainers have acknowledged or disputed the findings. The study is a peer-reviewed arXiv preprint rather than a regulator report, court filing, or formal benchmark retraction. No corrected benchmark leaderboards or official statements from benchmark operators are documented in the available record.

The research signals a structural problem: evaluator design (especially when evaluators are themselves LLMs) can systematically overstate agent reliability. As enterprises scale agentic systems into production environments—autonomous trading, HR operations, legal research—the difference between 50% and 33% resolution accuracy translates directly to downstream errors, rework, and risk.