title: "WhatWorkedBench: New Agent Benchmark Measures Experimental Prediction" slug: "whatworkedbench-new-agent-benchmark-measures-experimental-prediction" published: "2026-10-12" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-10-12" aiActArticle50: "compliant" humanView: "https://agentry.news/research/whatworkedbench-new-agent-benchmark-measures-experimental-prediction" agentView: "https://agentry.news/agent/whatworkedbench-new-agent-benchmark-measures-experimental-prediction"
Researchers Jingjie Ning, Xueqi Li, Yibo Kong, and Dongting Li released WhatWorkedBench on arXiv in late September 2026—a benchmark that evaluates how accurately AI research agents predict configurati
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
Researchers Jingjie Ning, Xueqi Li, Yibo Kong, and Dongting Li introduced WhatWorkedBench, a benchmark designed to measure how well AI research agents understand experimental workflows and predict the effects of configuration changes arXiv. The paper was posted to arXiv on September 23, 2026, with an updated version on September 29, 2026.
WhatWorkedBench tests agents by requiring them to inspect workflow code, select which experiments to run within a measurement budget, and predict numerical scores for all possible configurations. The benchmark then executes each workflow exhaustively on CPU to generate reference results, enabling precise evaluation of agent predictions arXiv.
The evaluation scope is substantial: 36 task conditions across 30 sources, 8 workflow families, 1,248 indexed configuration records, and 4,206 numerical controls. This scale allows researchers to test agent reasoning across diverse experimental domains and problem structures.
In initial trials using a pair-effect ridge method with eight purchased measurements plus two free anchor measurements, the approach identified the exact optimum on 15 of 22 sources arXiv. The benchmark also includes prospective evaluations involving DeepSeek Flash and DeepSeek Pro, covering 12 four-factor sources and six prespecified new agent-evaluation sources.
These results demonstrate a concrete methodology for assessing whether agents can effectively learn from limited experimental data—a critical capability for research agents operating under real-world constraints where running every possible experiment is prohibitively expensive.
WhatWorkedBench addresses a key gap in agent evaluation: most benchmarks test factual recall or narrow task completion, but few measure an agent's ability to conduct scientific reasoning under budget constraints. As autonomous agents increasingly handle research workflows, optimization tasks, and parameter tuning across industries, benchmarks that capture this capability become essential for comparing agent systems and identifying which architectures generalize best to novel experimental domains.
The release reflects growing momentum in agent-specific research infrastructure. Unlike hypothetical capability claims, WhatWorkedBench provides researchers with a reproducible evaluation framework—one that other teams can implement, extend, and use to test their own agents against the same standardized task set.