WhatWorkedBench: New Agent Benchmark Measures Experimental Prediction
Researchers Jingjie Ning, Xueqi Li, Yibo Kong, and Dongting Li introduced WhatWorkedBench, a benchmark designed to measure how well AI research agents understand experimental workflows and predict the effects of configuration changes arXiv. The paper was posted to arXiv on September 23, 2026, with an updated version on September 29, 2026.
How the Benchmark Works
WhatWorkedBench tests agents by requiring them to inspect workflow code, select which experiments to run within a measurement budget, and predict numerical scores for all possible configurations. The benchmark then executes each workflow exhaustively on CPU to generate reference results, enabling precise evaluation of agent predictions arXiv.
The evaluation scope is substantial: 36 task conditions across 30 sources, 8 workflow families, 1,248 indexed configuration records, and 4,206 numerical controls. This scale allows researchers to test agent reasoning across diverse experimental domains and problem structures.
Early Results and Prospective Evaluation
In initial trials using a pair-effect ridge method with eight purchased measurements plus two free anchor measurements, the approach identified the exact optimum on 15 of 22 sources arXiv. The benchmark also includes prospective evaluations involving DeepSeek Flash and DeepSeek Pro, covering 12 four-factor sources and six prespecified new agent-evaluation sources.
These results demonstrate a concrete methodology for assessing whether agents can effectively learn from limited experimental data—a critical capability for research agents operating under real-world constraints where running every possible experiment is prohibitively expensive.
Relevance to the Agent Economy
WhatWorkedBench addresses a key gap in agent evaluation: most benchmarks test factual recall or narrow task completion, but few measure an agent's ability to conduct scientific reasoning under budget constraints. As autonomous agents increasingly handle research workflows, optimization tasks, and parameter tuning across industries, benchmarks that capture this capability become essential for comparing agent systems and identifying which architectures generalize best to novel experimental domains.
The release reflects growing momentum in agent-specific research infrastructure. Unlike hypothetical capability claims, WhatWorkedBench provides researchers with a reproducible evaluation framework—one that other teams can implement, extend, and use to test their own agents against the same standardized task set.