AI agents solve 67% of science tasks in new benchmark
Researchers released a benchmark for evaluating AI agents performing autonomous scientific discovery on October 31, 2026, providing the first systematic measurement of how well current systems can execute real-world lab experiments arXiv.
What the benchmark tested
Autonomous Science Bench, authored by Yibo Wen and Han Liu, contained six budgeted design–build–test–learn campaigns grounded in published experiments across biology, chemistry, and materials science. The researchers tested eight agent–model configurations across 144 total trials to measure success rates on concrete scientific tasks.
The strongest performer—Opus 5.5 with Claude Code—resolved 12 of 18 trials, a 67% success rate arXiv. However, performance varied sharply by task. The hardest campaign was solved in only 3 of 24 attempts, highlighting significant gaps in agent capability across different experimental domains. Across all trials, 31 of 144 passed both benchmark gates, indicating substantial room for improvement in agent autonomy for scientific work.
Why this matters for the agent economy
The benchmark directly addresses a core question in agent development: can systems reliably execute multi-step, real-world tasks without human intervention? Science offers an ideal proving ground because experiments have measurable outcomes and published protocols that can be replicated programmatically.
Results show agents can succeed at routine tasks but struggle with harder challenges, a pattern that mirrors findings in earlier benchmark work. The 67% best-case success rate reflects progress in agent reasoning and tool use, but the 3-of-24 failure rate on the hardest campaign suggests agents remain far from autonomous operation in complex or novel scientific scenarios.
What ships next
The benchmark itself is now open for use by researchers and developers building agent systems. As agent frameworks and models improve—particularly in reasoning, long-horizon planning, and error recovery—the benchmark can track whether these improvements translate to measurable gains in real-world task execution.
The research team's focus on budgeted, reproducible experiments positions this benchmark as a practical tool for evaluating agent systems before deployment in high-stakes scientific environments, where cost and safety matter.
Broader context
Autonomous science is emerging as a major application area for AI agents. Unlike chat-based or content-generation tasks, scientific agent deployment requires verifiable success on defined objectives—exactly what benchmarks like this one measure. The concrete pass/fail framing avoids the vagueness that has plagued earlier agent claims and grounds capability measurement in documented experimental outcomes.