FIRE-Bench: AI Agents Fail to Rediscover Science at Scale
Researchers introduced FIRE-Bench (Full-cycle Insight Rediscovery Evaluation) in February 2026 to objectively measure how well AI agents can rediscover established, verifiable scientific findings from recent machine learning research arXiv.
Benchmark Design and Scope
FIRE-Bench evaluates agents on 30 core LLM-behavior tasks, extended to a 10-task cross-domain set, each grounded in peer-reviewed discoveries that agents should be able to independently reproduce arXiv. The benchmark deliberately avoids measuring novel claim generation—instead focusing on whether agents can systematically work through the scientific method to confirm known results. This distinction matters: rediscovery is tractable, falsifiable, and benchmarkable in ways that pure novelty is not.
State-of-the-Art Performance Falls Short
Across frontier agent backbones including GPT-5, the strongest performers achieve less than 50% F1 success, with significant run-to-run variance arXiv. This means that even the most capable agents fail to consistently reproduce results they should, in principle, be able to find and verify.
The consistency problem is critical: high variance suggests agents are not reliably applying the reasoning steps needed for scientific work, even when they possess the underlying capabilities.
Where Agents Break Down
Failures cluster in two cognitive bottlenecks: research planning and evidence-to-conclusion reasoning. Agents struggle most when designing appropriate experimental controls and translating observed evidence into valid conclusions arXiv. Low-level coding tasks—where LLMs tend to be strong—are not the constraint.
This points to a fundamental gap: current agent architectures excel at syntax and code generation but falter when reasoning must span multiple interdependent steps with domain-specific constraints. The failures reveal not capability ceilings, but reasoning fragility under methodological complexity.
Implications for Agent Deployments
The benchmark suggests caution for real-world scientific agent deployment in high-stakes domains. If agents cannot reliably rediscover known findings—where ground truth exists and verification is straightforward—their trustworthiness on novel problems in biology, chemistry, or materials science remains questionable.
FIRE-Bench's focus on rediscovery provides a practical foundation for iterative improvement. Developers can now target the specific reasoning gaps the benchmark exposes: planning robustness, control design, and evidence synthesis. This concrete diagnostic capability makes FIRE-Bench a rare research contribution—not just a problem statement, but a measurable path toward better agent reasoning.
As agent deployment accelerates across enterprises, benchmarks that expose failure modes in controlled settings become essential safety infrastructure.
