Agents Show 'Real Discovery' in SAE Research but Trail Human Experts

Researchers at Harbin Institute of Technology published a benchmark study on September 8, 2026, testing whether frontier AI agents can operate as autonomous scientists conducting mechanistic interpretability research arXiv.

Benchmark Design and Scope

The study, authored by Yuqiao Tan, Shizhu He, Jun Zhao, and Kang Liu, introduced SAEScientist-Bench to measure agent capability in using Sparse Autoencoders (SAEs) for autonomous discovery tasks arXiv. The benchmark evaluated performance across 131K+ features in the Gemma Scope dictionary, a large-scale SAE dictionary designed for mechanistic interpretability work on the Gemma-2-9B-IT model arXiv.

The research targets a specific frontier in agent autonomy: whether agents can move beyond task execution into hypothesis formation, experimental design, and scientific discovery—tasks that traditionally require human expertise and judgment.

Key Findings

The paper reports that frontier agents showed "real discovery ability" on the autonomous research tasks but remained substantially below expert baselines overall arXiv. On certain metrics—particularly feature separation—agents approached expert-level performance. However, they lagged meaningfully on causal steering tasks, which require agents to manipulate model behavior in targeted ways and validate the causal relationships they discover arXiv.

This mixed result suggests agents can identify patterns and generate discovery-oriented outputs but struggle with the validation and causal reasoning layers that distinguish robust science from pattern-matching.

Implications for Agent Research

The benchmark addresses a practical question shaping the agent economy: can autonomous systems reduce the human burden in interpretability research, a field currently bottlenecked by the scarcity of expert researchers? The finding—that agents possess some discovery signal but lack the depth to match experts—implies near-term roles in augmentation rather than replacement.

The work also establishes a concrete, measurable benchmark for evaluating progress in agent reasoning and scientific autonomy, contrasting with the vague capability claims common in agent product announcements. By anchoring agent evaluation to a defined task (mechanistic discovery) and a specific model (Gemma-2-9B-IT) with a large feature dictionary, the researchers created a replicable standard arXiv.

Developer and Research Community Relevance

The paper supports growing interest in agent evaluation frameworks specific to knowledge work domains. As AI agents move beyond transactional tasks into research and reasoning roles, benchmarks that measure actual discovery—not just task completion—become critical infrastructure for distinguishing capability from simulation.

The arXiv preprint and associated resources are available for peer review and reproduction, establishing a foundation for future work on agent-assisted interpretability research arXiv.