title: "RExBench: Coding agents hit 33% success on research tasks" slug: "rexbench-coding-agents-hit-33-success-on-research-tasks" published: "2026-07-29" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-07-29" aiActArticle50: "compliant" humanView: "https://agentry.news/research/rexbench-coding-agents-hit-33-success-on-research-tasks" agentView: "https://agentry.news/agent/rexbench-coding-agents-hit-33-success-on-research-tasks"
Researchers at the Association for Computational Linguistics published RExBench, a benchmark evaluating 12 LLM agents on their ability to autonomously implement extensions of real research papers, fin
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
Researchers evaluated 12 large language model coding agents on their ability to autonomously implement realistic research extensions and found the best performer achieved only a 33% success rate without substantial human guidance, according to a paper published in the Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics Association for Computational Linguistics.
The RExBench benchmark, authored by Nicholas Edwards, Yukyung Lee, Yujun Audrey Mao, Yulu Qin, Sebastian Schuster, and Najoung Kim, builds on 12 research papers with realistic domain-expert-written instructions for extending each work. The evaluation uses automatic execution-based assessment rather than human review, allowing researchers to measure agent performance on concrete, reproducible tasks arXiv.
The benchmark represents a significant expansion of evaluation methodology beyond earlier agent benchmarks, addressing a gap in measuring whether coding agents can handle the kind of incremental research work that typically falls to graduate students and research engineers.
All 12 agents failed to autonomously implement the majority of extensions evaluated. Even when researchers provided additional human-written hints to guide the agents toward solutions, the best-performing agent could not exceed 44% success, demonstrating that the task remains substantially beyond current autonomous agent capabilities Association for Computational Linguistics.
The 33% baseline success rate—achieved without external hints—indicates that coding agents struggle with tasks requiring understanding of research context, architectural decisions, and integration with existing codebases, all common requirements in academic extension work.
The findings highlight a critical limitation in the agent economy: while agents can execute isolated coding tasks, translating research intent into working implementations remains difficult. This matters to labs and companies evaluating agents for research velocity, and to developers building tools that agents use.
The RExBench evaluation methodology itself represents a shift toward more realistic agent assessment. By grounding evaluation in actual research papers and using automatic execution as the success metric, the authors move beyond synthetic or gaming-prone benchmarks. This approach aligns with industry movement toward benchmarks like SWE-Bench that measure agents on tangible, verifiable outcomes rather than human judgment or simulated environments.
The work also underscores why coding agent companies continue to position human-in-the-loop systems as the near-term reality: fully autonomous research implementation is not yet achievable at scale.