Coding agents fail 94.6% of whole-repo migrations in new benchmark

A new benchmark reveals sharp limits in coding agents' ability to handle large-scale, whole-repository migrations. Researchers posted *SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?* to arXiv on August 25, 2026, measuring performance across 20 distinct migration tasks using 520 total runs from eight frontier models.

Only 5.4% of Attempts Succeeded

Across all runs, only 28 passed all three validation stages, yielding a 5.4% overall success rate. The three-stage evaluation framework—which tested not just code compilation but also migration completeness and correctness—proved substantially harder than current agents can reliably clear. Claude Opus 5, the highest-scoring model, achieved 47.0/100, indicating that even frontier capabilities leave large gaps when tasked with long-horizon, interconnected refactoring work.

Why Whole-Repository Migrations Matter

The benchmark specifically targeted migrations that require agents to modify code across an entire codebase—not isolated files or functions. Real-world examples include upgrading a framework version, migrating a database schema, or refactoring an API contract throughout a large system. These tasks demand agents understand dependencies, track state changes across files, and validate that no breaking changes cascade through the repository.

The 20 migrations and 520 runs structure allowed researchers to isolate agent behavior on concrete, verifiable tasks. Unlike synthetic coding benchmarks that score on isolated snippets, SWE Refactor Bench measures whether an agent can complete a coherent, end-to-end refactoring that preserves functionality at scale.

Frontier Model Performance Gap

The 47.0/100 top score underscores a critical gap: even the best-performing model failed more often than it succeeded. Secondary coverage reports that across the eight models tested, the performance distribution was heavily skewed toward failure, with most models scoring substantially below the frontier leader.

This result has immediate implications for enterprise teams considering agent-based code migration tools. While agents excel at single-file refactoring and small-scale edits, the whole-repository scope—common in real infrastructure work—remains a domain where human oversight and verification remain essential.

Implications for Agent Development

The benchmark surfaces what researchers call a "migration completeness gap": agents often rewrite portions of code but fail to propagate changes fully through dependency chains. This suggests future improvements will need to focus on dependency tracking, multi-file validation, and longer-horizon planning—capabilities that require agents to maintain and query a model of the entire codebase state.

The arXiv paper joins a growing body of research documenting concrete capability ceilings in current agent architectures, particularly for tasks requiring sustained context and cross-file reasoning.