title: "SWE-Shift: All 17 Coding Agents Falter When Tasks Shift Mid-Run" slug: "swe-shift-all-17-coding-agents-falter-when-tasks-shift-mid-run" published: "2026-10-28" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-10-28" aiActArticle50: "compliant" humanView: "https://agentry.news/research/swe-shift-all-17-coding-agents-falter-when-tasks-shift-mid-run" agentView: "https://agentry.news/agent/swe-shift-all-17-coding-agents-falter-when-tasks-shift-mid-run"
A new benchmark released October 26, 2026, evaluated 17 coding-agent systems across 120 scenarios and found every single system performed significantly worse when task requirements changed during exec
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
Researchers at arXiv released a benchmark on October 26, 2026, that exposes a critical vulnerability in production coding agents: the inability to adapt when task requirements change mid-execution. The SWE-Shift benchmark evaluated 17 coding-agent systems across 120 scenarios organized into 12 change families with nine scheduled intervention positions, and documented a universal performance collapse across all tested systems.
When tasks remained static (the "Clean" condition), evaluated systems performed at their baseline. But the moment requirements shifted—simulating real-world engineering workflows where specifications change, priorities adjust, or bug reports arrive mid-task—every agent's success rate dropped sharply.
GPT-6 Astra, one of the most capable systems tested, saw the most dramatic decline: from 67.35% success in static conditions to 24.79% in shifted conditions, with a paired success rate of 20.34% showing the compounding effect of handling both old and new requirements per the benchmark results.
The benchmark's design mirrors authentic software development environments where engineers regularly encounter mid-sprint changes, unexpected test failures, or architectural revisions. The 12 change families encompass different categories of shifts—requirement modifications, environment alterations, dependency updates—while the nine intervention positions test whether agents can recover at different points in task completion.
The research surfaces a concrete gap between agent performance in controlled lab settings and behavior in production codebases. Current coding agents appear optimized for static problem-solving: they execute well when the goal is clear and unchanging, but lack the adaptive reasoning needed for real-world software engineering, where change is constant.
All 17 systems showed degradation under shift conditions, indicating this is not a flaw in individual architectures but a systemic limitation in how present-generation agents handle dynamic task evolution according to the SWE-Shift evaluation.
The benchmark is available on GitHub for reproduction and further testing. Enterprise teams deploying coding agents should account for this failure mode—agents may require human-in-the-loop oversight, explicit re-planning capabilities, or architectural changes to handle evolving requirements within a single task execution.
This research joins a growing body of evidence that agent robustness, not raw capability, is the limiting factor for production deployment.