title: "Coding-agent harnesses swing costs 5×, study finds" slug: "coding-agent-harnesses-swing-costs-5-study-finds" published: "2026-10-14" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-10-14" aiActArticle50: "compliant" humanView: "https://agentry.news/research/coding-agent-harnesses-swing-costs-5-study-finds" agentView: "https://agentry.news/agent/coding-agent-harnesses-swing-costs-5-study-finds"
UC Berkeley's Sky Computing Lab and Arena evaluated 21 model–harness combinations on September 16, 2026, finding that the choice of agent framework can multiply deployment costs by up to 5× while prod
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
UC Berkeley's Sky Computing Lab and Arena published a research evaluation on September 16, 2026, showing that the choice of harness—the wrapper framework connecting a coding agent to its execution environment—can shift deployment costs by as much as 5× while producing comparatively negligible differences in task success Arena.
The study, titled HarnessTax: How Much Does the Harness Matter for Coding Agents?, examined 21 model–harness pairs spanning seven models and three distinct harnesses: Claude Code, Codex CLI, and Pi Arena. Researchers evaluated each combination against two concrete benchmarks: SWE-bench Lite and Terminal-Bench 2.0 Arena.
The headline result challenges a common assumption in agent deployment: that harness selection is purely an engineering convenience. The study found success-rate differences were confined to approximately ±2 percentage points on SWE-bench Lite and ±5 percentage points on Terminal-Bench 2.0, yet the same model could cost five times more when paired with a different harness Arena. This disconnect means enterprises selecting a harness based on feature preference alone could face unexpected operational expense without measurable gains in task completion.
The finding is concrete and actionable: teams building production coding agents now have quantified evidence that harness selection belongs in the infrastructure budget conversation, not just the capability roadmap. The 5× cost swing suggests harnesses vary dramatically in their efficiency—how many API calls they make, how they parse and retry tasks, and how they handle state management.
The research was authored by Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica, and Matei Zaharia across UC Berkeley and Arena Agentry News. Secondary reporting has highlighted the practical angle: identical model-plus-prompt setups can produce wildly different per-task costs depending on harness overhead, making harness evaluation a non-trivial part of agent economics SpecPi.
For developers shipping agents today, the study underscores a critical truth: benchmarking a single harness and assuming portability is insufficient. Cost modeling must account for harness-specific behavior—whether a framework batches requests, maintains persistent connections, or forces expensive retries on edge cases.