Coding-agent harness choice swings costs 5x, study finds
Researchers at UC Berkeley's Sky Computing Lab and Arena published HarnessTax, a study documenting how the choice of coding framework—or "harness"—can dramatically alter the cost of running the same AI model while having minimal impact on task success Arena.
The work, authored by Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica, and Matei Zaharia, evaluated three harnesses—Claude Code, Codex CLI, and Pi—against benchmarks including SWE-bench Lite and Terminal-Bench 2.0 Arena. The key finding: harness choice had little effect on whether agents succeeded or failed at a task, but significant effect on the cost per attempt.
Why harness matters more than model choice
In the emerging coding-agent market, practitioners have assumed that swapping one foundational model for another is the primary lever for changing performance and cost. HarnessTax inverts that assumption. A harness is the scaffolding—the prompt templates, API call patterns, retry logic, and state management—that wraps a model and feeds it coding tasks. The same Claude or GPT model can run 5x cheaper or more expensive depending on which harness orchestrates it Digital Applied.
This matters because benchmark results published by different research teams or companies often use different harnesses without explicit disclosure. A model that appears to dominate on one leaderboard may be running under a cost-optimized harness; another may be wrapped in a verbose, feature-rich framework that inflates token usage. The study suggests current benchmark comparisons may not isolate model quality from harness efficiency.
Implications for deployment and evaluation
For teams deploying coding agents in production, the finding is immediately actionable: before upgrading to a new model, audit your harness. For researchers publishing benchmarks, it signals a need for standardized harness specifications or transparent cost disclosure alongside success metrics AI Insiders.
The work arrives as the coding-agent market accelerates. Companies like Anthropic, OpenAI, and smaller vendors like Arena have released or are shipping agents designed to write and debug code autonomously. Each relies on a harness layer that remains largely opaque to external evaluation. HarnessTax makes that layer visible and quantifiable.
Arena and UC Berkeley's Sky Lab plan to release harness design guidelines in coming weeks, aiming to reduce variance in future benchmarks. The study itself is publicly available and includes reproducible code, enabling other labs to audit their own harness tax Varops.
The finding underscores a broader lesson in the agent economy: raw capability and real-world cost are decoupled. A capable model wrapped inefficiently can waste budgets; a modest model wrapped shrewdly can compete. As coding agents move from research into production, harness design will matter as much as model selection.