Sequential Coding Tasks Double Agent Cost Without Gains
Researchers from JetBrains analyzed 4,782 professional developer sessions to expose a mismatch between how coding-agent benchmarks are structured and how developers actually work. The resulting study, published on arXiv, introduces SWE-TaskFlow, a method for adapting issue-based benchmarks to measured interaction patterns and reveals that sequential problem-solving approximately doubles agent inference costs without delivering consistent performance gains arXiv.
The Cost-Performance Tradeoff
When researchers compared single-message tasks against split, multi-step workflows using 700 SWE-Bench Pro tasks, the economics became stark. Sequential interaction roughly doubled agent inference cost across multiple model variants. For GPT-5.6 Luna, costs increased from $0.014 to $0.034; for GPT-5.6 Terra, from $0.152 to $0.400; and for Gemini 3.7 Flash, from $0.675 to $1.370 arXiv.
The paradox: that cost spike did not translate into better results. GPT-5.6 Luna resolution rates fell from 50.0% to 45.8%, while GPT-5.4 mini dropped from 46.5% to 42.5%. Gemini 3.7 Flash declined from 60.0% to 55.6%. Only GPT-5.6 Terra showed marginal improvement, rising from 54.2% to 54.8%. The pattern suggests that fragmenting tasks into developer-like workflows may actually confuse agents trained on monolithic problem statements arXiv.
Measuring Real Developer Behavior
The study's strength lies in its empirical grounding. Rather than assume how developers interact with code, the team captured actual sessions from professionals using JetBrains IDEs—the dominant toolchain in enterprise development. This ground truth revealed that real workflow patterns are far messier than benchmark assumptions, with developers asking clarifying questions, iterating on partial solutions, and consulting repository context before committing to fixes.
SWE-TaskFlow translates those patterns into benchmark design. The methodology splits traditional issue-based tasks into multi-step sequences and tests whether agents handle repository-context queries better. The answer, so far, is mixed: agents incur higher costs chasing context without necessarily solving problems faster GitHub.
Implications for Benchmark Design
The findings matter because coding-agent benchmarks drive product development, vendor claims, and enterprise procurement decisions. If benchmarks reward monolithic problem-solving but users operate sequentially, vendors will optimize for the wrong thing. The research suggests benchmarks should either mirror observed developer workflows or explicitly document the gap between benchmark and reality.
The dataset, released on Hugging Face, contains the full 4,782 session trajectories paired with SWE-Bench Pro task data, enabling other researchers to validate and extend the work Hugging Face.
This is the kind of unglamorous, infrastructure-level research that shapes what agents can actually accomplish in production. Doubling cost without gains is not a feature—it is a signal that the measurement itself may be broken.