Study: Agent Harness Quality Varies Wildly Across Models

Researchers have published a systematic evaluation of how agent harnesses perform across different models, finding that the choice of harness can dramatically reshape which systems succeed or fail at the same task.

Study Design and Scale

The paper, titled "When Agents Fail: Model or Harness—Who's to Blame?" evaluated four general-purpose harnesses—OpenCode, Codex CLI, Pi CLI, and DeepAgents—paired with four core models, plus two fixed native model-harness systems Scholar Inbox. The team ran 52,168 trajectories across 111 evaluation settings spanning six benchmarks that tested context management, tool use, long-context planning, multimodal interaction, and artifact handoff AIcrier.

Key Finding: Rankings Are Not Universal

The critical discovery is that harness rankings vary substantially across models and benchmarks the-decoder.com. A harness that ranks first on one benchmark with one model may rank last on another task or with a different foundation model—a pattern that challenges the assumption that a single "best" harness exists.

OpenCode's Cross-Model Edge

OpenCode achieved the highest cross-model performance, with a reported Cross-Model Fit score of 87.48/100 and a pointwise 95% confidence interval of 84.06–89.99 AIcrier Digest. However, this leadership was measured across models and benchmarks as a portfolio metric, not as dominance on every individual task.

Implications for Agent Developers

The findings matter because harnesses—the execution layers that route model outputs to tools, manage state, and handle errors—are not neutral infrastructure. They actively shape what agents can accomplish. Teams building production systems cannot assume that picking the highest-ranked harness from one benchmark will transfer to their own use case.

The scale of the evaluation—111 configurations and over 52,000 trajectories—provides empirical grounding that moves beyond anecdotal product comparisons. Each harness-model pair was tested consistently, allowing researchers to isolate how harness design choices interact with model capabilities.

What's Next

The study does not prescribe a single optimal harness. Instead, it documents the variance itself as a key finding: developers must test harness-model combinations on their specific benchmarks before committing to production deployments. The research provides a framework for that testing and baseline data that teams can use to inform their own architecture decisions.