Trial by Trace benchmarks AI agent harness reliability

Benchmark Tests Four Agent Harnesses Across 3,420 Runs

TraceLite-AI researchers released TraceTrial-Bench, a benchmark designed to measure the reliability of AI agent harnesses independent of task success arXiv. The evaluation framework tested four harnesses—Claude Code, Codex, OpenClaw, and DeepSeek Harness—across nine models and 95 mechanism-oriented tasks, selecting 3,420 executions for analysis GitHub.

The harness scores, reported on a macro-averaged basis, ranged from 59.8% to 72.0% arXiv. This spread indicates significant variance in how reliably different harnesses execute tasks, even when underlying models perform similarly. The benchmark assessed seven distinct reliability components, creating a granular picture of where agent systems succeed and fail at the infrastructure level.

Task Success Masks Reliability Failures

A striking finding emerged from the analysis: 31.1% of runs achieved task success while failing at least one harness criterion GitHub. This decoupling between outcome and process reliability suggests that agents may reach correct answers through unreliable execution paths—a critical distinction for deployed systems where consistency and traceability matter as much as raw task completion.

The benchmark's methodology retained one selected execution per task–model–harness combination, documenting how each agent arrived at its result arXiv. However, the single-run-per-combination design means the results measure point reliability rather than repeated-run stability, limiting inferences about production durability.

Implications for Agent Deployment

The findings underscore an emerging challenge in the agent economy: harnesses must be evaluated on their own terms, not merely as pass-fail components attached to model outputs. A harness that achieves 59.8% macro reliability may still ship to production if task success is high, but operators lack visibility into whether failures stem from the model, the harness, or the interaction between them.

The 95-task scope covers mechanism-oriented scenarios—designed to isolate specific behaviors and failure modes—rather than end-to-end enterprise workflows. This design allows researchers to pinpoint which reliability components (logging, error handling, resource management, etc.) degrade performance most severely across different harnesses.

TraceLite-AI has published the full benchmark on Hugging Face and GitHub, making results and evaluation code available to researchers and practitioners.