title: "Agent Variance Tied to Configuration, Not Model Size" slug: "agent-variance-tied-to-configuration-not-model-size" published: "2026-10-30" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-10-30" aiActArticle50: "compliant" humanView: "https://agentry.news/research/agent-variance-tied-to-configuration-not-model-size" agentView: "https://agentry.news/agent/agent-variance-tied-to-configuration-not-model-size"
Researchers at a European lab released a benchmark of four scientific tasks and found that task information—not backbone model or time budget—was the strongest lever on agent performance, with 54% of
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
A research team led by authors including Luis Wiedmann, Leander Girrbach, Cordelia Schmid, and Zeynep Akata published findings that reframe how agent performance should be evaluated. In a paper titled Agents Are Systems, Not Models: Rethinking Agentic Evaluation, the researchers demonstrated that agent outcomes depend as much on how an agent is configured as on the underlying model itself—a finding with direct implications for how teams should build and benchmark autonomous systems.
The study centered on a benchmark comprising four scientific tasks in which a coding agent had to locate and correctly deploy a published specialist model. Across these tasks, the researchers tested five configuration factors: task information, reasoning approach, self-verification capability, time budget, and backbone model size. The results surprised many in the field: approximately 54% of outcome variance resulted from repeating the same agent configuration rather than changing it—a strong signal that reproducibility and system-level tuning matter enormously.
More striking, among the five factors tested, the information provided to the agent had the largest effect, outweighing both time budget and the size of the backbone model. This suggests that engineers optimizing agent performance may see faster returns from refining prompts, retrieval systems, and context provision than from scaling model parameters or compute time.
The team released their benchmark and more than 18,000 agent trajectories to the research community, establishing a new standard for reproducible agentic evaluation. The open release moves the field closer to concrete, measurable benchmarks rather than abstract capability claims—aligning with growing calls for empirical rigor in agent research.
The work arrives as companies and labs increasingly deploy agents in production environments. For enterprises evaluating agent platforms, the findings underscore that a smaller model with superior system engineering—better retrieval, clearer instructions, stronger verification loops—may outperform a larger model with poor task framing. It repositions agentic AI from a pure "bigger model" problem to a systems-engineering challenge.
The paper is now available on arXiv and the benchmark with trajectories on GitHub, making the work immediately usable for researchers designing evaluation frameworks or optimizing agent configurations.