---
title: "LiveMACE: Five LLMs Tested in Live Markets Reveal Capability Gaps"
slug: "livemace-five-llms-tested-in-live-markets-reveal-capability-gaps"
published: "2026-10-21"
beat: "Research"
tags: ["Research"]
creator: "Agentry Newsroom"
editor: "Susanne Sperling, Editor — Human in the Loop"
tools: ["Claude (Anthropic)", "Perplexity Sonar"]
creativeWorkStatus: "verified"
dateReviewed: "2026-10-21"
aiActArticle50: "compliant"
humanView: "https://agentry.news/research/livemace-five-llms-tested-in-live-markets-reveal-capability-gaps"
agentView: "https://agentry.news/agent/livemace-five-llms-tested-in-live-markets-reveal-capability-gaps"
---# LiveMACE: Five LLMs Tested in Live Markets Reveal Capability Gaps

> Researchers evaluated five frontier large language models over 30 days in live financial markets and found that realized trading outcomes often diverged sharply from measured capabilities in tool use,

*Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. [AI policy](/ai-policy).*

Researchers have published a process-aware evaluation framework revealing significant gaps between measured LLM capabilities and real-world trading performance in live financial markets. The study, titled *LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets*, was submitted to arXiv on October 7, 2026, with an update posted October 8, 2026 [arXiv](https://arxiv.org/abs/2610.09872).

## Study Design and Scope

The research team—led by Jun Zhao, Leiming Fu, and collaborators at multiple institutions—introduced **LiveMACEBench**, a framework designed to evaluate persistent LLM agents operating in active financial markets. Over a **30-day evaluation period**, researchers tested **five frontier LLMs** under matched configurations that stressed four key agent capabilities: tool use, persistent memory, rule following, and multi-agent collaboration [arXiv](https://arxiv.org/abs/2610.09872).

Unlike controlled lab benchmarks, LiveMACE operates on live market data where conditions shift continuously—a critical test of whether measured capabilities translate to actionable agent performance. The framework captures not just *what* agents achieve but *how* they achieve it, measuring the specific mechanisms each model employed to reach its outcomes.

## Key Findings: Capability-Performance Mismatch

The research uncovered a critical insight: **realized returns often diverged materially from capability-specific measurements**. This means two models could post identical trading results despite demonstrating vastly different patterns of tool use, memory management, rule adherence, and collaboration strategies. Conversely, similar measured capabilities did not guarantee similar real-world outcomes.

This disconnect challenges a core assumption in agent evaluation—that higher scores on isolated capability tests predict better performance in dynamic, competitive environments. The findings suggest that how agents *combine* capabilities under market pressure matters more than isolated capability strength.

## Implications for Agent Developers and Enterprises

For teams building agentic systems intended for financial services, trading, or other high-stakes environments, LiveMACE highlights the risk of relying solely on laboratory benchmarks. Capability measures like "tool-use accuracy" or "memory retention" provide incomplete signals of production readiness. Real-world deployment demands stress-testing in live or high-fidelity simulated conditions where agent decisions propagate consequences.

The research also points to a methodological gap: existing benchmarks may not capture emergent behaviors that arise only when agents operate continuously with persistent state and real market feedback. The framework offers researchers a template for process-aware evaluation—examining not just outcomes but the decision sequences and mechanism selections that produce them.

## Next Steps

The paper's release into the research community signals growing attention to the gap between agent capability measurement and real-world performance. As enterprises accelerate agentic deployments, demand for evaluation frameworks that stress-test agents under live conditions—rather than static labs—is likely to increase.