Stripe launches evals to benchmark AI coding agents
Stripe published a public benchmark on November 19, 2026 measuring how well coding agents build Stripe payment integrations, with a methodology post updated the following day (Stripe, Stripe). The benchmark, called Stripe Evals, describes itself as "a benchmark measuring how coding agents build high-quality Stripe integrations" (Stripe).
What Stripe tested
Stripe ran 69 scenarios across one-time payments, subscriptions and Connect — covering hosted Checkout, metered billing, per-seat subscriptions and marketplace payment splitting — against 14 model-harness pairs, each tested with and without Stripe's own agent plugin (Stripe). In each scenario a coding agent built the integration while a separate role-playing agent posed as a developer asking follow-up questions; Stripe then graded each criterion pass or fail using the resulting conversation and code. This matters to anyone building or buying coding agents because payment integration errors — a missed webhook, an unhandled subscription edge case — can directly cost money or break customer billing, and Stripe's benchmark is one of the first public, vendor-run tests of that specific failure mode.
Results and caveats
Stripe's own results state that "all 14 coding agents scored higher with it, by roughly 3 to 8 percentage points, and web searches fell 96% as models relied on trusted guidance," referring to the Stripe agent plugin (Stripe). Secondary coverage adds further figures not confirmed in Stripe's own published materials: a report citing Stripe employees Kiran Prasad, Yiing Chau Mak and Manish Kurse calculates 276 total runs (69 scenarios × 14 pairs × plugin on/off) and states agents averaged 78% of criteria while fully satisfying all criteria in only 29% of runs (reveneau.com). A social post from Stripe circulating the same figures echoes this (X). Stripe's primary benchmark page and methodology post do not themselves state the 78% average, the 29% full-pass rate, the 276-run total, or any author list.
Why it matters
For teams evaluating coding agents on real integration work rather than synthetic coding puzzles, Stripe Evals offers a vendor-specific but concrete yardstick: a measurable gap between agents with and without domain-specific tooling, and a documented drop in unnecessary web searches when agents are given trusted context. Developers can inspect the methodology directly, including scenario design and grading criteria, through Stripe's published materials (Stripe).


