Stripe evals: Measuring real-world coding agent performance
Coding agents’ average success rate plunges from 78.1% to 29.4% on Stripe’s benchmark when every required integration criterion must pass.
Coding agents’ average success rate plunges from 78.1% to 29.4% on Stripe’s benchmark when every required integration criterion must pass.
One thoughtful comment unlocks the ability to create threads.
disclosure: i didn't read the article, just skimmed thru, it looks like they argue that telling agent how to achieve task ahead (MCP and skills) is better then let agent to figure it on it's own, but that is kinda well known, or i completely missed the point?
I read it less as just the claim that it's better to let agents figure things out on their own, and more as the claim that they're in a position to offer a certain kind of benchmark of how they do in a very MCP- / skill-rich environment. I agree with you that if the interest of the article were just that claim, it wouldn't be interesting, but I think they're offering a bit more.
Misc. notes:
Stripe is great at this sort of writeup.
One of the interesting findings here (kingfisher.tldr.fm) is that "Coding agents’ Stripe integration scores plunge from 78.1% to 29.4% when passing requires meeting every essential requirement." This is a well-known but IMO still underrated aspect of working with agents (and humans!): if you're willing to find and fix the 5% they get wrong, you often do way better just reaping the 95% they get right over and over. Obviously that's easier said than done.
Opus 5.5 unsurprisingly looks good here, and I feel nudged to look harder at Sol and Astra.