The Gamma Model Index: OpenAI takes the lead for the first time
OpenAI’s GPT-6.1 Terra cut Gamma’s slide-repair failure rate from 9.1% to 4.7%, pushing OpenAI past Anthropic in usage for the first time since 2023.
5 comments
Misc. reactions:
This is cool and I'd love to see more analyses like this that combine qualitative and quantitative description and describe performance in terms of roles like "editor."
I strongly agree with the note that open models' performance on benchmarks is far ahead of their actual utility. I've found this to be true just about every time I've tested one.
What I like about this piece (the mix of qualitative and quantitative description of real-world usage) means, I think, that it really ought to have been written entirely by a human. Pangram and I agree that it was not, and I think it's important to interpret the various conclusions as AI-guided.
There's always a difference between research benchmarks and industrial use case results, so it's interesting to see this comparison and analysis done. Sometimes (if not most of the time), industrial use cases are generally looking for "good enough" "gets the job done" at a low cost and latency rather than the best possible results
Really interesting counter-narrative chart showing Gamma's OpenAI spend passing Anthropic.
IMO Anthropic's models are primarily better in agentic use cases, we mostly use Gemini / OpenAI for non-agentic (vision / writing / classification, etc.) use cases.
Our internal slide deck visual QA runs on GPT-6.1-Sol, so it doesn't surprise me that Gamma's also using an OpenAI for that.