Tracking AI claims against independent evidence
AI companies make extraordinary claims. Benchmarks are manipulated, demos are cherry-picked and press releases treat internal evaluations as gospel. The AI Hype Index runs the numbers independently: claim by claim, model by model, quarter by quarter. We publish our methodology, our data and the vendors' right of reply in full.
Fourteen vendors, forty-two benchmark claims, eighteen independent reproductions. The correlation between marketing language and actual performance was not what vendors hoped.
Every new model release claims state-of-the-art performance. We tracked what that phrase meant against a fixed evaluation set across 18 months of releases.
The 'reasoning' framing is the current marketing frontier. We ran a structured evaluation against tasks that require genuine multi-step inference. Results varied widely.
A full-year accounting of the claims, the evidence and the outcomes. Twelve months of AI marketing analysed against verifiable results.
Enterprise AI readiness claims come with conditions. We extracted the footnotes, the terms of service carve-outs and the SLA small print from six major platforms.
+9 more articles in this series. Subscribe to the newsletter to receive every new instalment.
Every new article in The AI Hype Index goes straight into the weekly newsletter. No separate subscription required.
No spam. No affiliate links. Just honest analysis. Unsubscribe any time.