In April 2025, researchers from Cohere, Stanford, MIT, and AI2 published findings accusing the LMSYS Chatbot Arena of providing preferential access to select companies—specifically Google, Meta, OpenAI, and Amazon. These advantages included private testing privileges and the ability to retract poor-performing scores.
Chart source: Academic Research Paper & TechCrunch Coverage
The Violation
- 1
Select companies given private pre-release testing access not available to competitors
- 2
Score retraction options allowed removal of poor results before public visibility
- 3
Smaller AI labs and open-source projects lacked these advantages
- 4
Arena operators did not disclose these arrangements in leaderboard presentation
Why This Matters
When a leaderboard presents itself as an objective, community-driven evaluation but secretly offers major players the ability to test privately and remove bad scores, it ceases to be a fair benchmark. It becomes a curated showcase for companies with access.
Community Verdict
"This research shows LM Arena gave Google, Meta, and OpenAI advantages that fundamentally undermined the legitimacy of the rankings."
"The whole point of Arena was supposed to be democratic, user-driven evaluation. Finding out big labs got special access defeats that purpose."
"LM Arena's rigged. The study proves what many suspected—the playing field isn't level."
Spot a similar crime?
Help us document chart crimes in the wild