Are AI labs pelicanmaxxing?
- ID
- 7033
- Status
- summarized
- Published
- 23 Jul 2026, 7:01 AM
- Fetched
- 23 Jul 2026, 7:33 AM
- Provider
- Simon Willison
- Category
- developer-ai
- Original URL
- https://simonwillison.net/2026/Jul/22/are-ai-labs-pelicanmaxxing/
- Source URL
- https://simonwillison.net/atom/everything/
Summary
- Score
- 6.5
- Created
- 23 Jul 2026, 7:34 AM
- Tags
- Audience
- developersai_ml_learnersai_agent_users
What happened
Dylan Castillo ran a rigorous evaluation of Simon Willison's informal 'pelican riding a bicycle' image benchmark across 7 major AI models, using 48 prompt combinations (8 animals × 6 vehicles) run three times each. The study found no evidence that AI labs are deliberately training models to ace this specific benchmark — pelicans on bicycles aren't drawn any better than other animal-vehicle combinations.
Why it matters
This is a fun but genuinely useful lesson in benchmark methodology and overfitting detection. For anyone building with or evaluating AI models, it demonstrates how to design controlled experiments to test whether models are gaming specific benchmarks, and it's reassuring that at least this informal benchmark hasn't been contaminated by targeted training.
Discussion angle
How do you detect benchmark contamination in your own evals, and should the community be worried about labs quietly optimizing for popular informal benchmarks as they become cultural touchstones?