What We Learned by Reproducing 2,200 papers from ICML
- ID
- 13906
- Status
- summarized
- Published
- 13 Aug 2026, 8:00 AM
- Fetched
- 14 Aug 2026, 1:43 AM
- Provider
- Hugging Face Blog
- Category
- developer-ai
- Original URL
- https://huggingface.co/blog/icml-2026-open-reproductions
- Source URL
- https://huggingface.co/blog/feed.xml
Summary
- Score
- 7.5
- Created
- 14 Aug 2026, 1:44 AM
- Tags
- Audience
- developersai_ml_learnersai_agent_users
What happened
Hugging Face ran a 19-day hackathon (July 15–Aug 2, 2026) where 1,200+ participants used coding agents (Claude Code, Codex, Cursor, OpenResearch's orx) to reproduce ICML 2026 papers claim by claim, producing 6,816 Trackio logbooks covering 2,226 of the conference's 6,352 accepted papers (~a third). ICML 2026 saw 23,918 submissions and 6,352 acceptances, roughly double the prior year, partly attributed to AI agents accelerating experiment cycles and writing.
Why it matters
If you build or rely on ML research outputs, this signals that a meaningful fraction of top-conference papers may not have been rigorously verified by reviewers—one spotlight paper's proofs went unchecked despite strong scores. Coding agents can now attempt full reproductions in an afternoon that would cost a human reviewer a weekend, so teams shipping ML should consider running agent-based reproduction checks on papers they depend on rather than trusting acceptance as a quality signal.
Discussion angle
What does it mean for the ML ecosystem when conference acceptance no longer implies reproducibility, and coding agents can both flood submissions and audit them at the same scale—should builders treat papers as unverified until agent-reproduced?