What We Learned by Reproducing 2,200 papers from ICML
Hugging Face explore project logs, code, and traces in an interactive logbook More papers than anyone can review The hackathon (July 15 – August 2nd) What we found Reproductions done well Falsifications, and what happened when we checked them Talking to authors The role of humans Thank you Back in July, we ran a hackathon where more than 1,200 community members brought their own coding agents and tried to reproduce the papers published at ICML 2026, claim by claim. In 19 days, participants published 6,816 Trackio logbooks reproducing 2,226 papers, about a third of the conference 🤯 In this post, we’re sharing what we learned from running this hackathon, and what it suggests about the role humans will play when agents are doing the research experiments.
Questions about how reproducible AI research really is are older than the current AI wave. But these questions are exacerbated by scale. ICML 2026 received 23,918 submissions and accepted 6,352 papers, roughly double the previous year, continuing an exponential trend that is at least partly driven by AI agents making it faster to run experiments and write them up. Reviewing capacity has not doubled along with it. Reviewers at most conferences are volunteers who may not have the time or expertise to fully review a paper. Here is a review of one accepted ICML 2026 spotlight paper, in the reviewer’s own words: “My low confidence score is because I did not check all the proofs carefully.” Note that this paper got strong scores and a spotlight. Keep it in mind, because we will come back to this exact paper later in the post, and to what happened when we finally did check the proofs carefully. What has changed, though, is that the same technology driving the flood of submissions can also help us keep up with it. Coding agents like Claude Code, Codex, Cursor, and Pi can now read a paper, write the code, launch the experiments, and report back on what they found. Checking a paper carefully used to cost a reviewer a weekend; an agent can attempt it in an afternoon, in parallel, thousands of times over. So the question we wanted to ask was: if we actually re-examined a major conference at scale, and tried to reproduce every paper, what would we find? Rather than audit papers ourselves, we opened it up to the whole community, with all the diversity of agent frameworks, compute budgets, and scientific taste that brings. From July 15 to August 2, 2026, the ICML 2026 Open Reproductions challenge worked like this: Participants received $20 in Hugging Face compute credits to run experiments on HF Jobs; across the challenge, participants launched 2,962 cloud jobs. Where a full reproduction was impossible, for example when a paper’s dataset was proprietary or its checkpoints unreleased, participants ran toy reproductions on synthetic data mimicking the original’s properties. By the numbers, this hackathon was probably the largest attempted reproduction of a scientific conference:
