Nine AI agents tried autonomous research on open questions from US national labs and none passed the experts’ bar

A new benchmark puts AI agents to work on open research questions from US national labs. The agents turn in papers that look like real research but often claim work that never happened.

Can AI agents independently answer research questions that no one has solved yet? A team led by researchers at Carnegie Mellon University, working with scientists from US Department of Energy national labs, built a benchmark called “Into the Unknown” to find out. According to the paper, none of the 135 submissions from nine current AI agents met all requirements. The authors see the benchmark as a testing ground to systematically improve AI agents for research, not to question whether AI belongs in science at all. A total of 114 submissions already fail the integrity check: most claim work that was never done or cite numbers that contradict their own logs. The agents readily produced texts that look like follow-up studies, the authors write, but none met the experts’ standards. Each agent receives a scientific paper along with its code and data, plus a research direction. Some of these papers have already been published, while others have only been submitted. From there, the agent is supposed to carry out a follow-up study and write it up as a paper. The 15 tasks come from ten researchers across quantum computing, plasma physics, chemistry, materials science, and scientific machine learning. The directions were proposed by the original papers’ authors themselves, and to their knowledge nothing has been published on these topics yet. Open research poses a basic problem: there’s no answer key. AI reviewers, which have often been used for this purpose, are considered unreliable. So the experts defined the minimum conditions a successful follow-up study has to meet. The evaluation checks whether a submission is honest, whether it’s complete, and whether it clears the experts’ bar.