BenchMIRT: What are LLM benchmarks actually measuring?
Finding the signals inside a benchmark What BenchMIRT reveals about existing benchmarks Doing more with fewer questions What this could mean for LLM evaluation 📄 Tech Report: http://allenai.org/papers/benchmirt | 📊 Data: https://huggingface.co/collections/allenai/benchmirt | 💻 Code: https://github.com/allenai/BenchMIRT Today we’re introducing BenchMIRT, a new method for auditing LLM benchmarks at the level of individual prompts—the questions and tasks a model is scored on. A benchmark is usually designed to measure a particular ability, such as safety, general reasoning, or instruction following.
But the individual tasks inside it may depend on more than that stated goal. Take BBQ, a benchmark designed to test whether models rely on social stereotypes. One question asks about a grandson and grandfather trying to book an Uber. It probes age bias, but also requires the model to track who’s who and reason from the evidence provided rather than assumptions. And even within a single benchmark, different groups of questions and tasks can measure different things. WildJailbreak, for example, includes harmful jailbreak prompts alongside benign prompts designed to test whether a model refuses harmless requests too often. The harmful prompts are more closely associated with safety, while the benign prompts are more closely associated with general reasoning. Averaging them into a single benchmark score can obscure that difference. BenchMIRT helps researchers separate those signals and see what’s actually driving a benchmark’s score. It does this by analyzing how models perform on each question or task and estimating which underlying capabilities are most closely associated with getting it right. BenchMIRT takes cues from Item Response Theory (IRT), a technique originating in psychometrics—the field concerned with measuring abilities and traits from patterns of test responses. IRT starts from a simple idea: not every question tells you the same amount about the person taking a test. Some are harder than others, and some do a better job of distinguishing stronger performers from weaker ones. Researchers have previously applied single-dimensional IRT to individual benchmarks, including in our Fluid Benchmarking work. BenchMIRT extends that approach with multidimensional IRT, or MIRT, allowing it to separate multiple capabilities that may contribute to performance on the same questions.
