Study Reveals Why Larger Language Models Excel Over Smaller Counterparts

Small language models fail at rare tasks because frequent ones constantly overwrite what they’ve learned. A new study with models ranging from 4 million to 4 billion parameters shows this mechanism in detail and offers a practical fix: instead of scaling up models, it may be enough to increase how often the target task appears in the training data.

In some cases, small models can’t reliably learn rare tasks even with extremely long training runs. Even well-known scaling laws show that a small model never reaches the loss of a large one, no matter how much data you throw at it. To isolate the mechanism, the researchers tested a mix of tasks with varying frequency and complexity. A model with N neurons gets assigned the N “most useful” features, where usefulness is based on how often a task appears and how important it is. Frequent, simple tasks get priority. Rare, complex ones get dropped. In the experiments, only models that were large enough learned tasks that made up just 0.25 percent of the training data. The core of the paper is its explanation of why size helps. As long as frequent tasks aren’t well-learned yet, they pull the model strongly in their direction at every training step, overwriting much of what the model picked up about rare tasks. Once a large model has mostly mastered the frequent tasks, that pull fades. The freed-up capacity goes to rare tasks, and learned signals are more likely to stick. Small models rarely reach that point, according to the study. They fall into an “update-and-forget” loop. A rare example gets briefly learned, then largely erased by the next training steps on frequent tasks. When the next rare example shows up, the model starts over from scratch.