Emergent Abilities in Language Models Through the Lens of Loss

In the paper "Understanding Emergent Abilities of Language Models from the Loss Perspective," Zhengxiao Du et al. from Zhipu AI and Tsinghua University challenge the belief that emergent abilities in language models are exclusive to large models. The study proposes analyzing these abilities through pre-training loss rather than model size or compute. Findings indicate that models with similar pre-training losses, regardless of size, perform equally well on downstream tasks, suggesting that emergent abilities are linked to lower pre-training losses, rather than model scale.