Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention

,a Daniel Wurgaft

,a Rachit Bansal b Laura Ruis c

Jing Huang

Naomi Saphra b David Alvarez-Melis b Andrew Lampinen d

Christopher Potts a Ekdeep Singh Lubana

a Stanford University b Kempner Institute at Harvard University c MIT d Anthropic

#Abstract

Larger models learn tasks smaller models do not. What drives this phenomenon? We develop a simple phenomenological argument that power-law scaling already suggests that a larger model will be able to learn a part of the data distribution that a smaller model fails to learn, even with infinite training data. To validate this claim and identify its causes, we study the effects of model scaling on a synthetic setup consisting of a mixture of tasks that show monotonic scaling curves. The results point to a data-induced competition over resources (neurons). Specifically, smaller models allocate their neurons to high frequency or low complexity tasks, and so they learn solutions that perform poorly on rare and complex tasks. Moreover, this happens even when solutions capable of expressing the desired task exist. We then assess how a larger model circumvents this data-centric bottleneck, finding that it traces to a reduced interference mechanism: larger models can allocate enough resources to common tasks that the gradient updates for those tasks become weak, which means that they do not overwrite rare-task features as they slowly accumulate. Finally, to further validate these claims, we pretrain OLMo models (4M to 4B parameters) on novel tasks of varying frequency and complexity. The results mirror those from our synthetic data experiments: only the larger OLMo models learn the infrequent and complex tasks, and these larger models embed more task features in their representations and show less gradient interference between tasks. Overall, we offer a data-centric account of why larger models learn tasks that smaller models fail to. This helps explain why larger models are better in practice, and it can inform practical questions concerning model sizing and training data mixtures.

#1 Introduction

Modern machine learning is celebrated for its massive generalist models, which are capable of handling arbitrary inputs in diverse and complex environments [ 1 – 10 ]. Based on the empirical finding that larger models often excel where smaller 1 models show random-chance performance, prior work has claimed that the ability to solve certain critical tasks only emerges in larger models [ 11 – 19 ]. Such arguments have fueled the drive towards increased scaling. However, given the large training and inference costs that large models impose, it is worth identifying precisely what marginal benefits are unlocked by larger models and whether scaling parameters is the sole way of realizing those benefits.

Our argument begins from the observation that power-law scaling [ 20 – 22 ] already suggests that there is a regime in which a smaller model fails to learn parts of a data mixture that a larger model succeeds on, even under asymptotic training (Fig. 1, Sec. 2). This suggests that larger

1 We use

the terms “larger” and “smaller” informally here but develop a precise relational definition of these terms in Sec. 2.

First, scaling enables learning rare and complex tasks (Sec. 3.1). Our experimental setting defines controlled manipulations of task frequency and complexity. We present an analytic argument that only larger models will (on average) learn the rare and complex tasks present in this setting, and we verify this analysis experimentally (Fig. 2).

Second, reduced competition for resources enables learning rare and complex tasks (Sec. 3.2). Here, we extend our formal analysis to show that, upon observation of samples from a rare task, model parameters update, but only larger models, by virtue of having more parameters and hence less gradient interference, are able to retain memory of a previously observed batch of data from a rare task. Thus, when the next batch of rare-task data comes in, the larger model builds on its prior knowledge, which ultimately leads to success despite the impoverished learning signal. In contrast, the smaller model is forced to start from scratch and consequently fails. We again verify these findings experimentally in our regression setting (Figs. 3 and 4).

Finally, we validate the above theoretical arguments in real LLMs (Sec. 4). Specifically, we pretrain OLMo models (4M to 4B parameters) on the Dolma v1.7 corpus with completely novel tasks injected at controlled frequency. We find that only the larger OLMo models are able to learn the infrequent and complex tasks (Sec. 4.2). Furthermore, these OLMo models mirror our toy-task models in deeper ways: larger OLMo models have more task features embedded in their representations (Sec. 4.3) and show less gradient interference (Sec. 4.4). Beyond supporting our theoretical claims, these results can provide practical guidance to large-scale model training efforts.

Overall, the data-centric nature of our analysis suggests that understanding why larger models learn more requires not only asking what they can represent, but also what is learnable under gradient-based optimization from a given data mixture.

#2 A Phenomenological Model Predicts Larger Models Learn More

Neural network scaling is known to predictably and monotonically improve loss [20, 28, 51]:

L (N, D) = L 0 + A

N α + B

$D β , \quad (1)$

where L 0 denotes the irreducible loss, A, B are constants, and α, β are parameter / data exponents (α ≈ 0. 46 and β ≈ 0. 51 for Chinchilla-scaling [28]). Training in a compute-optimal manner, i.e.,

finding the model size and data configuration that helps achieve the minimum loss at a given compute budget C, gives us L C (N) ∝ N − γ, where γ = 0. 34, and L C (N) denotes the optimum loss achieved when training a model with N parameters under resource constraints. The relation shows larger models are expected to achieve a smaller loss. However, resource-constrained training by itself does not inform what a model can actually express. Specifically, even though a smaller model may have a worse compute-optimal loss, we do not know if it is fundamentally incapable of achieving the same loss as the larger model. To assess that statement, we must evaluate a model’s loss under asymptotic resources (i.e., infinite data): 2

L ∞ (N) ∝ N − α. If α > γ, as is the case in practice, we again see gains from merely scaling the model size. That is, the asymptotic loss achieved by a larger model is better than the smaller one. This indicates there is a part of the training distribution a smaller model, despite observing infinite data, fails to learn. Based on this phenomenological argument, we define the following. Definition 1 (Learnable via data scaling). Consider a target model with N l number of parameters that we call “large”. We say a “smaller” model, i.e., for which parameter count N s < N l, can recover the loss of a larger model via data scaling if L C (N s) − L C (N l) > 0, but L ∞ (N s) − L C (N l) < 0.

Def. 1 thus captures the scenario put forward in Sec. 1. That is, the smaller model may in fact be just undertrained: the larger model learns more sample efficiently and reduces loss faster, but a smaller model can eventually catch up [ 18, 23 – 29 ]. Correspondingly, the marginal ability of a larger model to explain the data distribution (i.e., the loss) can be recovered by a smaller model merely observing more data. Nevertheless, there exist regimes where data scaling will not suffice, as described next. Definition 2 (Learnable via model scaling). Consider a target model with N l number of parameters that we call “large”. For a small scalar value ϵ, we define N ∗ s (ϵ) as the largest “small” model if L ∞ (N ∗ s (ϵ)) − L C (N l) > ϵ. That is, even asymptotically, the smallest model never reaches the same loss as the large model. Correspondingly, for a given model size N, we call it “small” if N < N ∗ s (ϵ) and say recovering the loss of the larger model requires model scaling.

This latter scenario thus captures the case where, when two models with parameter counts N s, N l, with N s < N l, are trained, there is truly a marginal improvement for explaining the data that can be attributed to the larger model having more parameters. This is the most interesting case that warrants further study: what is it about the data that only a larger model can learn, such that the smaller model cannot, even after observing infinite data? How precisely does having more parameters aid this learning? We aim to answer these questions in the following sections.

#3 Scaling Allows Learning Rare Tasks by Reducing Gradient Interference

Our phenomenological argument in Sec. 2 motivates the claim that larger models are likely to learn a part of the data distribution smaller models will fail to learn. We next aim to get more concrete about this claim. Specifically, we exploit the fact that our argument is merely based on monotonic (power-law) scaling—a phenomenon even synthetic tasks can recapitulate [ 30 – 35, 37 – 40 ]. Such tasks have in fact been used in prior work to make accurate predictions about scaling behavior for large-scale models [51, 52]. We thus follow this line of work and develop a multi-task learning setup that helps assess which tasks a larger model can learn but a smaller model cannot. We generalize our claims to an off-the-shelf language model pretraining pipeline [53] in Sec. 4, finding the core hypotheses derived out of this toy setting hold true on even a large-scale training pipeline.

Data. We consider a multi-task learning setup where samples are drawn from a mixture of K linear regression tasks. Specifically, the k th task is assumed to appear with frequency π k > 0, such that P

j ≥ 1 λ k,j b k,j b ⊤ k,j. Here, the “feature matrix” B k = [ b k, 1, b k, 2,... ] is assumed to have orthonormal columns; Λ k = diag(λ k, 1, λ k, 2,...) with λ k, 1 ≥ λ k, 2 ≥· · · ≥ 0; and different tasks occupy orthogonal blocks, i.e., B ⊤ k B ℓ = 0 for k ̸ = ℓ. If the spectrum { λ k,j } decays slowly, the task requires more directions for producing the corresponding target—we can thus compare the relative complexity of two tasks by comparing the rate at which their spectra decay. Compared to prior work studying theory of scaling laws based on toy regression tasks, we emphasize that our setup involves the learning of multiple tasks simultaneously.

k π k = 1, and has covariance C k = B k Λ k B ⊤ k = P

2 We note

power-law scaling need not hold asymptotically [31, 52], which is why we call this argument phenomenological. It motivates the subsequent, rigorous claims.

Teacher / Student Models. For a given input x ∼N (0, I), the teacher for task k is defined as y k = Λ 1 / 2 k B ⊤ k x. The student uses a shared width- N encoder U ∈ R d × N, U ⊤ U = I, with projector P U = UU ⊤, together with task-specific linear decoders D k to discern between tasks. Correspondingly, the student prediction isŷ k = D k U ⊤ x. The total mixture loss is the weighted sum L N (U) = P K k =1 π k ℓ k (U), where ℓ k (U) = E  ∥ y k − D k U ⊤ x ∥ 2 2  is loss of the k th task. Note that

herein, since the optimal decoder admits a closed-form solution D ∗ k = Λ 1 / 2 k B ⊤ k U, we solely analyze the dynamics of the encoder, which produces features used by the student for making predictions.

#3.1 Larger Models Learn Rarer, More Complex Tasks

In order to narrow down a mechanism that explains how larger models may be able to learn more, we must first identify precisely what it is that a larger model learns but a smaller one fails to. We begin with answering this question in our toy setup. Theorem 3 (Features are Learned in Order of Utility). For a given U, the mixture loss reduces to L N (U) = Tr(M) − Tr(U ⊤ MU), where M:= P K k =1 π k C k. Hence, a width- N minimizer spans the top- N eigenspace of M, whose eigenvalues are defined by the weighted per-task spectra:

u k,j:= π k λ k,j. (2)

Thus, the optimal encoder keeps the N features (k, j) with largest u k,j —we call these terms utilities. This implies if n k (N) denotes the number of retained features from task k, then ℓ ∗ k (N) = P

j>n k (N) λ k,j. Conversely, the minimum width at which a model learns at least m features for all tasks is N ∗ (m) = min  N: n k (N) ≥ m.

In the context of our toy task, the statement above helps answer the question “what does width buy?” by defining a concrete ranking rule for feature learning. 3 Specifically, it says a larger model, asymptotically, learns exactly those features whose utilities are lower than those of the features learned by a smaller model. This implies if a task is observed infrequently or it involves several features, e.g., if its spectrum decays very slowly, then (on average) only a larger model will learn it.

Verification. We verify the claim above by training our student model on a mixture of K = 32 tasks, using the Adam optimizer for 100 K steps (the loss does not improve beyond this budget even when trained up to 10 × longer; see Fig. 21). We use a power-law prior k − β to define task frequencies, and a power-law per-task spectrum λ k,j ∝ j − α. For simplicity of visualization, we let α = 2 be shared across tasks and only vary task frequencies by changing β (see App. D for experiments modulating complexity by varying α). Results are reported in Fig. 2 (also see App. E for further results). We find (a) the per-task loss and (b) the overall residual loss predictably reduce with model width. Critically, we see larger models learn infrequent tasks better than smaller ones.

#3.2 Scaling Reduces Interference and Allows for Retention of Rare Task Observations

While the argument above—i.e., a larger model learns low utility, infrequent features—is intuitively reasonable, it is critical to note that if the frequency at which a task or its features are seen is very

3 This claim

can also be seen as a static ordering rule that local optima visited by a model during training will be expected to dynamically follow in its saddle-to-saddle dynamics [ 54 – 57 ]

low, then, regardless of size, there is a statistical bottleneck here that a model needs to circumvent. For example, in the experiments shown in Fig. 2 b, a model must learn a task that constitutes merely 0. 25 % of observations. We next analyze how width helps surmount this challenge. To this end, note that for the k th task, the Riemannian gradient is G k (U) = 2(I − P U) C k U, and hence the mixture gradient is ˙ U = 2(I − P U) MU. We then have the following claim. Theorem 4 (Residual Controls Learning). Let F ⊆ [ K ] denote the common or frequent tasks. Define these tasks’ weighted covariance M F:= P

k ∈ F π k C k and residual signal δ F (U):= Tr (I − P U) M F
. Then, the aggregate common-task gradient G F (U) = 2(I − P U) MFU obeys the bound

∥ G F (U) ∥ F ≤ 2 p

λ 1 (M F) δ F (U). (3)

The statement above says a set of tasks move the model only through the part of their covariance that is not already explained by the current representation, i.e., the residual δ F (U). Correspondingly, once the high-utility common-task features have been learned, their updates become weak (i.e., low norm). This leaves any spare width available to rare tasks. More precisely, let µ F 1 ≥ µ F 2 ≥· · · be the eigenvalues of M F. The best width- N representation for the common tasks alone leaves residual δ ∗ F (N) = P

i>N µ F i. Then, via Theorem 4, we get the following.

i>N µ F i ≤ ε. For every N ≥ N F (ε), there exists an encoder for which δ ∗ F (N) ≤ ϵ and ∥ G F (U) ∥ F ≤ 2 p

Corollary 5 (Width-Scaling Reduces Competition). Define N F (ε):= min  N: P

µ F 1 ϵ.

That is, once N ≳ N F (ε), the model contains enough resources that can be allocated to the common tasks, rendering the gradient towards them weak. This makes the remaining resources available to rare tasks. However, even once interference is weak enough for a rare task to be learned, it is unclear whether gradient descent can actually consolidate that signal across its infrequent observations. To this end, we next characterize the local condition under which a specific rare feature can pull the model towards itself, without forcing the forgetting of well-learned tasks. Specifically, assume we wanted to learn a rare rank-one task C r = λ r b r b ⊤ r orthogonal to the common block. Let U (N) F be top- N eigenspace of M F with eigenvalues µ F 1 ≥ µ F 2 ≥· · ·. Then, we have the following claim.

Validation. We train models of varying width on the same setup as Fig. 2 and plot how much signal from directions describing a task is present in the model’s intermediate representation. Specifically, since ℓ k (U) = Tr((I − P U) C k), the signal captured for task k is Tr(PUC k), where Tr(.) denotes

trace of a matrix. We thus measure s k (U) = Tr(PUC k)

Tr(C k). To contextualize this value, we normalize

with respect to a random baseline, yieldings̃ k (U) = s k (U) − N/d

1 − N/d;

N/d denotes the expected value of s k (U) if U were a randomly drawn matrix from the Stiefel manifold. We call this quantity “Normalized Signal”. Results are shown in Fig. 3. We see when the model width is small, frequent tasks have a high residual signal remaining to be explained; here the set of frequent tasks is defined as top- K tasks whose prior sums to 0. 8, resulting in K = 3. Correspondingly, rare tasks’ signal in model representation is no better than random. Meanwhile, as we scale, once the width crosses our predicted threshold δ ∗ F (N crit r), we find the bulk of the frequent tasks’ signal is explained away and rare tasks start to get learned.

To isolate how the gap between observations interacts with width, we also design a matched-frequency injection experiment: the rare task is excluded from training for G steps, then injected in a batch enlarged to m = G · B · ρ r rare samples so that its long-run frequency exactly matches the setup of Fig. 2. This emphasizes the ability of a model to retain memories about observed data, while preserving the total frequency with which it is seen. Results are shown in Fig. 4. We see at the end of training, rare-task signal decays monotonically with G at all widths, but far more steeply for smaller models. Meanwhile, the learning dynamics in panel (b) show that after each injection, a larger model accumulates rare-task signal and retains enough of it to build on the next injection, while a smaller model decays back to near-zero in between (an intuitive model explaining this dynamic is shown in Fig. 11 and analytically described in App. C.4). Overall, our results showing how larger models learn tasks smaller models do not can be summarized as follows.

Hypothesis: Scaling Enables Learning via Reduced Interference and Better Retention

Given the same training setup, larger models better learn tasks in the tail of the data distribution. When such a task is observed, a larger model can retain part of the update and build on it when the task appears again. In a small model, the same parameters face more competition: updates from frequent tasks undo the rare-task update before the next rare batch arrives. Rare-task learning then becomes an update-and-forget loop.

#4.1 Setup

A key variable in our claims is the frequency of a task. 4 However, measuring the frequency of a naturally occurring task in pre-training data is challenging, as instances from the same task can occur in many surface forms. To tightly control task frequency, we adopt a data injection framework from the memorization literature [ 58 – 61 ]. We inject different instances sampled from the distribution of a “special” task T at a controlled frequency f to measure whether a model has learned the task distribution. The task T is special in the sense that it is unlikely to be part of normal pre-training data. We then train models of various sizes on data mixtures generated from different values of f.

Tasks. We consider two special tasks T: comparison (T CMP) and modular addition (T ADD). Both tasks are encoded as a sequence of three tokens: TOK1, TOK2, LABEL, where TOK1, TOK2 ∈S, a set of 100 tokens randomly sampled from the vocab. There are exactly 10 K instances per task, which are split 50 / 50 for training and testing. Critically, both tasks require models to learn certain geometrical structures to generalize [62]. This provides a measure for learning a task (as opposed to memorizing training instances) and a set of features to verify the interference hypothesis of Sec. 3.

Data. We use Dolma v1.7 as the pre-training corpus [63]. Given a task T, we inject instances sampled from its train split at a frequency of 7. 8 × 10 − 3 to 2. 4 × 10 − 8, roughly from 1K instances per batch to 1 instance every 10 batches. To ensure the injected task frequency is comparable to the frequency of tasks learned in pre-training, we sample two reference tasks R cmp and R add from pre-training that involve similar high-level functions. The three-token sequence plus an end of document token replace the first four tokens of a training sequence. See App. B.3 for further details on the experimental setup.

4 Defining the complexity of a natural task is difficult, and hence we solely focus on frequency in this section.

Models. We train OLMo models [64] with 4M, 20M, 300M, 1B, and 4B parameters. We focus on scaling the models’ hidden and MLP dimensions and the number of attention heads; the 4M parameter model has depth 8 and the rest have depth 16. See App. B.2 for further details.

#4.2 Behavioral Evidence

Larger Models Learn Rarer Tasks. We first replicate the behavioral findings in Sec. 3.1. We measure the effect of task frequency by comparing multiple training runs that only differ by the frequency of the injected task. As shown in Fig. 5, larger models learn lower-frequency tasks much better than smaller models do. This matches the pattern in Fig. 2. Moreover, tasks are learned in the order of frequency. For each model run, we compare the order in which the injected task T CMP and the reference tasks are learned, as shown in Fig. 6 a. Most importantly, larger models do not just lead to better memorization of training instances, i.e., low training loss, but also learn generalizable task structures, i.e., high eval accuracy. On T ADD, only larger models trained on higher frequency exhibit the grokking phenomenon [65].

Rare-Task Retention Has an Effect on Learning. We conduct the matched-frequency injection experiment as described in Fig. 4, i.e., injecting N task instances every N batches, for N = 1, 10, 20, 50, 100. Fig. 6 b shows the effects of retention on learning, as models trained with larger gaps between task instances have higher task loss, even though the global task frequency of all runs is equivalent.

#4.3 Representational Evidence

Task Features. In our toy setting (Sec. 3), we know analytically which features are necessary for learning the k th task, i.e., B k, and to what extent the model can represent these features, i.e., P U. For our OLMo models, we can empirically identify a set of causal features that a pre-trained LM would use to solve the task and localize them in the model representations. Specifically, for T CMP, the task feature of core relevance is the global order of the tokens, which allows number comparisons; meanwhile for T ADD, task features are the Fourier modes [ 66 – 68 ], as shown in Fig. 7. We visualize the feature geometry using the largest model trained on the most frequent task data. These task features allow us to conduct versions of the gradient and representation-level analyses in Sec. 3.

More Task Features are Present in Larger Model Representations. We first localize the task features in models that have clearly learned the task. We then measure to what extent these target task features are present in all models, which parallels the metric ℓ k (U) used in the toy setting. For localization, we use distributed alignment search (DAS) [69] which finds subspaces that causally

encode the features. For T CMP, a global ordering of the tokens can be localized to a 1-D subspace in the residual stream of the first layer. For T ADD, Fourier modes can be identified in the residual stream from earlier layers to the last layer. We then use task-specific metrics to measure to what extent these task features are present in model representations. For T CMP, since the geometry of the task feature is a single direction, we apply linear regression to representations spanning the top K = 50 principal components. For T ADD, we measure the total number of Fourier modes present through all layers. We include the details in App. B.4. The last two columns of Fig. 7 show the extent to which the target task features are present in each model across checkpoints. For the frequency (middle) column, we fix the model size to 1B. For the model size (last) column, we fix the task frequency to be 2. 4 × 10 − 6 for T CMP and 7. 8 × 10 − 3 for T ADD. We see that (i) the presence of task features is highly correlated with high accuracy on the test set, and (ii) larger models and models trained on more frequent task data clearly learn these task features faster.

#4.4 Gradient Evidence

We now connect the behavioral evidence (Sec. 4.2) and the internal representation account (Sec. 4.3) by analyzing how task gradients interfere with non-task gradients on a set of neurons that implement the task circuit. We focus on T CMP training runs in Fig. 8, where 100 task instances are injected every 100 steps.

Task Neurons. We first identify which MLP layers implement the task features defined in Sec. 4.3. For all the models that we compared, the first layer MLP has the largest causal effects on task predictions. We further identify the top K neurons in the first layer MLP that have the largest gradient magnitude and use the gradients of these neurons for analysis. Details can be found in App. B.5.

Figure 8: Rare-Task Retention. Larger models can retain the injected task information better, i.e., larger task eval loss drop, when injecting task instances every 100 batches.

Task Reference Direction g r. We estimate the task reference direction using the aggregated gradient of the task loss computed over all 10K task instances, an analogy to G r in the toy setting. This direction may shift across training steps; however, at a given step, it is the optimal task direction.

Larger Models Show Less Gradient Interference Between General Language Modeling and Our Injected Task. We quantify the relation between the task reference g r and the batch gradient g, which can be further decomposed into gradient from the task tokens g t (if exists in batch) and non-task tokens g nt, i.e., g = g t + g nt. We first measure the cosine similarity between task reference and batch gradient direction, replicating the results in Fig. 4. We additionally analyze whether task or non-task tokens contribute to this similarity; while task token gradient aligning with task reference g r is expected, non-task token gradient with non-zero cosine similarity suggests that the language modeling direction is interfering with the task gradient direction.

Results are shown in Fig. 9. In the top panel, larger models have higher similarity between g and g r at the injection steps, 0. 08 ± 0. 02 for the 1B model and 0. 04 ± 0. 04 for the 300M model, the similarity typically regresses towards zero between injections. For the 20M model, the similarity scores oscillate wildly across batches, even at the injection step. In fact, the high similarity between non-task gradient g nt and g r reveals that for the 20M model the batch gradient similarity mostly comes from random collisions with task direction, with a similarity score of 0. 10 ± 0. 09, while for larger models, g nt is almost orthogonal to g r, with 7. 58 × 10 − 5 ± 0. 02 for the 1B model, suggesting little to no gradient interference on this set of neurons.

#5 Discussion

We develop a data-centric account of why larger models can learn tasks that smaller models fail to learn. Specifically, we show that larger models can learn rare tasks from the data mixture, and this phenomenon is explained by learning dynamics, i.e., competition of resources and retention of memories, as well as the task frequency and complexity. Our perspective highlights that understanding scaling requires thinking beyond model expressivity. We need to understand how learning dynamics are at play with task frequency and complexity. It also points toward more intentional design of data mixtures to better elicit target capabilities. For example, simply scaling up the frequency of a target task might provide a more efficient way to learn the task than scaling up the model size. Lastly, our findings on how better retention of memories enables learning rare tasks offers a new perspective that views memorization as a mechanism that can support learning abstraction: by retaining task instances longer, models can accumulate signals across batches to learn more generalizable structures of the task. This suggests memorization can in fact be beneficial, in line with arguments by Feldman [70].

#Limitations

As noted above, our account for why larger models learn more emphasizes the interplay of learning dynamics, task frequency, and task complexity. However, as discussed in Sec. 1, there are other plausible accounts for explaining this phenomenon, e.g., ones that focus on model expressivity and sample efficiency. Our explanation hence should not be interpreted as a complete account of scaling. Instead, these explanations are complementary: expressivity constrains what can be represented, sample efficiency shapes how effectively data is used, and our account highlights how learning dynamics interact with the frequency and complexity of tasks. A full understanding likely requires accommodating all these explanations, rather than viewing them as competing hypotheses. We also note we validated our key theoretical results using the OLMo pre-training pipeline, finding the empirical results on the injected tasks strongly match what the theoretical results predict. However, we acknowledge that empirical validation in a realistic pre-training setting could still leave some analytic gaps. For example, we did not empirically verify behavior of larger-scale language models or over-trained language models. We also selected injected tasks that matched the frequency of tasks learned in OLMo pre-training, which does not rule out other scaling behaviors with extreme task frequency. Our empirical results should therefore be viewed as supporting evidence. We encourage future work to explore different training regimes, more tasks, and different frequency ranges.

#Acknowledgments

The authors thank Blake Bordelon, Jacob Zavatone-Veth, and Core Francisco Park for several useful references that helped concretize the claims posited in this work, and Yasaman Bahri, Surya Ganguli, Ari Holtzman, Stephanie Chan, Freya Behrens, Tom McGrath, Owen Lewis, Atticus Geiger, Jack Merullo, and Thomas Fel for fruitful conversations during the course of this project. The authors also thank Thomas Icard for several comments on an earlier version of this draft. This research is supported in part by a grant from Open Philanthropy (Coefficient Giving) to CP.

#References

[1] Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025.

[2] Anthropic. System Card: Claude Mythos Preview, 2026. https://www-cdn.anthropic. com/08ab9158070959f88f296514c21b7facce6f52bc.pdf.

[3] Google DeepMind. Gemini 3 Pro - Model Card, 2026. https://storage.googleapis. com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf.

[4] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.

[5] DeepSeek-AI. DeepSeek-V4-Pro, 2026. https://huggingface.co/deepseek-ai/ DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf.

[6] Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026.

[7] Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, et al. Measuring AI ability to complete long software tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.

[8] Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:2411.04872, 2024.

[9] ARC Prize Foundation. ARC-AGI-3, 2026. https://arcprize.org/arc-agi/3.

[10] Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Bench-marking agents on hard, realistic tasks in command line interfaces. In The Fourteenth Inter-national Conference on Learning Representations, 2026. URL https://openreview.net/ forum?id=a7Qa4CcHak.

[11] Anthropic. Responsible Scaling Policy, 2026. https://www.anthropic.com/ responsible-scaling-policy.

[12] OpenAI. Our Approach to Frontier Risk, 2023. https://openai.com/global-affairs/ our-approach-to-frontier-risk/.

[13] Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025.

[14] Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun. Predicting emergent abilities with infinite resolution evaluation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=lDbjooxLkD.

[15] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=yzkSU5zdwD.

[16] Jason Wei. 137 emergent abilities of large language models, 2022. https://www.jasonwei. net/blog/emergence.

[17] Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023.

[18] Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum? id=35DAviqMFo.

[19] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023.

[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.

[21] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017.

[22] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ryenvpEKDr.

[23] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hee-woo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020.

[24] Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021.

[25] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021.

[26] Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300– 22312, 2022.

[27] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

[28] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=iBBcRUlOAPR.

[29] Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/ forum?id=NLoaLyuUUF.

[30] Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=nbOY1OmtRc.

[31] Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=dEypApI1MZ.

[33] Licong Lin, Jingfeng Wu, Sham M. Kakade, Peter L. Bartlett, and Jason Lee. Scaling laws in linear regression: Compute, parameters, and data. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 60556–60606. Curran Associates, Inc., 2024. doi: 10. 52202/079017-1937. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/file/6fcb1afcc1e9c2c82c8ddddf03bcf0f6-Paper-Conference.pdf.

[34] Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 28699–28722. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/file/5b6346a05a537d4cdb2f50323452a9fe-Paper-Conference.pdf.

[35] Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.16859, 2022.

[36] Ekdeep Singh Lubana, Kyogo Kawaguchi, Robert P. Dick, and Hidenori Tanaka. A percolation model of emergence: Analyzing transformers trained on a formal language. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=0pLCDJVVRD.

[37] Francesco Cagnetta, Hyunmo Kang, and Matthieu Wyart. Learning curves theory for hierarchically compositional data with power-law distributed features. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id= Lw0kC75dY0.

[38] Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi, and Matthieu Wyart. Scaling laws and representation learning in simple hierarchical languages: Transformers vs. convolutional architectures. arXiv preprint arXiv:2505.07070, 2025.

[39] Francesco Cagnetta, Allan Raventós, Surya Ganguli, and Matthieu Wyart. Deriving neural scaling laws from the statistics of natural language. arXiv preprint arXiv:2602.07488, 2026.

[40] Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Pareto frontiers in deep feature learning: Data, compute, width, and luck. Advances in Neural Information Processing Systems, 36:48021–48034, 2023.

[41] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=i1uGbfHHpH.

[42] Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforce-ment learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026.

[43] Bloomberg. OpenAI Claims DeepSeek Distilled US Models to Gain an Edge, 2026. https://www.bloomberg.com/news/articles/2026-02-12/ openai-accuses-deepseek-of-distilling-us-models-to-gain-an-edge?

[44] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.

[45] Huajian Xin, Z. Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du,, et al. Deepseek-prover-v1.5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=I4YAIwrsXa.

[46] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=3zKtaqxLhW.

[47] Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026.

[48] Yunhao Tang, Sid Wang, Lovish Madaan, and Remi Munos. Beyond verifiable rewards: Scaling reinforcement learning in language models to unverifiable data. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview. net/forum?id=pc6M9h3T9m.

[49] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025.

[50] Cody Blakeney, Jessica Zosa Forde, Jonathan Frankle, Ziliang Zong, and Matthew L Leav-itt. Reduce, reuse, recycle: Improving training efficiency with distillation. arXiv preprint arXiv:2211.00683, 2022.

[51] Shikai Qiu, Lechao Xiao, Andrew Gordon Wilson, Jeffrey Pennington, and Atish Agarwala. Scaling collapse reveals universal dynamics in compute-optimally trained neural networks. In Forty-second International Conference on Machine Learning, 2025. URL https:// openreview.net/forum?id=Fvq9ogLnLN.

[52] Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+3 phases of compute-optimal neural scaling laws. Advances in Neural Information Processing Systems, 37:16459–16537, 2024.

[53] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 OLMo 2 Furious, 2024. URL https://arxiv.org/abs/2501.00656.

[54] Yedi Zhang, Andrew M Saxe, and Peter E. Latham. Saddle-to-saddle dynamics explains a simplicity bias across neural network architectures. In The Fourteenth International Con-ference on Learning Representations, 2026. URL https://openreview.net/forum?id= Vit5M0G5Gb.

[55] Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory, pages 2552–2623. PMLR, 2023.

[56] Arthur Jacot, François Ged, Berfin ¸Sim¸sek, Clément Hongler, and Franck Gabriel. Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv:2106.15933, 2021.

[57] Daniel Kunin, Giovanni Luca Marchetti, Feng Chen, Dhruva Karkada, James B Simon, Michael R DeWeese, Surya Ganguli, and Nina Miolane. Alternating gradient flows: A theory of feature learning in two-layer neural networks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum? id=t7LKc0MMW6.

[58] Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, and Chiyuan Zhang. Measuring forgetting of memorized training examples. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=7bJizxLKrR.

[59] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284, Santa Clara, CA, August 2019. USENIX

Association. ISBN 978-1-939133-06-9. URL https://www.usenix.org/conference/ usenixsecurity19/presentation/carlini.

[61] Johnny Wei, Ameya Godbole, Mohammad Aflah Khan, Ryan Yixiang Wang, Xiaoyuan Zhu, James Flemings, Nitya Kashyap, Krishna P. Gummadi, Willie Neiswanger, and Robin Jia. Hubble: a model suite to advance the study of LLM memorization. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview. net/forum?id=ZfdnZhOP0k.

[62] Hyeonbin Hwang and Yeachan Park. Intrinsic task symmetry drives generalization in algorithmic tasks, 2026. URL https://arxiv.org/abs/2603.01968.

[63] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024. URL https://huggingface.co/datasets/allenai/dolma.

[65] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.

[66] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Confer-ence on Learning Representations, sep 2022. URL https://openreview.net/forum?id= 9XFSbDPmdW.

[67] Tianyi Zhou, Deqing Fu, Vatsal Sharan, and Robin Jia. Pre-trained large language models use fourier features to compute addition. In The Thirty-eighth Annual Conference on Neu-ral Information Processing Systems, 2024. URL https://openreview.net/forum?id= i4MutM2TZb.

[68] Sheridan Feucht, Tal Haklay, Usha Bhalla, Daniel Wurgaft, Can Rager, Raphaël Sarfati, Jack Merullo, Thomas McGrath, Owen Lewis, Ekdeep Singh Lubana, Thomas Fel, and Atticus Geiger. Arithmetic in the wild: Llama uses base-10 addition to reason about cyclic concepts, 2026. URL https://arxiv.org/abs/2605.01148.

[71] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.

[72] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201–34227, 2023.

[73] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024.

[74] Sang Michael Xie. Foundation Models from a Data-Distributional View. Stanford University, 2024.

[75] Rahul Ramesh, Jialin Mao, Itay Griniasty, Rubing Yang, Han Kheng Teoh, Mark Transtrum, James P Sethna, and Pratik Chaudhari. A picture of the space of typical learnable tasks. arXiv preprint arXiv:2210.17011, 2022.

[76] Rahul Ramesh. The Principles of Learning on Multiple Tasks. PhD thesis, University of Pennsylvania, 2025.

[77] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobei-dli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Bench-marks Track, 2023. URL https://openreview.net/forum?id=kM5eGcdCzq.

[78] Guilherme Penedo, Hynek Kydlíˇcek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024.

[79] Pratyush Maini, Vineeth Dorna, Parth Doshi, Aldo Carranza, Fan Pan, Jack Urbanek, Paul Burstein, Alex Fang, Alvin Deng, Amro Abbas, et al. Beyondweb: Lessons from scaling synthetic data for trillion-scale pretraining. arXiv preprint arXiv:2508.10975, 2025.

[80] Dylan Sam, Sachin Goyal, Pratyush Maini, Alexander Robey, and J Zico Kolter. When should we introduce safety interventions during pretraining? arXiv preprint arXiv:2601.07087, 2026.

[82] Rich Caruana. Multitask learning. Machine learning, 28(1):41–75, 1997.

[83] Rahaf Aljundi. Continual learning in neural networks. arXiv preprint arXiv:1910.02718, 2019.

[84] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in neural information processing systems, 34: 18878–18890, 2021.

[85] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019.

[86] Runzhe Wu, Ankur Samanta, Ayush Jain, Scott Fujimoto, Jeongyeol Kwon, Ben Kretzu, Youliang Yu, Kaveh Hassani, Boris Vidolov, and Yonathan Efroni. Imbalanced gradients in rl post-training of multi-task llms. In Findings of the Association for Computational Linguistics: EACL 2026, pages 3137–3150, 2026.

[87] Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34:1256–1272, 2021.

[88] Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pages 4028–4079. PMLR, 2022.

[89] Martin Marek, Dongkyu Cho, Shikai Qiu, Rumi Chunara, Pavel Izmailov, and Andrew Gordon Wilson. Forgetting in language models: Capacity, optimization, and self-generated replay, 2026. URL https://arxiv.org/abs/2605.26097.

[90] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33:5824–5836, 2020.

[91] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.

[92] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gra-dient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794–803. PMLR, 2018.

[93] Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal gradients. arXiv preprint arXiv:1912.06844, 2019.

[94] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International conference on artificial intelligence and statistics, pages 3762–3773. PMLR, 2020.

[95] Peter L Chen, Xiaopeng Li, Xi Chen, and Tianyi Lin. Reward-free alignment for conflicting objectives. arXiv preprint arXiv:2602.02495, 2026.

[96] Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=GhVS8_yPeEa.

[97] Darshil Doshi, Tianyu He, Aritra Das, and Andrey Gromov. Grokking modular polynomials. arXiv preprint arXiv:2406.03495, 2024.

[98] Pulkit Gopalani, Ekdeep S Lubana, and Wei Hu. Abrupt learning in transformers: A case study on matrix completion. Advances in Neural Information Processing Systems, 37:55053–55085, 2024.

[100] Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics. In The Twelfth International Confer-ence on Learning Representations, 2024. URL https://openreview.net/forum?id= vt5mnLVIVo.

[101] Dashiell Stander, Qinan Yu, Honglu Fan, and Stella Biderman. Grokking group multiplication with cosets. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=hcQfTsVnBo.

[102] Mohamad Amin Mohamadi, Zhiyuan Li, Lei Wu, and Danica J. Sutherland. Why do you grok? a theoretical analysis on grokking modular addition. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=ad5I6No9G1.

[103] Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390, 2023.

[104] Depen Morwani, Benjamin L. Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham M. Kakade. Feature emergence via margin maximization: case studies in algebraic tasks. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=i9wDX850jR.

[105] Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=MO5PiKHELW.

[106] Chen Cheng, John Duchi, and Rohith Kuditipudi. Memorize to generalize: on the necessity of interpolation in high dimensional linear regression. In Conference on Learning Theory, pages 5528–5560. PMLR, 2022.

[108] Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667–766, 2022.

[110] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Confer-ence on Learning Representations, 2020. URL https://openreview.net/forum?id= B1g5sA4twr.

[111] Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent. arXiv preprint arXiv:1912.07242, 2019.

[112] Preetum Nakkiran, Prayaag Venkat, Sham M. Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=7R7fAoUygoa.

[113] Daniel Wurgaft, Ekdeep Singh Lubana, Core Francisco Park, Hidenori Tanaka, Gautam Reddy, and Noah Goodman. In-context learning strategies emerge rationally. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https: //openreview.net/forum?id=bBUUOQI0N6.

[114] Aaditya K Singh, Ted Moskovitz, Felix Hill, Stephanie C. Y. Chan, and Andrew M Saxe. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=O8rrXl71D5.

[115] Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers. Advances in neural information processing systems, 36:27801–27819, 2023.

[116] Core Francisco Park, Ekdeep Singh Lubana, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning. In The Thirteenth International Confer-ence on Learning Representations, 2025. URL https://openreview.net/forum?id= XgH1wfHSX8.

[117] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International conference on machine learning, pages 15696–15707. PMLR, 2023.

[119] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=TatRHT_1cK.

[120] Rohith Kuditipudi, Jing Huang, Sally Zhu, Diyi Yang, Christopher Potts, and Percy Liang. Blackbox model provenance via palimpsestic membership inference. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL

https://openreview.net/forum?id=VRhVS59yhP.

[121] Dmitrii Krasheninnikov, Richard E. Turner, and David Krueger. Fresh in memory: Training-order recency is linearly encoded in language model activations. In The Fourteenth Interna-tional Conference on Learning Representations, 2026. URL https://openreview.net/ forum?id=Tn6famjSxN.

[122] Sunny Duan, Mikail Khona, Abhiram Iyer, Rylan Schaeffer, and Ila R Fiete. Uncovering latent memories in large language models. In The Thirteenth International Conference on Learning Representations, 2025.

[123] Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=TYdzj1EvBP.

[124] Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274–38290, 2022.

[125] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487, 2022.

[126] Steven T Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112–1130, 2014.

[127] Aapo Hyvärinen, Jarmo Hurri, and Patrick O Hoyer. Natural image statistics: A probabilistic approach to early computational vision., volume 39. Springer Science & Business Media, 2009.

[128] Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Scaling and renormalization in high-dimensional regression. arXiv preprint arXiv:2405.00592, 2024.

[129] Yunwei Ren, Eshaan Nichani, Denny Wu, and Jason D. Lee. Emergence and scaling laws in SGD learning of shallow neural networks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum? id=kA2H90nm26.

[130] Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter J Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, et al. Scaling exponents across parameterizations and optimizers. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=0ksNeD1SJT.

[131] Eric J Michaud, Liv Gorton, and Tom McGrath. Understanding sparse autoencoder scaling in the presence of feature manifolds. arXiv preprint arXiv:2509.02565, 2025.

[132] Yoonsoo Nam, Nayara Fonseca, Seok H Lee, Chris Mingard, and Ard A Louis. An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem. Advances in Neural Information Processing Systems, 37:39632–39693, 2024.

[133] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJl-b3RcF7.

[134] Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International conference on machine learning, pages 6682–6691. PMLR, 2020.

[135] Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopou-los. Optimal lottery tickets via subset sum: Logarithmic over-parameterization is sufficient. Advances in neural information processing systems, 33:2599–2610, 2020.

[136] Ian Magnusson, Nguyen Tai, Ben Bogin, David Heineman, Jena D. Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A. Smith, Pang Wei Koh, and Jesse Dodge. Datadecide: How to predict best pretraining data with small experiments. In Forty-second International Conference on Machine Learning, 2025. URL https:// openreview.net/forum?id=p9YlQPF8fE.

[137] Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=u2vAyMeLMm.

[140] Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, and Cengiz Pehlevan. Feature-learning networks are consistent across widths at realistic scales. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=LTdfYIvbHc.

#A Related Work

Multi-Task Learning. Data distributions neural networks are trained on are often deemed as a mixture of tasks [ 63, 71 – 81 ]. This motivated works analyzing both the learning dynamics of training toy models on multi-task distributions and defining methods aimed at reducing interference between updates caused by learning a task in the presence of other ones. For example, the notion of “catastrophic interference” has been often characterized in the multi-task learning and continual learning literature [82, 83], where task gradients conflict or are imbalanced in scale, leading to learning of only a subset of tasks instead of the entire mixture. Such phenomenology can be intuitively [ 84 – 86 ] and theoretically explained: e.g., Pezeshki et al. [87] posit the idea of gradient starvation, whereby a model trained on a mixture of tasks that have different prior frequencies is unable to learn the infrequent task due to its gradient getting “starved” out, i.e., becoming zero; meanwhile, Evron et al. [88] characterize how tasks’ observation frequency induces the forgetting of another learned task in a sequential linear regression setting. Concurrent to our work, Marek et al. [89] show that forgetting of prior tasks occur when a model has little remaining capacity. These analyses have also motivated methods to avoid interference and enable learning of multiple tasks: e.g., methods that perform “surgery” on model gradients [ 90 – 92 ] to make two conflicting tasks’ gradients to have zero interaction by removing one’s projection towards another [93, 94]; these methods have seen use at scale as well [95].

It is worth noting that our results are in a similar vein as literature above, but augment prior work by characterizing the effects of scale and showcasing that even extremely rarely observed tasks can eventually be learned if one’s model is large enough—empirically, related results corroborating our claim in a vision scenario was also made in the continual learning literature by Ramasesh et al. [96]. That said, we emphasize that neither do our results imply nor do we claim (in fact, we say otherwise) that scale alone is the mechanism to enable the learning of a rare task in the presence of other frequent ones. Indeed, methods discussed above from multi-task / continual learning literature show multiple tasks can be simultaneously learned by a model.

Memorization and Scaling. The core mechanism posited in our work for how larger models learn rare tasks involves a model retaining some signature of observed data from a small batch of samples. In extreme scenarios, e.g., when only a few samples are contained in the batch, such a signature cannot possibly correspond to a general, abstract task representation. Instead, the signature can be thought of as a model (at least partially) “memorizing” an observation—once enough observations occur and the memories aggregate, in our simple toy settings, we find the model consolidates the memories into an abstract representation that generalizes well. In this sense, we emphasize our core proposition suggests memorization is not an undesirable property, but instead a prerequisite to eventual generalization for rare tasks. This mechanism is highly reminiscent of the observations [65, 97, 98] and posited learning dynamics for grokking, where a model transitions from memorizing observations to generalizing to novel inputs [ 66, 99 – 104 ]. Critically, our language model pretraining results, where we use modular addition, i.e., the prototypical grokking task, and find our posited dynamics hold is suggestive that grokking-like dynamics may in fact occur in practice, especially for rarely observed tasks (closest result to this end is perhaps the syntax acquisition dynamics demonstrated by Chen et al. [105]). This argument is in keeping with theoretical works on classification that have argued that memorization is necessary for generalization—for example, to handle label noise [106], or to handle rare examples [70]. Brown et al. [107] provide a particularly interesting demonstration that learning rare structures effectively requires memorizing even irrelevant information about the data. On the other hand, it is worth considering if an opposite mechanism may occur for frequently observed tasks: e.g., if a model sees too many observations of the same task, does it perhaps undergo phenomenology such as overfitting, which is generally associated with generalization to memorization dynamics; if so, does scaling help avoid this dynamic via mechanisms such as double descent [ 108 – 112 ]? Recent work on transient nature of in-context learning capabilities in toy scenarios [ 113 – 116 ] is suggestive such a dynamic may occur, and we thus argue it is worth investigating what the counterpart of our work for learning dynamics of frequent tasks looks like.

Building on the above, we also note memorization and the effects of scaling have been often studied in literature; these results are in line with our claims on reduced interference over model parameters via scaling, enabling models to eventually learn rare tasks. For example, studying memorization in the sense of verbatim match (e.g., ktoken string match), works show larger models learn knowledge present in the tail-end of the distribution better [117], larger models [118, 119] and later checkpoints

tend to memorize more [60], not just individual data points but also training data order [120, 121], and these memories are retained for longer across injection events [122, 123]. Tirumala et al. [124] show that larger models memorize more, but also can memorize more of the data before they begin to overfit.

Generalization and Scaling. Improved performance as a function of scaling has defined the spirit of machine learning since scaling laws first started being used for identifying training configurations [ 20 – 25, 28, 125 ]. The precise mechanism as to how scaling helps produce better models is unclear, but a few propositions have been made. For example, in works assessing how power-law scaling as a function of data and parameters emerges, prior work has exploited the argument that natural data statistics are heavy-tailed and follow power-law trends (e.g., Zipf-priors in language [126] and vision [127]); correspondingly, scaling enables access to lower-order modes of the data distribution [32, 33, 35, 37, 38, 128, 129], and arguments to this end have been verified in recent work by Cagnetta et al. [39]. Our toy setup was in fact inspired by these papers, especially Ren et al. [129] and Maloney et al. [35], but is a substantial simplification since, unlike these prior works, our goal was not to characterize the eventual steady state optima a model arrives at, but instead the dynamics that lead to it. Works closer to this dynamical motivation are by Bordelon et al. [30, 31], Paquette et al. [52], Atanasov et al. [128], and Everett et al. [130], who analyze learning dynamics of toy settings that exhibit power-law scaling curves. However, since we primarily aimed to posit a concrete mechanism via which larger models may be able to learn tasks smaller models do not, we note the concrete results emphasized and takeaways across these works versus ours are fairly different. In particular, these papers primarily focus on the interaction between learning dynamics and data statistics to identify different regimes of scaling, i.e., what functional form, e.g., power-law or otherwise, results in the best effective characterization of learning dynamics. Finally, works by Michaud et al. [34, 131] and Nam et al. [132] are fairly related to our paper: specifically, these works characterize an explicitly multi-task construction to posit a model for how power-law scaling can emerge in neural networks. While still related to the data statistics argument mentioned above, these works also have an explicit notion of prior frequency and (implicitly) show scaling helps learn tasks that are rarely observed in the training distribution. Our work makes this claim explicit, but also characterizes how, i.e., a mechanism, via which scaling aids learning of rare tasks.

Lottery Tickets and Scaling. Another thread of research that partially connects the work listed above on scaling and learning of specific tasks is on the lottery ticket hypothesis [133]: a lottery ticket is defined as a subnetwork identified from a larger, initial network that, even at random initialization, shows the ability to perform the task one is training their model for. Theoretical work on lottery ticket hypothesis has characterized bounds on how much larger a model has to be in order to possess a subnetwork that can, up to some error, approximate the model eventually learned via training [40, 134, 135]. Especially related here is the work of Edelman et al. [40], who show that via scaling model width (the scaling axis we consider as well), the odds that a subset of representations with non-trivial alignment with true task features exist substantially increases. Correspondingly, scaling improves sample efficiency of learning tasks that require more features (i.e., are more complex); critically, if one slightly generously interprets the authors’ results, they are suggestive that a larger model will be able to learn rare tasks by virtue of already possessing features a smaller model will be unable to learn (due to sparsely observed training signal for such tasks). While this work partially informed the intuition guiding this paper, we note the eventual results for our setting and verification on large-scale scenarios are more concrete.

#B Experimental Details

B.1 Synthetic Experiment

In the following, we describe experiment details and metrics relevant to results for the synthetic setup.

Data-generating process. All synthetic runs use the orthogonal-block instantiation of the mixtureof-regressions setup proposed in Sec. 3. We fix the ambient dimension at D = 1024, the number of tasks at K ∈{ 16, 32 } (almost all figures use K = 32), and a per-task block dimension d T such that K · d T ≤ D and the task blocks are mutually orthogonal. Concretely, task k occupies coordinates [ k d T, (k +1) d T) of R D, and its within-block spectrum is the power-law σ k,j = j − α k for j = 1,..., d T. Unless stated otherwise we use a shared exponent α k ≡ α across tasks (α = 1 in the orthogonal-block experiments, making the within-block decay slow enough that capacity reliably spreads beyond the leading mode of each task). The task prior is the power-law π k ∝ k − β, normalized to sum to one over k = 1,..., K; β = 2 in most experiments. Inputs are sampled fresh each step as x ∼N (0, σ 2 in I D) with σ in = 1 in all orthogonal-block runs. The per-task targets are y k = Λ 1 / 2 k B ⊤ k x, restricted to the task’s block; because each task block has rank d T, the output dimension of the regressor is d T (and reduces to 1 when d T = 1, e.g., in the rank-1 specialization of App. E.1.2).

Model. The student is the linear-bottleneck regressor of Sec. 3: a shared encoder W ∈ R N × D that maps the input to an Ndimensional hidden, followed by per-task linear decoders D k ∈ R d T × N selected by the ground-truth task index supplied in the batch. We do not explicitly constrain W to have orthonormal rows: the relevant object for Theorem 3 is the projector P W = W ⊤ (WW ⊤) − 1 W, which is invariant to the right-multiplicative gauge of W and which gradient flow drives toward the top- N eigenspace of M = P

k π k C k regardless of the parametrization. The encoder is initialized such that W ⊤ W = I N at step zero, and the per-task decoders are initialized with Kaiming-uniform fan-in / linear gain. The decoders are jointly optimized with the encoder rather than analytically closed-formed at each step, since learned decoders will converge to D ∗ k = Λ 1 / 2 k B ⊤ k U at any stationary point, so the joint optimization does not change the encoder fixed point but does match the practical setting in which both ends of the bottleneck are learned simultaneously.

Optimizer. We use AdamW with default hyperparameters and an inverse-square-root learning-rate schedule. Gradients are clipped at maximum norm 1. 0. Batches are drawn fresh each step (no fixed dataset, no replay) with batch size B = 1 024 for the phase-diagram and rank-1 sweeps, and B = 512 for the matched-frequency retention sweeps; the smaller batch in the retention runs is required so that an injection batch with m ≤ B rare-task slots can match the long-run frequency ρ r = m/ (G · B) at the ρ r ≈ 6 × 10 − 4 end of the sweep.

Metrics. We track three families of metrics, all reported on freshly sampled batches separate from the training stream. The first is the per-task loss, i.e., the unnormalized population MSE ℓ k (U) = E  ∥ y k − D k U ⊤ x ∥ 2 2 , and its normalized counterpart ℓ k (U) /ℓ k, baseline with ℓ k, baseline = ∥ a k ∥ 2 2 /d T the mean-predictor MSE per task. The second is the per-task subspace alignment, the basis-free quantity s k (U) = Tr(PUC k) / Tr(C k) = ∥ P U a k ∥ 2 2 / ∥ a k ∥ 2 2, computed via the SVD of W so that it is independent of the gauge of the encoder. s k lies between N/D at random initialization and 1 when the task block is fully captured. We also report its random-baseline-corrected normalizations̃ k (U) = (s k (U) − N/D) / (1 − N/D), which equals 0 at random initialization and 1 at full capture. The third is the residual common-task signal: we compute δ F (U) = P

k ∈ F π k (1 − s k (U)) ∥ a k ∥ 2 2, the residual energy of the frequent block. The frequent set F is the smallest top-prior set with cumulative mass at least 0. 8; under our power-law prior this yields | F | = { 6, 3, 2, 2 } for β ∈{ 0. 5, 1. 0, 1. 5, 2. 0 } respectively. Standard evaluation is performed every 1 000 – 2 000 steps on a held-out probe of the same population distribution, with the final checkpoint additionally re-evaluated for end-of-training summary statistics.

B.2 OLMo Pretraining Pipeline

Table 1: Model configurations by size.

Models. We use the OLMo model architecture [64]. For 4M to 1B models, we follow the model configuration and naming convention of Magnusson et al. [136]. We additionally include a 4B model to further evaluate width scaling, as shown in Table 1.

Training hyperparameters. We use the same batch size of 1024, window size of 4096 for T CMP and 1024 for T ADD, and a learning rate schedule with an initial learning rate of 3 × 10 − 4 and cosine with warmup schedule for all models. For the retention window ablation experiment, we use a smaller window size of 512 to reduce the training cost. For a full list of hyperparameters, refer to the OLMo-7B-0724 configuration. 5

Training pipeline. We use the OLMo code base. 6 Our usage is in line with its Apache-2.0 license. 7

Compute resources. All models are trained on a cluster of NVIDIA H200 GPUs.

B.3 Pre-training and Injected Task Data

Pre-training data. We use Dolma v1.7 as the pre-training corpus [63]. Specifically, we use the 210B tokens corresponding to the first 50K batches that OLMo-7B-0424 and OLMo-7B-0724 are trained on, in the exact same order.

Reference tasks. To ensure the injected task frequency is comparable to the frequency of tasks learned in pre-training, we sample two reference tasks R cmp and R add from pre-training that involve similar high-level functions. R cmp predicts a number larger than x in the prompt “it has increased from {x} to”. R add predicts the sum of two numbers smaller than 100 with the prompt “ {x} + {y} =”. We estimate the lower bound of their frequency in pre-training data using infini-gram [137] and observe models’ next token prediction loss on the task, which corresponds to the two dashed lines in Fig. 6 panel (a).

Injected tasks. We elaborate the task label here. Let val (·): S 7→ [0, 99] be a bijective mapping that assigns an integer value between 0 and 99 to each token. For T CMP, LABEL is one of two tokens randomly chosen from the vocab indicating whether val (TOK1) < val (TOK2). For T ADD, LABEL is the token in S whose value equals (val (TOK1) + val (TOK2)) mod 100. Below are a few instances from the comparison task: address analyze pony, resort zebrafish pony, cavities misconduct provisional, where pony and provisional are the two label tokens that represent True and False.

5 https://github.com/allenai/OLMo/blob/main/configs/official-0724/OLMo-7B-0724.yaml 6 https://github.com/allenai/OLMo 7 https://github.com/allenai/OLMo?tab=Apache-2.0-1-ov-file

B.4 Localizing and Measuring Task Features in Sec. 4.3

The comparison task T CMP. We first use distributed alignment search (DAS) to verify that the model’s prediction is causally dependent on a global token order feature, which is encoded in a 1-D subspace in the residual stream of the first few layers.

The task has a simple high-level causal model, namely X → O ← Y, where X, Y are the two inputs and O is the binary output. We consider the following intervention on the input variable X (or Y): Let a base example be x b, y b, o b and a source example be x s, y s, o s, an interchange intervention on X that sets the value of x b to x s should lead to a counterfactual label that corresponds to x s < y b.

In the neural model, we search for a low-dimensional subspace in the residual stream that plays the same causal role as the input variable X by training on 1 K counterfactual data pairs defined above. We use DAS to search across all layers above the input token position. We are able to find a 1-D subspace in the residual stream of the first layer that has an interchange intervention success rate of 96%. This proves that the model not only encodes the global token order in a low-dimensional space but actually uses this feature for prediction on the task T CMP. This allows us to use the global order feature to measure to what extent a model has learned the abstract task structure.

We use the DAS implementation from pyvene [138].

The modular addition task T ADD. As prior work studying modular addition has identified that grokked models use Fourier modes for addition [66], we conduct Fourier analysis on residual stream to measure the presence of Fourier modes.

For modulus P, define a real discrete Fourier transform basis on R P as follows:

ϕ cos k (n) = cos(2 πkn/P) || cos(2 πk · /P) ||, ϕ sin k (n) = sin(2 πkn/P) || sin(2 πk · /P) ||, k = 1,...,  P

.

At layer l, collect residual stream vectors h l ∈ R d grouped by output c = (a+b) mod P and compute the mean representation as:

v (l) c = E  h (l) | c = (a + b) mod P  ∈ R d, c ∈{ 0,..., P − 1 }.

This yields a matrix V (l) ∈ R P × d.

After row-centering V (l), the fraction of variance captured by frequency k is

P d j =1  ⟨ ϕ cos k, V (l):,j ⟩ 2 + ⟨ ϕ sin k, V (l):,j ⟩ 2 

P (l) k =

k P (l) k = 1

P d j =1 || V (l):,j − ¯ V (l):,j || 2, X

We consider a null-baseline for detecting Fourier modes in representations. Under uniform variance allocation across frequencies,

P null = 2 P − 1

Hence, a frequency k at layer l is identified as a Fourier mode if

P (l) k > θP null

We choose θ = 2 in our experiment, as we do not observe significant difference in grokking behavior for models that represent Fourier modes with stronger signals, e.g., θ = 3.

Finally, we define the total number of detected modes as follows, where L is the total number of layers.

L X

n k: P (l) k > 2 P null o

N features =

l =0

This corresponds to the y-axis in Fig. 7 (b) right panel.

B.5 Localizing Task Neurons in Sec. 4.4

We conduct null interventions on MLP layers to identify layers that have the largest causal effects on the model output. The results are shown in Fig. 10. This aligns with our observation from the DAS localization experiment that first layer is the earliest layer where the global token order is causally encoded. The localization result is consistent across the three models used in Sec. 4.4.

#C Proofs

C.1 Proof of Theorem 3

For fixed U, the taskk population loss is

ℓ k (U, D k) = E  ∥ Λ 1 / 2 k B ⊤ k x − D k U ⊤ x ∥ 2 2  (4)

= ∥ Λ 1 / 2 k B ⊤ k − D k U ⊤ ∥ 2 F, (5)

where the second identity uses x ∼N (0, I) and the standard relation E ∥ Ax ∥ 2 2 = ∥ A ∥ 2 F. This is a linear least-squares problem in D k, so the minimizer is

D ∗ k = Λ 1 / 2 k B ⊤ k U. (6)

Substituting back gives

ℓ k (U) = Λ 1 / 2 k B ⊤ k (I − P U) 2 F (7)

= Tr (I − P U) C k
. (9)

Summing with weights π k yields

K X

K X

k =1 π k ℓ k (U) = Tr(M) − Tr(U ⊤ MU), M:=

L N (U) =

k =1 π k C k. (10)

Because M is symmetric positive semidefinite with finite trace, minimizing L N (U) is equivalent to maximizing Tr(U ⊤ MU) over all orthonormal U. By Ky Fan’s maximum principle [139],

N X

max U ⊤ U = I N Tr(U ⊤ MU) =

i =1 µ i, (11)

where µ 1 ≥ µ 2 ≥· · · are the eigenvalues of M. Therefore any minimizer spans the top- N eigenspace of M and the optimal loss is

N X

i =1 µ i = X

L ∗ N = Tr(M) −

i>N µ i. (12)

For our generative process, we have

M = X

k,j π k λ k,j b k,j b ⊤ k,j, (13)

so the vectors b k,j are eigenvectors of M with eigenvalues u k,j = π k λ k,j. Thus the width- N optimum keeps the N largest utilities. If task k contributes n k (N) retained coordinates, then its residual loss is ℓ ∗ k (N) = X

j>n k (N) λ k,j. (14)

C.2 Proof of Theorem 4

Write G F (U) = 2(I − P U) M 1 / 2 F M 1 / 2 F U. Using ∥ AB ∥ F ≤∥ A ∥ F ∥ B ∥ op, where ∥. ∥ op denotes the operator norm, gives

$∥ G F ( U ) ∥ F ≤ 2 ∥ ( I − P U ) M 1 / 2 F ∥ F ∥ M 1 / 2 F U ∥ op \quad (15)$

≤ 2 q

Tr (I − P U) M F
p

λ 1 (M F) (16)

= 2 p

λ 1 (M F) δ F (U). (17)

C.3 Proof of Proposition 6

Let U denote the common-task width- N solution and let u i be one occupied common eigenvector with eigenvalue µ F i. Replace that vector by v i (θ) = cos θ u i + sin θ b r, (18) while keeping the remaining N − 1 directions fixed. Because u i is an eigenvector of M F and b r is orthogonal to the common block, the contribution of this one direction to the objective Tr(PUM) is ⟨ v i (θ), Mv i (θ) ⟩ = µ F i cos 2 θ + π r λ r sin 2 θ. (19) Subtracting the value at θ = 0 gives ∆Tr(PUM) = (π r λ r − µ F i) sin 2 θ. (20) Since L = Tr(M) − Tr(PUM), the loss change is ∆ L = (µ F i − π r λ r) sin 2 θ. (21) Hence perturbations toward b r decrease the loss if and only if π r λ r > µ F i. The rare feature invades first through the weakest occupied common direction, which has curvature µ F N, proving the stated threshold.

Example 7 (One neuron, two orthogonal tasks). Let a, b ∈ R d be orthonormal and consider rank-one tasks with covariances C a = aa ⊤ and C b = bb ⊤. A width-1 encoder is a unit vector u = cos θ a + sin θ b. (22) The task losses are ℓ a (u) = sin 2 θ, ℓ b (u) = cos 2 θ. (23) A gradient step on task a obeys θ + = θ − η sin(2 θ) + O (η 2), while a step on task b obeys θ + = θ + η sin(2 θ) + O (η 2). If task a appears with probability p and task b with probability q < p, then E [∆ θ | θ ] = η (q − p) sin(2 θ) + O (η 2), (24) which drives the neuron toward the common task. Near θ = 0, if a rare-task update is followed by G common-task updates, then θ G ≈ (1 − 2 η) G θ 0 ≈ e − 2 ηG θ 0. (25) Thus rare-task alignment decays exponentially across the gap between rare observations.

Proof. The loss identities follow from u ⊤ aa ⊤ u = cos 2 θ and u ⊤ bb ⊤ u = sin 2 θ. Differentiating yields d dθ sin 2 θ = sin(2 θ), d dθ cos 2 θ = − sin(2 θ), (26)

which gives the stated updates under gradient descent. Taking the expectation under the task mixture yields the drift formula. Linearizing sin(2 θ) ≈ 2 θ near zero gives the exponential decay estimate.

The dynamics posited above are also exemplified in Fig. 11.

#D Further Experimental Results: Complexity Sweeps

In the main paper, we kept “complexity”, i.e., the number of directions used for defining the target variable constant across tasks; specifically, tasks in the main paper require 5 directions to cover 90 %

P j = r j =1 λ k,j P

of the energy in the task spectrum (i.e., solving for r in arg min r

j λ k,j > 0. 90 gives r = 5). In this section, we vary this property by changing the power-law coefficient underlying the task spectrum, i.e., since λ k,j ∝ j − α k, we vary the range of α k across tasks. We define ranges of [ α min, α max ], split the range uniformly into K values, and assign the k th value to α k. The most frequent task is assigned the value α max, giving it the fastest decaying spectrum and hence making it the simplest task, while the rarest task is assigned the value α min, giving it the slower decaying spectrum and making it most complex. In particular, we choose ranges (see Fig. 12) such that the task complexity varies between [4, 7] and [2, 12] across K = 32 tasks.

D.1 Feature Utility Predicts Learning Order

D.2 Competition Dynamics Disallow Learning of the Rarest and the Most Complex Task

Our results above showed that learning trends under varying task complexity are modulated by both task frequency and complexity, as predicted by our account in Sec. 3. We now show the competition dynamics picture posited in that section continues to follow in these settings as well. In particular, we plot the learning of the top-3 most frequent tasks (measured by normalized signal; see Sec. 3 for details) and the rare-most task as a function of residual, i.e., signal remaining to be explained in the frequent tasks. As shown in Fig. 15, the critical width predicted to be necessary for learning of the rarest task, by rendering the residual sufficiently small for most frequent tasks, continues to hold true in this setup as well.

D.3 Reduced Interference Aids Learning of the Rarest and the Most Complex Task

We now validate our argument for how data-centric bottlenecks, i.e., the low-frequency and highcomplexity nature of a task, is circumvented by a larger model: by virtue of having more parameters, a larger model witnesses reduced interference over per-task gradients. To this end, we redo the batch-injection experiments from Fig. 4 and plot the retention dynamics for the lowest aggregate utility task across settings. Results are shown in Fig. 16. We see similar results as before: larger models show better retention of observed signal from a task, allowing them to bootstrap on these past observations and eventually learn the task. Meanwhile, a medium width model is able to do so only when the task is observed sufficiently frequently, i.e., the gap is low. Comparing with the case when the width is too low, we see the model never learns the task and the retention dynamics concretely show why: the model is unable to retain signal for the observed task for long enough.

#E Further Experimental Results: Frequency Sweeps

E.1 Features and Tasks are Learned in Order of Utility

E.1.1 Extending Phase Diagram

E.1.2 Simplified Case: Rank-1 Tasks

Theorem 3 predicts that a width- N minimizer retains the N task-features with largest utility u kj = π k λ kj. We obtain a sharp quantitative test of this claim by collapsing the orthogonal-block setup to its rank-1 specialization: setting d T = 1 and α k = 1 makes every task rank-1 with λ k = 1, so the utility ordering reduces to the prior ordering, and the predicted critical width for task k becomes

N crit (k) = # { j ̸ = k: π j λ j > π k λ k } = k, (27)

i.e., a perfectly linear staircase in task index.

Setup. We train the linear-bottleneck student described in Sec. 3 on a mixture of K = 32 rank-1 orthogonal tasks, ambient dimension D = 1024, and a power-law prior with exponent β = 2. We sweep the encoder width N ∈{ 1, 2, 3, 4, 6, 8, 10, 12, 14, 16, 20, 24, 28, 32, 40, 48, 64 } and read out the per-task subspace alignment s k (U) = ∥ P U b k ∥ 2 at the end of training (rank-1 specialization of the per-task signal of Sec. 3, so s k (U) lies between N/D at random initialization and 1 when b k is fully captured by the encoder subspace).

Result. See Figure 18. We overlay the empirical transition width N emp (k) = min { N ∈ grid: s k (U) > 0. 5 } on the theoretical staircase (27), finding almost perfect alignment (minimal disparities are an artifact of the sampled width grid).

E.2 Residual Controls Learning

We next provide further validation for results in Fig. 3 by repeating experiments across different values of exponents in the power-law task prior. Specifically we vary β from the set { 0. 5, 1. 0, 1. 5, 2. 0 }. For each run we compute the per-task signal s k (U) = Tr(PUC k) / Tr(C k), i.e., how well the model representation encodes information about the k th task, and the residual

k ∈ F π k (1 − s k (U)) ∥ a k ∥ 2 from the final checkpoint. The frequent set F is defined as the smallest set of tasks whose cumulative mass meets 0. 8; under β ∈{ 0. 5, 1, 1. 5, 2 } this yields | F | = { 6, 3, 2, 2 } respectively, reflecting how a flatter prior spreads the loss budget across more frequent tasks. Results are reported in Figure 19. We see a precise kink similar to Fig. 3 when rare-task signal drops to zero once δ F exceeds the analytic threshold δ ∗ F (N crit r), and rises steeply to near-unity once δ F falls below it. We also see the threshold itself shifts left as β grows: a steeper prior makes the rare task’s leading utility π r λ r much smaller, so a smaller residual is required to “free up” encoder directions that can then capture the rare task.

δ F (U) = P

E.3 Per-gap dynamics: Reproducing retention results across different injection gaps and widths

We reproduce results from Fig. 4 a by reporting the joint dynamics of the normalized rare-task signals̃ r (U t) and its gain / decay dynamics as a function of rare-task injection events, as shown in Fig. 20.

Similar to results seen in the main paper, we find larger models retain and preserve the learned signal, while smaller models require the gaps to be sufficiently small if learning is to occur at all.

E.4 Effects of Scaling Data: Learning Bottleneck Persists at Long Training Horizon

In the main paper, especially Sec. 2, we distinguish between finite vs. asymptotic training. However, most of our training runs use a budget of 100 K training iterations. To contextualize that this budget is sufficient for the claims made in the paper, we extend training runs to 1 M steps for 6 values of model widths N = { 8, 16, 32, 64, 128, 256 }, subsampling the range of widths analyzed in the main paper; the setup remains the same otherwise as Fig. 2. We find that results (see Fig. 21) are stable at much longer horizons: above-capacity tasks do not slowly close the gap given more training; instead they remain at or below the random-projection baseline indefinitely.

#F Further Experimental Results in OLMo Setting

Given the same language modeling loss, larger models can achieve lower task-specific loss. We analyze the relationship between learning “frequent tasks” and learning the injected task. For evaluating “frequent tasks”, we measure the language modeling loss on the C4 validation set using a context window size of 256. As shown in Fig. 22, when the task frequency is relatively high, i.e., from 2. 4 × 10 − 6 to 2. 4 × 10 − 5, loss curves from all model sizes follow roughly the same trajectory. This suggests that in this frequency range, model size only improves sample efficiency, but these models still have similar training dynamics [140]. However, when the task frequency gets lower, i.e., 2. 4 × 10 − 7, larger models achieve lower task loss given the same language modeling loss. Moreover, smaller models diverge from larger models one by one, with their injected-task loss plateauing at different values. This supports our hypothesis that for rare tasks, larger models have different learning dynamics that unlock the ability to learn rare tasks.

In Fig. 5, we compare models trained on the same amount of data. In Fig. 23, we further show that larger models are more compute-efficient at learning low-frequency tasks. When the frequency is one task instance per batch, i.e., 2. 4 × 10 − 7, given the same compute budget, estimated as 6 × the number of model parameters × the number of training tokens following Chinchilla scaling laws, larger models achieve lower task loss. Moreover, consistent with our observation in Fig. 22, smaller models initially follow the learning dynamics of larger models, but after a certain point their injected-task loss curves plateau and deviate from the larger models’ curves.