Bridging Compute- and Data-Optimal Pretraining

Tian Qin$^{*}$
Harvard University

Kimia Hamidieh$^{*}$
MIT CSAIL

David Alvarez-Melis
Harvard University

Abstract

Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering the regime where compute is growing faster than high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, in which data scales freely with compute, and data-optimal scaling, in which the corpus is fixed and compute can grow unbounded. CD scaling extends classic scaling by introducing a token effectiveness function $\eta$ that quantifies how much a derived token, produced for instance by multi-epoch repetition or paraphrasing, is worth relative to a fresh one, ranging from a perfect substitute to no value at all. Fitting $\eta$ for two data expansion strategies (multi-epoch repetition, paraphrasing) across model sizes ranging from 14M to 600M parameters on the Dolma corpus, we find that it is far from constant: it depends jointly on model size, the tokens-per-parameter ratio, and the amount of derived data, and saturates as the corpus is expanded. The functional form of the token effectivness function $\eta$ implies that substituting compute for data diminishes with both model size and data availability, and partitions training into three operational regimes: compute-bound, data-bound, and model-bound, showing that classic compute-optimal allocation is suboptimal across most of the practically relevant regime.

$^{*}$ Equal contributions. Correspondence to [email protected], [email protected]

Executive Summary: Pretraining of large language models has long relied on classical scaling laws that assume fresh data can scale freely with compute. In practice, high-quality data is now finite and expensive while compute continues to grow rapidly, forcing practitioners to train on repeated or paraphrased tokens derived from a fixed corpus. This mismatch makes traditional compute-optimal rules, such as Chinchilla, increasingly unreliable for deciding how much additional training is worthwhile.

The document develops a unified Compute-Data (CD) scaling framework that bridges the two extremes of unbounded data and unbounded compute. It introduces a token-effectiveness function η that measures how much value a derived token provides relative to a fresh one. The authors fit this function empirically by training models from 14 million to 600 million parameters on the Dolma corpus, testing both multi-epoch repetition and paraphrasing across a wide range of data budgets and expansion ratios.

The analysis shows that η is not constant. It declines with larger models, higher tokens-per-parameter ratios, and greater amounts of derived data, and it saturates at a finite ceiling that itself shrinks with scale. These patterns define three distinct operating regimes—compute-bound, data-bound, and model-bound—and yield a joint compute-data Pareto frontier that generalizes the classic Chinchilla allocation. They also indicate that paraphrasing outperforms repetition for models below roughly 600 million parameters and sub-Chinchilla data budgets, while repetition becomes preferable at larger scales; recommended epoch counts fall as both model size and data availability increase.

These results matter because they give practitioners a concrete way to decide whether extra compute on derived data, additional fresh data, or a larger model will deliver the greatest loss reduction. The framework recovers prior scaling laws as special cases and extrapolates reliably to held-out model sizes an order of magnitude larger than those used in fitting.

Practitioners should therefore replace single-resource Chinchilla calculations with CD scaling when planning runs, using the predicted saturation ceilings to set epoch or paraphrasing budgets and to choose the expansion method. Further work is needed to test the laws at multi-billion-parameter scale, to incorporate additional strategies such as distillation, and to predict η directly from corpus statistics rather than fitting it after the fact. The main limitations are that the functional form remains sensitive to the quality of the one-epoch baseline and that all experiments were conducted at modest model sizes with only two expansion methods.

1. Introduction

Section Summary: Classical scaling laws for training large AI models assume unlimited fresh data and focus on balancing model size against compute, but this no longer holds as high-quality data becomes scarce and expensive. The paper therefore introduces compute-data scaling laws centered on an effectiveness factor that measures how much value derived tokens—produced by repetition, paraphrasing, or similar methods—provide compared with new data. This unified description bridges the usual compute-optimal regime and the data-limited limit, revealing how performance gains taper off and offering practical guidance on resource allocation.

$**Figure 1:** **CD-scaling law overview.** *(a)* For a model of size $N$, classical scaling laws assume 1-epoch training on fresh data, which coincides with the compute-optimal Pareto frontier. Spending additional compute on *derived* tokens $D'$ produced from $D$, via data expansion strategies such as multi-epoch repetition, paraphrasing, or distillation, departs from this frontier and reaches lower loss without curating more fresh data. *(b)* To describe training behavior beyond the compute-optimal frontier, CD-scaling laws introduce a token effectiveness function $\eta$ that maps each derived token to a fresh-data equivalent. Under this mapping, runs beyond the compute-optimal frontier collapse onto the classical scaling line. *(c)* Both the saturation level of effective tokens and the rate of approach depend on $N$ and $\mathrm{TPP}$ : as the model grows and data becomes more abundant, additional compute brings diminishing returns.$

The classical scaling laws ([1, 2]) characterize how pretraining loss decreases with model size $N$ and dataset size $D$, prescribing a compute-optimal allocation between the two. Implicit in this formulation is a data-abundance assumption: fresh tokens are freely available, and the pretraining corpus $D$ scales naturally with compute. But this assumption is becoming inaccurate. Compute continues to grow at exponential rates ([3]), while high-quality pretraining data remains finite and costly to curate ([4]). Pretraining is therefore entering a regime that classical scaling laws were not designed for: one in which $D$ is bounded and additional compute must be spent on derived tokens $D'$ produced from $D$ via multi-epoch repetition, paraphrasing, or distillation. The central question becomes:

For a model of size $N$ trained on a fixed corpus $D$, how much can additional compute reduce loss?

We approach this question through the lens of two limiting regimes. Compute-optimal scaling $\mathcal{L}^{\mathrm{Chin}}$ describes the setting where data is unbounded and compute is the bottleneck, and it is the regime characterized by classical scaling laws. Data-optimal scaling $\mathcal{L}^D$ describes the opposite limit, where $D$ is fixed and compute is unbounded and it is the asymptote of what additional training, in any form, can achieve on a given corpus $D$. Modern pretraining sits between these two limits (Figure 1a).

We propose Compute-Data (CD) scaling laws, a unified framework that bridges the compute- and data-optimal scaling. At its center is an effectiveness function $\eta^{\mathrm{strat}}\in [0, 1]$ that quantifies, for a given data expansion strategy such as paraphrasing or repetition, how much a derived token $D'$ is worth relative to a fresh one (Figure 1b). The resulting scaling law is

$ \mathcal{L}(N, D, D') = E + \frac{A}{N^\alpha} + \frac{B}{(D + \eta \cdot D')^\beta}, $

with $\eta = 1$ meaning derived tokens are as informative as fresh data and $\eta = 0$ meaning they provide no benefit. The law continuously interpolates between the two regimes: when $D' = 0$ it recovers compute-optimal scaling, and as $D' \to \infty$ it approaches the data-optimal limit, predicting the best achievable loss for a given $(N, D)$.

We focus on two data-expansion strategies, multi-epoch repetition and paraphrasing. To fit CD-scaling laws, we sweep model sizes $N$ from 14M to 600M parameters, fresh-data sizes $D$ from 30M to 30B tokens, and derived-token budgets $D'$ from 30M to 120B tokens, with hyperparameter search over learning rate and weight decay. We find that the effective-token count $D_{\mathrm{eff}} = D + \eta, D'$ saturates as $D' \to \infty$, with both the saturation level and the rate of saturation depending on $N$ and the tokens-per-parameter ratio $\mathrm{TPP} = D/N$ (Figure 1c). This observation allows us to determine the functional form of the effectiveness coefficient $\eta$, which has a power-law decay over $N$, $D$ and $D'$. The data-constrained scaling law of [5] is recovered as the special case in which the saturation level is constant in $(N, D)$.

The fitted $\eta^{\mathrm{strat}}$ yields three practical implications. First, between the compute-optimal scaling $\mathcal{L}^{\mathrm{Chin}}$ and data-optimal scaling $\mathcal{L}^D$, the proposed CD scaling law identifies three regimes: compute-bound, data-bound, and model-bound. This informs a practitioner which resource is currently limiting their run. Second, for a given $(N, D)$, it prescribes the optimal compute to spend on $D'$, and generalizes Chinchilla's compute-only optimization into a joint compute–data Pareto frontier. Third, CD-scaling prescribes which expansion strategy to use and how much additional data to train on. Concretely, extrapolated CD-scaling law shows that the 4-epoch rule holds only for medium-scale models ($\sim 3$ B) near $1\times$ Chinchilla, with recommended epochs decreasing as $N$ and $\mathrm{TPP}$ grow. Paraphrasing is preferred for small models ($\lesssim 600$ M) and sub-Chinchilla data budgets, but is ineffective for $N \geq 7$ B or large budgets ($\geq 4\times$ Chinchilla).

In summary, our contributions are:

A unified CD-scaling framework. We propose CD-scaling laws, which bridge the compute-optimal and data-optimal regimes through an effectiveness function $\eta^{\mathrm{strat}}$.
Effectiveness of derived tokens diminishes with scale. Fitting $\eta$ on Dolma-3 across model sizes from 14M to 600M, we find that the effectiveness of spending additional compute beyond 1-epoch training decays with both $N$ and $\mathrm{TPP}$.
Three practical implications. (a) The CD law identifies three regimes—compute-, data-, and model-bound—that inform which resource is currently limiting the pretraining run. (b) For a given $(N, D)$, it prescribes the optimal compute–data allocation, generalizing Chinchilla's compute-only optimization into a joint compute–data Pareto frontier. (c) It prescribes when to switch from repetition to paraphrasing, and provides recommendation on optimal epochs or paraphrasing based on training configurations.

2. Related Work

Section Summary: Researchers have extended classical scaling laws for large AI models by examining how performance changes when training data is reused multiple times, when synthetic data is created to supplement limited real data, or when distillation transfers knowledge from one model to another under fixed data budgets. Studies on synthetic data show that techniques like rephrasing web text or generating targeted examples can speed up training, while distillation approaches treat extra model computations as a way to extract more value from the same corpus. Separate work optimizes how to blend different types of original data sources, whereas this paper focuses instead on deciding how much compute to allocate between fresh tokens and these derived ones.

Scaling laws.

The classical scaling laws of [2] and [1] characterize loss as a function of model size $N$ and fresh pretraining tokens $D$, prescribing a compute-optimal allocation under the assumption that $D$ scales freely with compute. Subsequent work has investigated each side of this assumption. [6] show that the Chinchilla recipe extrapolates reliably into the substantially over-trained regime ($\mathrm{TPP} \gg 20$). [5] extend Chinchilla to the data-constrained regime, fitting a single saturation budget for repeated tokens. [7] provide a complementary theoretical perspective, proving in linear regression that the effective reuse rate of repeated data grows with model size $N$, with $\Theta(\log N)$ scaling under strong convexity and power-law scaling under Zipf-distributed features. [8] take the opposite limit, asking what loss is achievable under fixed $D$ and unbounded compute. Concurrent to our work, [9] also extend Chinchilla to repeated data, but model repetition through an additive, model-size-dependent overfitting penalty that captures the regime where loss rises with further epochs, rather than through an effectiveness function on derived tokens. Our work unifies these threads by recovering [1] as the $D' = 0$ limit, the law of [5] as the constant-saturation special case, and the data-optimal limit of [8] as $D' \to \infty$.

Synthetic data for pretraining.

A growing body of work treats synthetic data as a lever for trading compute against fresh-token scarcity. [10] show that rephrasing web documents into structured styles accelerates pretraining by ${\sim}3\times$, and [11] extend the recipe to trillion-token regimes. [12] conduct a controlled study across synthetic-data types, finding that mixing ${\sim}30%$ rephrased data yields $5$ – $10\times$ training speedups. [13] dynamically target synthetic data at the model's current knowledge gaps; [14] learn inter-document relations to generate diverse synthetic continuations. Each of these works characterizes a single strategy in isolation. We instead fit a common functional form for the token-effectiveness function $\eta$ across multi-epoch repetition and paraphrasing, and treat each strategy as a parameterization of the same underlying law.

Distillation and other compute-for-data exchanges.

Distillation ([15]) has seen renewed interest in LLM pretraining as a way to extract additional signal from a fixed corpus. [8] show that single-student distillation from a multi-epoch teacher recovers most of the loss improvement of the ensemble. [16] establish a teacher–student scaling law that allocates compute optimally between training the teacher and the student. These works model distillation as an internal allocation of compute under a fixed corpus; in our framework, self-distilled tokens are a natural instance of $D'$ that the same $\eta$ function describes, although our empirical fits in this paper focus on multi-epoch repetition and paraphrasing.

Data allocation and mixing.

A complementary line of work allocates compute across the composition of the fresh corpus rather than between fresh and derived tokens: DoReMi ([17]), RegMix ([18]), and the data-mixing scaling laws of [19] optimize mixture weights across pretraining domains, and mixture dependent scaling laws of [20] predict model performance by accounting for data synergy. CD-scaling is orthogonal: we hold the fresh-corpus composition fixed and study how compute should be split between fresh $D$ and derived $D'$.

3. Methodology and Experiment Setup

Section Summary: The authors extend the standard Chinchilla scaling law into a compute-data version that incorporates both fresh training tokens and additional derived tokens from synthetic or repeated data, using an effectiveness factor to measure how much each derived token contributes relative to a new one. They then test this law through a series of pretraining runs on models ranging from 14 million to 600 million parameters, using subsets of a large public corpus for the fresh data and generating extra tokens either by repeating the corpus over multiple epochs or by paraphrasing documents with a separate instruction-tuned model. All runs follow a consistent training setup with fixed sequence length and batch size, tuned hyperparameters, and evaluation on held-out validation data, while excluding the cost of paraphrase generation from the reported compute.

3.1 Compute-Data Scaling Law

The classical Chinchilla scaling law is

$ \mathcal{L}^{\mathrm{Chin}}(N, D) ;=; E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},\tag{1} $

where $N$ is the number of model parameters and $D$ is the number of fresh pretraining tokens, each seen once. To accommodate synthetic and augmented data, we extend Equation 1 into a compute–data (CD) scaling law that shares the constants $(E, A, B, \alpha, \beta)$:

$ \mathcal{L}^{\mathrm{CD}}(N, D, D') = E + \frac{A}{N^\alpha}

\frac{B}{\left(D + \eta^{\mathrm{strat}}!\left(N, \mathrm{TPP}, D'/D\right) \cdot D'\right)^\beta},\tag{2} $

where $D'$ denotes additional tokens produced from a data-expansion strategy (multi-epoch repetition, paraphrasing, or other forms of synthetic data generation and augmentation methods). The effectiveness coefficient $\eta \in [0, 1]$ predicts how much one derived token in $D'$ is worth relative to a fresh token from $D$. In Section 4.1, we motivate parameterizing $\eta$ in terms of the three more interpretable quantities $(N, \mathrm{TPP}, D'/D)$ rather than $(N, D, D')$, and in Section 4.2, we derive a parametric form for $\eta^{\mathrm{strat}}$.

Equation 2 has three natural limits. Setting $D' = 0$ recovers the classic scaling law $\mathcal{L}^{\mathrm{Chin}}$. Taking $D \to \infty$ sends the third term to zero, leaving the model-bound floor $\mathcal{L}^{N} = E + A/N^\alpha$, the lowest loss a model of size $N$ can reach under unlimited fresh data. Finally, and most importantly, taking $D' \to \infty$ at fixed $D$ defines data-optimal scaling $\mathcal{L}^{D}$: the lowest loss attainable on a fixed corpus $D$ when unlimited compute is spent on derived tokens. The exact form of $\mathcal{L}^{D}$ depends on $\eta$ and is derived in Section 4.2.

3.2 Experiment Setup

Data, model, and training.

We use the OLMo3 ([21]) architecture and training infrastructure for all pretraining runs. We train models ranging from $N = 14$ M to $N = 600$ M parameters, with fresh data $D$ ranging from $30$ M to $30$ B tokens and derived tokens $D'$ from $30$ M to $120$ B. Fresh tokens are sampled from the Dolma-3 150B corpus ([21])^1, with smaller $D$ always a strict subset of larger $D$. All models are trained at sequence length 4096, batch size 512, with AdamW ([22]) and cosine learning-rate decay. We sweep learning rate $\in {1e-4, 3e-4, 1e-3, 3e-3}$ and weight decay $\in {0.1, 0.2, 0.4, 0.8, 1.6}$ at $N \in {30\text{M}, 370\text{M}}$, selecting the configuration that minimizes validation loss; for other model sizes we perform a local grid search around the selected configuration. Our primary metric is validation loss on a held-out set of 5M Dolma-3 documents.

Data expansion strategies.

We extend pretraining compute beyond 1-epoch training on $D$ via two data-expansion strategies, each producing additional tokens $D'$:

Multi-epoch ([5]): repeat the original corpus for additional epochs.
Paraphrasing (i.e., augmentation) ([10, 12]): rephrase each document in $D$ using SmolLM2-1.7B-Instruct as the paraphrasing model ([23]), sampling up to 16 paraphrases per document with the style for each seed drawn uniformly from question, math (or wiki when not applicable), FAQ, table, a four-style mixture identified as most effective in prior analysis ([24]). Models are then trained on a mixture of fresh $D$ and paraphrased $D'$ for one epoch. Paraphrased documents are always derived from documents in $D$, and smaller paraphrased corpora are nested in larger ones.

We do not include the FLOPs spent on paraphrase generation in training compute, as paraphrasing can be performed asynchronously and is not part of the training loop.

4. Parametric Form of CD-Scaling Law

Section Summary: The section outlines an empirical study of the effectiveness coefficient η, which quantifies how much value repeated or paraphrased data adds during training, and uses those measurements to derive a compact mathematical formula. Researchers first recast η in terms of intuitive ratios such as tokens per parameter and expansion factor, observe that its benefits plateau rather than grow indefinitely, and then propose an exponential saturation form whose ceiling itself follows a power law in model size and data ratio. Substituting this expression back into the original scaling law produces a closed-form “CD-scaling law” with eight fitted constants that predicts loss for any mixture of fresh and derived data.

We first characterize $\eta$ empirically (Section 4.1), then we use those observations to motivate a functional form (Section 4.2), and fit the resulting closed-form (Section 4.3). We validate that the fit extrapolates to held-out model sizes and show ablations on the parametric form choice (Section 4.4). Finally, we confirm that validation loss improvements transfer to downstream benchmark performance (Section 4.5) across training settings.

$**Figure 2:** **Empirical characterization of $\eta$** ($y$-axis is $\eta_{\mathrm{emp}}$ throughout). *(a)* $\eta$ vs. $\mathrm{TPP}$ at fixed $D'/D$: $\eta$ decreases monotonically with $\mathrm{TPP}$, so the portion of fresh data in the pretraining set matters. *(b)* $\eta$ vs. $D'/D$ at fixed $(N, D)$: monotone decay; together with (c), this is decay *toward a finite saturation ceiling*, not decay to zero. *(c)* $\eta \cdot D'/D$ vs. $r$ at fixed $(N, D)$: extra effective tokens *plateau* as $D'$ grows. The plateau is a fundamental property of $\eta$ (and does not arise from under-fitting): no matter how many derived tokens we add, the loss curve flattens, and any functional form for which $\eta \cdot D'$ diverges with $D'$ is ruled out. We also observe that larger models saturate faster, consistent with findings in Section 5.1.$

4.1 Empirical Characterization of $\eta$

Before specifying a parametric form for $\eta^{\mathrm{strat}}$, we measure it empirically and examine how it varies with $(N, D, D')$. To obtain $\eta_{\mathrm{emp}}$, we first fit the classic scaling law Equation (1) to obtain constants $(E, A, B, \alpha, \beta)$. For each repetition and paraphrasing run, we then define the empirical coefficient $\eta_{\mathrm{emp}}$ as the unique value that, substituted into Equation 2 alongside these constants, reproduces the observed loss $\mathcal{L}$. We expect $\eta_{\mathrm{emp}} \in [0, 1]$, though it can exceed $1$ when the 1-epoch fit underestimates a run's loss, as any such residual is absorbed into $\eta_{\mathrm{emp}}$. We discuss this limitation in Section 6.

Change of variables.

Although $\eta^{\mathrm{strat}}$ is naturally a function of $(N, D, D')$, we recast it in terms of three more interpretable variables: the tokens-per-parameter ratio $\mathrm{TPP} = D/N$, the expansion ratio $r = D'/D$, and the model size $N$. $\mathrm{TPP}$ measures fresh-data availability relative to model capacity, and is the conventional unit in which Chinchilla scaling is described. The expansion ratio $r$ measures how aggressively $D'$ expands beyond $D$. In the multi-epoch case, $r$ is simply the number of extra epochs. We retain $N$ as a separate variable so that $\mathrm{TPP}$ captures data availability and $N$ captures model size alone.

Empirical observations.

In Figure 2, we examine how $\eta_{\mathrm{emp}}$ depends on $(r, \mathrm{TPP}, N)$. Panel (a) shows that $\eta_{\mathrm{emp}}$ decays with $\mathrm{TPP}$, and the decay steepens as $r$ grows. Panel (b) shows that $\eta_{\mathrm{emp}}$ decays with $r$, and the decay rate depends on $\mathrm{TPP}$: slowly when $\mathrm{TPP}$ is small, sharply when $\mathrm{TPP}$ is large. The quantity that matters for total loss, however, is not $\eta$ but the effective derived data $\eta \cdot D'$, or $\eta \cdot r$ once normalized by $D$. This quantity represents the fresh-equivalent tokens the derived corpus $D'$ contributes. Rather than growing without bound, $\lim_{r \to \infty} \eta \cdot r$ should saturate. We visualize the saturation behavior in panel (c). Both the saturation limit and the rate of saturation depend on $N$, with larger models saturating at a lower value and at a faster rate.

4.2 Functional Form of $\eta$

The empirical observations impose three requirements: (i) $\eta_{\mathrm{emp}} \to 1$ as $r \to 0$, (ii) $\eta \cdot r$ saturates to a finite limit as $r \to \infty$, and (iii) the saturation behavior depends on $\mathrm{TPP}$ and $N$. Among the functional families satisfying the first two conditions, we adopt the exponential form below, and Section 4.4 shows it provides the best fit when compared to all other forms considered:

$ \eta = \frac{R^{}}{r}\left(1 - e^{-r/R^{}}\right)\tag{3} $

Interpretation of $R^{*}$.

The exponential form gives the identity $\lim_{r \to \infty} \eta \cdot r = R^{}$, so $R^{}$ can be interpreted as the saturation ceiling: the maximum number of fresh-equivalent tokens a corpus of size $D$ can yield through data expansion, expressed as a multiple of $D$. This ceiling implies that total effective data saturates at $\lim_{D' \to \infty} D_{\mathrm{eff}} = D(1 + R^{*})$. Applying this identity to 2 yields a closed form for data-optimal scaling $\mathcal{L}^{D}$, the lowest loss a given data-expansion strategy can reach on a fixed corpus of size $D$:

$ \mathcal{L}^{D}(N, D) ;=; E + \frac{A}{N^\alpha}

\frac{B}{\left[D, \left(1 + R^{*}\left(D, N\right)\right)\right]^\beta}.\tag{4} $

Functional form of $R^{*}$.

Based on observations in Section 4.1, we propose a power law for $R^{*}$:

$ R^{*}(D, N) ;=; K \cdot (D/N)^{\rho} \cdot N^{\sigma},\tag{5} $

where $\rho$ governs how $R^{}$ tightens with $\mathrm{TPP}$ and $\sigma$ how $R^{}$ tightens with $N$. We overlay the proposed form in Figure 2, and confirm that the power law fits the observed qualitative behavior.

Together, Equation 3 and 5 determine the token effectiveness function $\eta$. Substituting $\eta$ into Equation 2 gives the complete CD-scaling law, containing eight constants: the five 1-epoch Chinchilla parameters $(E, A, B, \alpha, \beta)$ and three strategy-specific parameters $(K, \rho, \sigma)$. The repeated-data law of [5] is the special case in which $R^{}$ is constant and independent of $(D, N)$. Ablations in Section 4.4 (Table 3) confirm that this constant- $R^{}$ assumption fits worse, and that the $(D, N)$-dependence in Equation 5 is what makes the law predictive across model sizes and data-availability regimes.

4.3 Fitting CD-Scaling Law

::: {caption="Table 1: Fitted CD-Scaling parameters with 95% CIs. Top: 1-epoch Chinchilla parameters, Bottom: per-strategy CD-Scaling parameters. RMSE on log L, reported separately for 1-epoch, repetition and paraphrasing training runs."}

:::

$**Figure 3:** **Fitted CD-scaling law across model sizes and strategies.** Data-reducible loss $\mathcal{L} - (E + A/N^{\alpha})$ across model sizes and training settings ($1$-epoch, repetition, and paraphrase). The solid line is the classic scaling-law fit on 1-epoch runs. *Left:* Data-reducible loss against *fresh* tokens D. Repetition and paraphrase runs train on extra $D'$ tokens, pushing their loss below what fresh D alone achieves, so these runs fall beneath the classic scaling law. *Right:* The CD-scaling law converts each $D'$ into its fresh-token equivalent $\eta_{\mathrm{strat}}D'$. Re-plotted against the effective token count $D_{\text{eff}} = D + \eta_{\mathrm{strat}}D'$, the paraphrase and repetition runs reduce back to the classic scaling law.$

$**Figure 4:** **Saturation ceiling $R^*$ across strategies.** Lines show fitted $R^*$, shaded bands show $95\%$ CIs, and the grey region marks where the two CIs overlap. *(a)* $R^*$ vs. N at $\mathrm{TPP}=20$. Paraphrasing significantly wins for small models ($N\lesssim50$ M) and repetition for large ones ($N\gtrsim190$ M). *(b)* $R^*$ vs. TPP for $N=600$ M. The CIs overlap at small TPP, so paraphrasing does not meaningfully beat repetition. Beyond $\mathrm{TPP}\approx8$, repetition significantly wins.$

We follow the fitting procedure introduced by [25]. Namely, we use a Huber loss on the residuals of $\log \mathcal{L}$. We jointly optimize all parameters: the $5$ parameters of the 1-epoch scaling law $(E, A, B, \alpha, \beta)$, together with $3$ per-strategy saturation parameters $(K, \rho, \sigma)$ for repetition and paraphrasing. We provide the detailed fitting procedure in Appendix A and report the fitted parameters in Table 1.

To assess fit quality, we separately report the RMSE of $\log \mathcal{L}$ for 1-epoch, repetition, and paraphrasing runs. We also use 95% confidence intervals (CIs) to quantify uncertainty in the fitted parameters. Specifically, we resample training runs with replacement and refit the parameters on the resampled data. We repeat this resampling and refitting procedure $200$ times and take the $95%$ confidence interval. We report RMSE and per-parameter confidence intervals in Table 1 as well.

Fitting the CD-scaling law provides a principled way to compare data-expansion strategies. As an example, we compare saturation ceilings $R^*$ (Section 4.2). Because we fit CIs on $R^*$, we can go beyond point estimates and ask whether the difference between strategies is statistically significant. In each panel of Figure 4, the shaded band marks where the two $95%$ CIs overlap. At the Chinchilla-optimal budget ($\mathrm{TPP} = 20$), Figure 4 (a) compares $R^*$ across model sizes: the repetition and paraphrasing CIs overlap in a band around the crossover ($N \approx 94$ M), but paraphrasing significantly wins for small models ($N \lesssim 50$ M) and repetition for large ones ($N \gtrsim 190$ M). At a fixed size ($N = 600$ M), Figure 4 (b) compares $R^*$ across TPP: the CIs overlap at small TPP, so paraphrasing does not meaningfully beat repetition, whereas beyond $\mathrm{TPP} \approx 8$ repetition wins. We map the full crossover boundary across $(N, \mathrm{TPP})$ in Section 5.2.

4.4 Cross-Scale Validation and Ablations

Cross-scale validation.

A practical motivation for fitting Equation 3 is to predict loss for the expensive large-model runs from cheaper small-model runs. We now examine whether the CD-scaling law can extrapolate beyond the model sizes used in fitting. To test this, we refit Equation 3 on runs at $N \in {14, 30}$ M only, then predict held-out losses at $N \in {60, 100, 190, 370, 600}$ M. The held-out set spans model sizes $2\times$ to $20\times$ larger than any used in fitting, TPP up to ${\sim}160$ ($8\times$ Chinchilla), and $r$ up to $63$ (64 epochs), which covers token-count regimes well outside the fitting range. Additionally, we sweep on model sizes used for fitting in Appendix B.1.

The small- $N$ fit achieves RMSE $0.079$ on $\log \mathcal{L}$ on the held-out points, against an in-sample RMSE of $0.048$. Our scaling law therefore transfers cleanly across an order of magnitude in model size, and a practitioner can fit Equation 3 on small models with modest budget and read off $R^{*}(D, N)$ at production scale to anticipate how much additional data will reduce loss.

Ablations.

We validate the exponential functional form Equation 3 and the $R^*$ form Equation 5 using multi-epoch training runs, reporting leave-one-out (LOO) RMSE on $\log \mathcal{L}$ under the same fitting procedure.

For the functional form of $\eta$, we compare nine candidates (Appendix A.3, Table 3). The forms that let $R^{}$ depend on $(D, N)$ via Equation 5 dominate the rest at every parameter count; in particular, the constant- $R^{}$ data-repetition baseline ([5]) is rejected by ${\sim}40%$ in relative LOO RMSE. Within the top group, our chosen form is statistically indistinguishable from the best, and we prefer it because it generalizes the constant- $R^{*}$ form and satisfies $\eta(0) = 1$ exactly.

For $R^{}$, we remove terms from Equation 5 and refit (Appendix A.4, Table 4). Dropping either exponent degrades the fit, and dropping both recovers the constant- $R^{}$ baseline: each exponent is individually necessary, and using both is strictly better than either alone.

4.5 Loss-to-Downstream Transfer

So far we have compared validation loss across training settings. We now ask whether validation loss remains a valid predictor of downstream capability across data-expansion strategies and model scales.

We group benchmarks into LM-based and accuracy-based tasks. Benchmarks such as GSM8K ([26]), TriviaQA ([27]), and HumanEval ([28]) yield near-random accuracy at small-to-medium scale. Following [6], we recast them as language-modeling tasks and report bits-per-byte (BPB) of the gold response given the prompt, which stays predictive even when accuracy is near-trivial. For accuracy-based tasks, we report mean accuracy across LAMBADA ([29]), HellaSwag ([30]), OpenBookQA ([31]), RACE ([32]), and SQuAD ([33]). We plot both against validation BPB in Figure 5.

In both cases, runs from all strategies and scales fall onto a single curve as a function of validation BPB alone, which confirms that validation loss predicts downstream capability independent of training setting. We report per-task breakdowns in Appendix B.2.

5. CD-Scaling Laws Bridge Compute- and Data-Optimal Training

Section Summary: CD-scaling laws extend standard compute-optimal scaling by treating fresh data D and extra compute spent on repeated or paraphrased data D' as separate resources, which creates a two-dimensional loss surface instead of a single curve. This surface reveals a practical Pareto frontier for allocating compute and data at any fixed model size, and it divides training into three regimes—compute-bound, data-bound, and model-bound—showing which resource is currently the bottleneck. The same laws also predict how quickly returns from extra compute diminish as models grow or as the amount of original data per parameter increases.

In this section, we discuss three implications of CD-scaling laws. First, for a fixed N, the CD law predicts the compute-data Pareto allocation between D and $D'$ (Section 5.1). Second, it partitions training into three regimes, identifying which resource is the binding constraint for a given $(N, D, D')$ (Section 5.2). Third, it tells a practitioner which data-expansion strategy to adopt and how much compute to spend before returns saturate (Section 5.3).

5.1 Compute-Data Pareto-Optimal Allocation

A practitioner training a model of size N on a fixed corpus D must decide how much compute to spend on further loss reduction. Under Chinchilla scaling, compute is fixed by $(N, D)$ via $C \approx 6 N D$, leaving no such freedom. Decoupling C from D through multi-epoch training or paraphrasing adds a degree of freedom, giving a two-dimensional loss surface over $(C, D)$ with $C \approx 6 N(D + D')$. Figure 6 (left) shows this empirical surface for $N = 30$ M, with compute C on the $x$-axis and fresh data D on the $y$-axis. The black dashed line traces single-epoch runs, where the surface collapses onto a single ray. The contours around single-epoch training are far steeper in C than in D, indicating that modest compute beyond the single-epoch buys substantial loss reductions that single-epoch scaling leaves on the table.

The empirical Pareto frontier is non-trivial.

The teal curve traces the loss-minimizing path through $(C, D)$ space via steepest descent on the empirical surface. Along it, the contours bend sharply in both directions, meaning increasing C or D alone would leave loss roughly unchanged. Since neither resource can reduce loss in isolation, this line traces the compute-data Pareto frontier. This frontier gives practitioners a general guidance on resource allocation. In the data-scarce, low-compute regime (bottom-left), increasing compute is far more effective than adding fresh data, even when data is scarce. As C grows, the frontier rotates toward scaling C and D in equal proportion, then enters a basin where fresh data dominates and further compute yields diminishing returns.

CD-scaling predicts this frontier.

Tracing the steepest-descent path on the CD-scaling loss surface (Figure 6 right) yields a trajectory that closely tracks the empirical Pareto curve, generalizing Chinchilla's compute-only optimization into a joint compute-data optimization.

5.2 Three Training Regimes

As fresh data become the binding constraint, instead of plotting validation loss against training compute, we plot against fresh data, with color indicating the compute budget $C \approx 6 N(D + D')$, shown in Figure 7. The three limits of CD-scaling: $\mathcal{L}^{\mathrm{Chin}}$ ($D' = 0$), $\mathcal{L}^{D}$ ($D' \to \infty$), and $\mathcal{L}^{N}$ ($D \to \infty$) appear as three boundary curves that carve the ($\mathcal{L}$, D) plane into three distinct regions:

Compute-bound (between $\mathcal{L}^{\mathrm{Chin}}$ and $\mathcal{L}^{D}$): for a given D, additional $D'$ still reduces loss. Compute is the binding resource.
Data-bound (between $\mathcal{L}^{D}$ and $\mathcal{L}^{N}$): $D'$ has saturated to $R^* D$, only additional data can further reduce loss.
Model-bound (at $\mathcal{L}^{N}$): both compute and data have saturated. N is the binding resource.

We then overlay the three region boundaries predicted by CD-law against empirical training losses. Single-epoch points lie on $\mathcal{L}^{\mathrm{Chin}}$. Intermediate- $D'$ traces countour lines in the compute-bound region. Importantly, the predicted data-optimal frontier $\mathcal{L}^{D}$ coincides with the observed $D' \to \infty$ asymptote of the training runs, validating that the CD-scaling's saturation limit correctly models the empirical limit.

The compute-bound region shrinks with both TPP and N.

The vertical gap between $\mathcal{L}^{\mathrm{Chin}}$ and $\mathcal{L}^{D}$ is governed by the saturation ceiling $R^{}$. When $R^{}$ is large, the data-optimal frontier sits well below the compute-optimal one. In this training regime, additional training compute spent on $D'$ delivers substantial loss reductions. In contrast, when $R^{}$ is small, the two frontiers nearly coincide and additional compute spent on $D'$ is largely ineffective. In Section 4.1, we empirically observed that $R^$ shrinks with both N and TPP. In Figure 7, we confirm that the fitted CD-scaling law correctly models the shrinking rate over $\mathrm{TPP}$ (left), and $N$ (right).

$**Figure 7:** **The CD-scaling law captures three training regimes.** *Left*: $N=30$ M empirical losses overlaid with CD-scaling law predictions. By sweeping $D'$ from $0$ ($\mathcal{L}^{\mathrm{Chin}}$, compute-optimal limit) to $\infty$ ($\mathcal{L}^{D}$, data-optimal limit), CD-scaling law traces out all achievable losses from $\mathcal{L}^{\mathrm{Chin}}$ to $\mathcal{L}^{D}$. *Right*: The same decomposition across model sizes $N$ . As N grows, the compute-bound region bounded by $\mathcal{L}^{\mathrm{Chin}}$ and $\mathcal{L}^{D}$ tightens, indicating that for larger models, increasing compute C yields diminishing returns in further loss reduction.$

5.3 Practical Implications

$**Figure 8:** **When to repeat, when to paraphrase, and how much.** *Top:* Saturation ceiling $R^*$ across $N$ and $\mathrm{TPP}$ for repetition (*left*) and paraphrase (*right*). The boundary marks where the winning strategy switches. Paraphrase wins in the lower-left (small $N$, low data budget), repetition wins in the upper-right. *Bottom:* Training compute to reach $90\%$ of $R^*$. We provide recommended epochs (*left*) for repetition and passes $K$ (*right*) for paraphrase. Both decrease as $N$ and $\mathrm{TPP}$ grow.$

By visualizing saturation ceiling $R^*$ predicted by CD-scaling laws, we can directly understand at different $N$ and $\mathrm{TPP}$, which of the two data expansion strategy to adopt and the best achievable outcome, as shown in Figure 8 (top). Figure 8 (bottom) then reports the training effort needed to reach $90%$ of such $R^*$ 's. Namely, we report the total epochs for repetition runs and total paraphrasing passes for paraphrase runs. We provide three takeaways:

The recommended epoch count decreases as model size and data budget grow. Extrapolating the fitted law to large model scales indicates that 4-epoch training is recommended only for medium-scale models ($\sim 3$ B) around $1\times$ Chinchilla data scale.
For smaller models and data budgets, paraphrasing is preferred. Concretely, paraphrasing beats repetition for models $\lesssim 600$ M and small data budgets (
lt; 1\times$ Chinchilla).
Extrapolating the fitted law to large model scales indicates that paraphrasing becomes ineffective for $N\geq 7$ B and large data budgets ($\geq 4\times$ Chinchilla).

6. Conclusion & Discussion

Section Summary: The researchers present a unified framework that links compute-optimal and data-optimal training through an effectiveness measure, showing how extra computation can reduce loss on a fixed dataset and guiding balanced choices between adding data, compute, or model capacity. This yields practical rules for spotting the main bottleneck in a project and deciding when to stop scaling one resource in favor of another. They also outline next steps such as predicting effectiveness from simple dataset properties, while noting limitations including heavy reliance on initial test runs, high computational costs, and testing only a narrow set of strategies on modest model sizes.

We establish a unified framework that bridges compute- and data-optimal training. Through the effectiveness function $\eta^{\mathrm{strat}}$, we characterize how much additional compute can reduce loss on a fixed pretraining corpus D. Building on it, we derive three practical implications: the compute-data Pareto-optimal allocation between D and $D'$, the three-regime picture (compute-bound, data-bound, and model-bound) that identifies the binding constraint, and guidance on which expansion strategy to adopt and how much compute to spend before returns saturate. Together these give practitioners a principled basis for deciding when to invest in compute, fresh data, or model capacity.

Predicting $\eta$ from corpus statistics.

We currently fit $\eta^{\mathrm{strat}}$ empirically per strategy. A natural extension is to predict $\eta$ from dataset statistics alone, using summary statistics such as token diversity, $n$-gram overlap, or perplexity distributions. Such a predictive theory would let practitioners assess an expansion strategy on a given corpus before committing compute, making the framework substantially more practical.

Limitations.

The functional form we adopt has a clean interpretation, but $\eta$ absorbs any residual misspecification of the 1-epoch fit (and can push $\eta_{\mathrm{emp}}$ above 1), so the overall fit is sensitive to the quality of the 1-epoch runs. Within each run, we swept only learning rate and weight decay. Other hyperparameters (batch size, sequence length, schedule) were held fixed and could in principle interact with $\eta$. Establishing these laws incurs substantiation computation cost. To complete training runs reported in this study, we have consumed over 250k H100-hours. Even with this compute budget, we could only fit scaling laws for two data expansion strategy on small-to-medium scale model sizes. Finally, we evaluate two data-expansion strategies (repetition and paraphrase); others such as self-distillation or synthetic structured data ([34]) remain unexplored, as do combinations of strategies that may yield higher effective $\eta$ than any single one.

Acknowledgments

Section Summary: The acknowledgments section notes that DAM received financial and institutional support from several sources, including the Kempner Institute, the FAS Dean's Competitive Fund for Promising Scholarship, the Aramont Fellowship Fund, and the NSF AI-SDM Institute under a specific grant. It also expresses thanks to Antonio Torralba for providing helpful discussions and guidance on the project. These contributions helped make the work possible.

DAM acknowledges support from the Kempner Institute, FAS Dean's Competitive Fund for Promising Scholarship, Aramont Fellowship Fund, and the NSF AI-SDM Institute (Grant No. IIS-2229881). Additionally, we thank Antonio Torralba for helpful discussions and guidance on this project.

Appendix

Section Summary: This appendix details the numerical procedures used to fit the CD-scaling law parameters to the combined single-epoch, repetition, and paraphrase data. It describes a stable log-space loss formulation, joint optimization of all parameters with L-BFGS, and an iterative trimming process that removes about 4% of noisy small-scale points. Additional ablations and cross-validation tests confirm that the chosen functional form for repetition effects is both necessary and reliable for predicting performance across model sizes.

A. Implementation Details and Ablations for the CD-Scaling Fit

This appendix supplies the implementation details, the outlier-trimming sweep, the functional-form ranking, and the per-term ablation of $R^{}$ referenced from Section 4.3 and Section 4.4. The main text reports the canonical $k = 15$ row and the chosen exp-sat $R^{}(D, N)$ form; the tables below show every cell of both sweeps so the choices can be audited.

A.1 Implementation details

Log-space objective.

We work in log-loss space using the numerically stable LSE-equivalent reformulation of Equation 2 ([1]):

$ \log \mathcal{L}(N, D, D') ;=; \mathrm{LSE}, !\bigl(e, ; a - \alpha \log N, ; b - \beta \log(D + \eta \cdot D')\bigr),\tag{6} $

algebraically identical to 2 with $(E, A, B) = (e^{e}, e^{a}, e^{b})$ but numerically stable. The fit minimises Huber loss on log-residuals with $\delta = 0.1$, which gives quadratic weight to typical residuals while suppressing the influence of stragglers an order of magnitude larger.

Optimiser and initialisation.

L-BFGS with strong-Wolfe line search. We initialize from a log-spaced grid over the parameters, find the best in-sample seed, and report the optimum. Across $30+$ random seeds the optimum is reproducible to four significant figures.

Joint fit.

The headline pipeline (Section 4.3) fits all eleven parameters $(E, A, B, \alpha, \beta)$ + $(\log K, \rho, \sigma){\text{rep}}$ + $(\log K, \rho, \sigma){\text{para}}$ jointly on the pooled 1-epoch + repetition + paraphrase data via a single Huber LSE with iterative residual trimming. The initialization grid brackets both signs of $\rho_{\text{para}}$ and $\sigma_{\text{para}}$ so the optimizer can land on either sign. Across every grid we tried, the optimum lands in the same negative- $(\rho_{\text{para}}, \sigma_{\text{para}})$ basin.

Iterative residual trimming.

A handful of noisy small-scale points can dominate the fit, so we drop them using an iterative variant of the residual-based trim of [25]: fit, drop the single worst residual, refit, drop the next worst, and so on for $k$ steps. Applied to the untrimmed fit, the standard Hampel outlier rule ([35]) identifies 14 points as outliers. We round to $k = 15$ for consistency with our sweep grid (Table 2), which trims ${\sim}4%$ of the data. The trimmed points are all small-scale $1$-epoch and repetition runs at 14M and 30M, the same class of outliers identified by [25] for Chinchilla itself.

A.2 Outlier-trimming sweep

::: {caption="Table 2: Outlier-trimming sweep for the joint fit. Iterative residual trimming on the pooled (n = 356) fit residuals. Per-source RMSEs are on kept points. The canonical k = 15 row (bold) is justified in Appendix A.1."}

:::

A.3 Functional-form ranking for $\eta$

We compared nine candidate forms for $\eta$ on the multi-epoch corpus, holding the backbone $(B, \beta)$ fixed at the Stage-1 anchors and scoring each form by leave-one-out (LOO) RMSE on $\log \mathcal{L}$ (Table 3). The three forms with a full $R^{}(D, N)$ via Equation 5 (sat, exp-sat, tanh) dominate the rest at every parameter count. We adopt exp-sat $R^{}(D, N)$ (close 2nd in LOO behind tanh, $\Delta = 0.001$) because it is the canonical data-repetition form ([5]), has $\eta(0) = 1$ exactly, and approaches the $R^{*}$ asymptote monotonically from below.

::: {caption="Table 3: Functional-form ranking for $\eta$. LOO RMSE on log L over 108 multi-epoch points (kept after pooled residual trimming), with the backbone (B, beta) frozen at the Stage-1 anchors. The bottom three rows use the full $R^{}(D$, N) ansatz from Equation 5 (log $R^{} =$ log K + ρ log(D/N) + sigma log N). Constant- $R^{} ($single-parameter Muennighoff baseline, $R^{} = K)$ is rejected by roughly 40% relative LOO at the same parameter count, confirming that the (D, N)-dependence of $R^{}$ is load-bearing. We adopt exp-sat $R^{}(D$, N) (row in bold) as our functional form."}

:::

A.4 Per-term ablation of Equation 5

Holding the exp-sat shape of $\eta$ fixed, we ablate the terms of Equation 5 to test the necessity of each (Table 4). Each exponent is individually necessary, and Figure 2(a, c) shows why: the $D/N$- and $N$-dependence of $\eta$ are not absorbable into one another.

\begin{tabular}{l l c}
  \toprule
  Ablation & $\log R^{*}$ & LOO RMSE \\
  \midrule
  original & $\log K + \rho \log(D/N) + \sigma \log N$ & \textbf{0.020} \\
  $\hookrightarrow$ removing $N^{\sigma}$ & $\log K + \rho \log(D/N)$ & 0.026 \\
  $\hookrightarrow$ removing $(D/N)^{\rho}$ & $\log K + \sigma \log N$ & 0.025 \\
  $\hookrightarrow$ removing both & $\log K$ & 0.028 \\
  \bottomrule
  \end{tabular}

A.5 Cross-scale validation across sizes

We now sweep the fit-set cutoff $N_{\max} \in {30, 60, 100, 190}$ M and predict all held-out points at $N > N_{\max}$ (Table 6).

Held-out RMSE is stable across cutoffs between $0.05$ and $0.08$, and comparable to the in-sample RMSE at each cutoff. Once $N_{\max} \geq 60$ M, the recovered $\beta$ is within $0.03$ of the full-data value. The CD-scaling law therefore extrapolates robustly across an order of magnitude in model size.

::: {caption="Table 5: Cross-scale validation across fit-set sizes. We refit on N <= $N_m$ax and predict all held-out N > $N_m$ax points. RMSE is reported on log L."}

:::

B. Additional Experiment Details and Results

B.1 Cross-scale validation across sizes

We now sweep the fit-set cutoff $N_{\max} \in {30, 60, 100, 190}$ M and predict all held-out points at $N > N_{\max}$ (Table 6).

::: {caption="Table 6: Cross-scale validation across fit-set sizes. We refit on N <= $N_m$ax and predict all held-out N > $N_m$ax points. RMSE is reported on log L."}

:::

B.2 Loss to Downstream Breakdown

In Figure 9, and Figure 10, we report loss-to-downstream for each benchmark.

B.3 Optimal Scaling Path for 370M model

In Figure 11, we visualize the optimal scaling path for 370M model and show that the scaling law path predicted by CD-scaling laws also follow closely with the empirical optimal path.

**Figure 11:** **Optimal compute-data allocation on (C, D) loss surfaces for $N=370M$.** Replicating results for Figure 6.

B.4 Training Run Details

Hyperparameter Sweep.

We visualize the hyperparamter sweep grid and validation loss for 30M, 370M models. Lower epoch requires smaller weight-decay and as we increase data repetition, higher weight decay is necessary. For larger epochs, 30M model requires WD as high as 1.6, which is consistent with observations made in ([8]).

Other Hyperparameters.

While we sweep learning rate and weight decay for our training runs, we list the rest of hyperparameter in Table 7.

:::: cols="1"

Figure 12: Hyperparameter grid example for 30M and 370M model. More data repetition requires much higher weight decay especially for small models. ::::

\begin{tabular}{lll}
\toprule
\textbf{Category} & \textbf{Hyperparameter} & \textbf{Value} \\
\midrule
\multirow{3}{*}{Data}
   {} & Global batch size (sequences) & 512 \\
   {} & Gradient accumulation steps & 64 \\
\cmidrule{1-3}
\multirow{5}{*}{Optimization}
   {} & Learning rate & tuned \\
   {} & Optimizer & AdamW \\
   {} & $(\beta_1, \beta_2)$ & $(0.9,  0.95)$ \\
   {} & Weight decay & tuned \\
   {} & Gradient clip & 1.0 \\
\cmidrule{1-3}
\multirow{4}{*}{LR Schedule}
   {} & Schedule & cosine with warmup \\
   {} & Warmup tokens & 1B \\
   {} & Min LR ratio ($\alpha_f$) & 0.1 \\
   {} & Units & tokens \\
\cmidrule{1-3}
\multirow{3}{*}{Regularization}
   {} & Precision & BF16 (AMP) \\
   {} & Softmax auxiliary loss & \checkmark \\
   {} & Auxiliary loss multiplier & $1 \times 10^{-5}$ \\
\bottomrule
\end{tabular}

References

Section Summary: This section compiles a list of research papers and preprints that examine how large language models scale with more compute, data, and training techniques. Many entries focus on challenges like running out of human-generated data, the benefits of synthetic data, distillation methods, and optimizing data mixtures for better efficiency. Later references also cover standard benchmarks used to test model capabilities on tasks such as math, reading comprehension, and code.

[1] Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556 [cs]. doi:10.48550/arXiv.2203.15556. http://arxiv.org/abs/2203.15556.

[2] Kaplan et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361 [cs]. doi:10.48550/arXiv.2001.08361. http://arxiv.org/abs/2001.08361.

[3] Sevilla et al. (2022). Compute Trends Across Three Eras of Machine Learning. arXiv preprint arXiv:2202.05924. https://arxiv.org/abs/2202.05924.

[4] Villalobos et al. (2024). Position: Will we run out of data? Limits of LLM scaling based on human-generated data. In Proceedings of the 41st International Conference on Machine Learning. pp. 49523–49544. https://proceedings.mlr.press/v235/villalobos24a.html.

[5] Muennighoff et al. (2025). Scaling Data-Constrained Language Models. arXiv:2305.16264 [cs]. doi:10.48550/arXiv.2305.16264. http://arxiv.org/abs/2305.16264.

[6] Gadre et al. (2024). Language Models Scale Reliably with Over-Training and on Downstream Tasks. https://arxiv.org/abs/2403.08540. arXiv:2403.08540.

[7] Yan et al. (2025). Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression. arXiv:2511.13421 [cs]. doi:10.48550/arXiv.2511.13421. http://arxiv.org/abs/2511.13421.

[8] Kim et al. (2025). Pre-training under infinite compute. arXiv:2509.14786 [cs]. doi:10.48550/arXiv.2509.14786. http://arxiv.org/abs/2509.14786.

[9] Justin Lovelace et al. (2026). Prescriptive Scaling Laws for Data Constrained Training. https://arxiv.org/abs/2605.01640. arXiv:2605.01640.

[10] Maini et al. (2024). Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. arXiv:2401.16380 [cs]. doi:10.48550/arXiv.2401.16380. http://arxiv.org/abs/2401.16380.

[11] DatologyAI et al. (2025). BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining. arXiv:2508.10975 [cs]. doi:10.48550/arXiv.2508.10975. http://arxiv.org/abs/2508.10975.

[12] Kang et al. (2025). Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls. arXiv:2510.01631 [cs]. doi:10.48550/arXiv.2510.01631. http://arxiv.org/abs/2510.01631.

[13] Askari-Hemmat et al. (2025). Improving the Scaling Laws of Synthetic Data with Deliberate Practice. arXiv:2502.15588 [cs]. doi:10.48550/arXiv.2502.15588. http://arxiv.org/abs/2502.15588.

[14] Yang et al. (2025). Synthetic bootstrapped pretraining. arXiv:2509.15248 [cs]. doi:10.48550/arXiv.2509.15248. http://arxiv.org/abs/2509.15248.

[15] Hinton et al. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531 [stat]. doi:10.48550/arXiv.1503.02531. http://arxiv.org/abs/1503.02531.

[16] Busbridge et al. (2025). Distillation Scaling Laws. arXiv:2502.08606 [cs]. doi:10.48550/arXiv.2502.08606. http://arxiv.org/abs/2502.08606.

[17] Xie et al. (2023). DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. In Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2305.10429.

[18] Liu et al. (2025). RegMix: Data Mixture as Regression for Language Model Pre-training. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2407.01492.

[19] Ye et al. (2024). Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance. https://arxiv.org/abs/2403.16952. arXiv:2403.16952.

[20] Hamidieh et al. (2026). Domain-aware scaling laws uncover data synergy. arXiv preprint arXiv:2607.11052.

[21] Olmo et al. (2025). Olmo 3. arXiv:2512.13961 [cs]. doi:10.48550/arXiv.2512.13961. http://arxiv.org/abs/2512.13961.

[22] Ilya Loshchilov and Frank Hutter (2019). Decoupled Weight Decay Regularization. https://arxiv.org/abs/1711.05101. arXiv:1711.05101.

[23] Loubna Ben Allal et al. (2025). SmolLM2: When Smol Goes Big – Data-Centric Training of a Small Language Model. https://arxiv.org/abs/2502.02737. arXiv:2502.02737.

[24] Joel Niklaus et al. (2026). The Synthetic Data Playbook: Generating Trillions of the Finest Tokens.

[25] Besiroglu et al. (2024). Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102.

[26] Cobbe et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

[27] Joshi et al. (2017). Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1601–1611.

[28] Chen et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

[29] Paperno et al. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031.

[30] Zellers et al. (2019). Hellaswag: Can a machine really finish your sentence?. arXiv preprint arXiv:1905.07830.

[31] Todor Mihaylov et al. (2018). Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In EMNLP.

[32] Lai et al. (2017). Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 conference on empirical methods in natural language processing. pp. 785–794.

[33] Rajpurkar et al. (2018). Know What You Don't Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 784–789. doi:10.18653/v1/P18-2124. https://aclanthology.org/P18-2124. arXiv:1806.03822.

[34] Dan Lee et al. (2026). Training Language Models via Neural Cellular Automata. https://arxiv.org/abs/2603.10055. arXiv:2603.10055.

[35] Bibliography entry for citation key "hampel1974influence" was not supplied with the source.