Why Do AI Scaling Laws Follow Power Laws? Four Explanations Tested

AI scaling laws are among the most dependable regularities in modern machine learning, and nobody can fully say why they take the shape they do. Train a language model with more parameters, more data or more compute, and its test loss falls along a straight line on log-log axes: a power law. Labs plan budgets on the assumption that the line keeps going. The open question is narrower than “does scaling work” and harder: what property of the data, the networks or the training procedure forces the curve to be a power law rather than an exponential, a logarithm or a curve that bends toward a wall?

The question matters for a practical reason. An empirical fit tells you where the line has been; a mechanism tells you where it must break. If the exponent comes from the statistics of language, better architectures cannot change it much and better data might. If the power law is mostly a convenient curve fitted over a limited range, extrapolations several orders of magnitude out are guesses.

People do ask this outside of papers. A 2026 LessWrong essay titled “Power Laws Are Not Enough” argues that the standard spectral explanation shows learning is happening without explaining why the learned representations become capable, and ICLR 2023 hosted a workshop on driving scaling-law theory through experimental insights. What follows covers what is established, the leading explanations, a small simulation of one of them, and tests that could separate them.

What the scaling-law measurements actually show

The empirical story is older. Hestness et al. (2017) at Baidu measured learning curves in translation, language modeling, image classification and speech recognition and found generalization error falling as a power of training-set size in each. They also observed that model improvements shifted the curve without changing its slope.

Kaplan et al. (2020) then fitted transformer language models across a much wider range. Cross-entropy loss followed L(N) = (N_c/N)^α_N with α_N ≈ 0.076 in non-embedding parameters, L(D) with α_D ≈ 0.095 in tokens, and a compute-optimal frontier with exponent about 0.050. Width versus depth barely mattered within a broad range. The caveats appendix is blunt:

“At present we do not have a solid theoretical understanding for any of our proposed scaling laws.”

Kaplan et al., 2020, Appendix C

The exponents are small. An α_N of 0.076 means a tenfold increase in parameters cuts the loss by only about 16 percent, which is why scale has to grow so fast to buy visible gains.

Two later results showed how fragile the fitted numbers are. Hoffmann et al. (2022) trained more than 400 models and concluded that parameters and tokens should grow in roughly equal proportion, so their 70-billion-parameter Chinchilla beat the 280-billion-parameter Gopher at the same compute; Kaplan’s allocation had favored much larger models on less data. Then Besiroglu et al. (2024, preprint) reconstructed data from Chinchilla’s plots, found the third estimation method inconsistent with the other two, and judged its confidence intervals far too tight.

So three things count as established: power-law fits describe loss over many orders of magnitude in many domains; exponents depend on task and data far more than on architecture; and the precise values guiding compute allocation are sensitive to fitting choices. The cause is not established.

Explanation one: networks are resolving a data manifold

The geometric account starts from a simple picture. Suppose the inputs that matter lie on a smooth manifold of intrinsic dimension d, even if they are embedded in a space of millions of pixels or tokens. A network with N parameters can carve that manifold into roughly N pieces and fit a simple function on each. Approximation error in each piece shrinks with its size, and piece size shrinks as N^-1/d. Sharma and Kaplan (2020, preprint) worked this through and predicted a loss exponent of about 4/d for both cross-entropy and mean-squared error, then checked it on teacher-student networks with controlled dimension, image classifiers and GPT-style models.

Bahri et al. (2021; published in PNAS, 2024) extended this into four regimes. When either data or width is effectively unlimited, loss approaches its limit with a simple 1/D or 1/N correction (they call this variance-limited). When the bottleneck is how finely the model can resolve the manifold, you get the non-trivial power laws (resolution-limited). Their paper opens with the observation it is trying to explain:

“The population loss of trained deep neural networks often follows precise power-law scaling relations”

Bahri et al., 2021/2024, abstract

The strength of this view is a testable link: independently measured intrinsic dimension should predict the exponent, and Sharma and Kaplan report rough agreement in controlled settings. Language is the weak spot. An exponent of 0.076 implies d near 50 under the 4/d rule, and nobody has a clean, independent way to measure the intrinsic dimension of text to check that number. The theory also predicts that parameter and data exponents should match in the resolution-limited regime, a duality Bahri et al. report evidence for; Kaplan’s 0.076 and 0.095 are close but not equal.

Explanation two: power laws inherited from the data spectrum

A close cousin locates the power law in second-order statistics. If the eigenvalues of the data covariance (or of the kernel a network effectively learns) decay as a power of their rank, then a model that captures the top N directions leaves a residual error that also decays as a power of N. Maloney, Roberts and Sully (2022, preprint) built a solvable model of exactly this kind, combining a generative data model with random features, and showed that power laws in the data statistics pass through nonlinear feature maps into power laws in test loss.

Their model also says where the law ends: real spectra are finite, so once a model has captured every direction with meaningful variance, performance plateaus. That built-in failure condition is more than most fitted curves offer.

The LessWrong essay cited above presses on a gap: a covariance spectrum can explain why loss falls smoothly, yet loss is not capability, and two models with equal loss could differ in what they can do.

Explanation three: a long tail of discrete skills

The quantization model of Michaud, Liu, Girit and Tegmark (2023, preprint) changes the unit of analysis. It proposes that prediction decomposes into discrete chunks of knowledge or skill, called quanta, that some quanta reduce loss more than others, and, critically:

“The frequencies at which the quanta are used for prediction follow a power law.”

Michaud et al., 2023, hypothesis QH3

If quantum k is used with frequency proportional to k^-(α+1) and a network learns quanta in order of frequency, then a model that has room for n quanta leaves only the tail unlearned. Summing that tail gives loss falling as n^-α, so as N^-α if each quantum costs a roughly fixed number of parameters. For data, a quantum is learnable once it has appeared enough times, which yields a data exponent of α/(α+1). It also explains sudden jumps on narrow tasks, since each quantum is either learned or not.

The supporting evidence is suggestive rather than decisive. The authors validated the mechanism on toy sparse-parity tasks, then clustered a real language model’s gradients into candidate skills. The cluster rank-frequency slope came out near −1.24 between ranks 100 and 1,000, steeper than the −1.08 implied by the measured parameter exponent; the authors judge the gap within the margin of error of their clustering method (Michaud et al. 2023, Figure 5). Related arguments derive scaling from Zipf’s law in word frequencies via Heaps’ law and Hilberg’s hypothesis (Dębowski, 2025, preprint), and Hutter (2021, preprint) built a toy learner whose learning curve can decay with any positive exponent depending on the data distribution.

Brill (2024, preprint) offers a partial reconciliation. Using percolation theory, he finds two regimes, one resembling power-law-distributed subtasks and one a dominant manifold, so the quantization and geometric stories may be two limits of one picture.

Explanation four: the power law is partly a fitting convenience

The skeptical position does not deny that the curves are smooth. It denies that “power law” is a law. Caballero et al. (2022, preprint) proposed a smoothly broken power law and reported that it extrapolates better than a single power law across vision, language, audio, reinforcement learning and other settings, including cases with double descent and sharp delayed inflections. A curve that needs breaks is a power law only piecewise.

Sorscher et al. (2022, preprint) went further on the data axis. With a good enough pruning metric that chooses which examples to keep, they argued, error can in theory fall faster than any power law, possibly exponentially, and they reported better-than-power-law scaling on CIFAR-10, SVHN and ImageNet. That does not refute the other theories, but it suggests the familiar exponent describes randomly sampled data, a fact about dataset collection as much as learning.

Add the Chinchilla replication episode and a cautious reader holds two thoughts at once: the shape is robust over the ranges measured, and the exponents are less certain than their two-decimal presentation suggests.

A small simulation of the quanta hypothesis

To see what the quantization account does and does not deliver, I built a small, deliberately stripped-down simulation for this article. The “world” contains K = 200,000 quanta with Zipf frequencies p_k ∝ k^-s. A model with capacity n learns exactly the n most frequent quanta, and every sample that needs an unlearned quantum costs one unit of loss. For the data axis, the model has unlimited capacity but only learns a quantum after seeing it at least five times in D random samples (five repeats per point, seed 42). Exponents are least-squares slopes on log-log axes over n from 10 to about 5,000 and D from 1,000 to about 3 million.

import numpy as np
rng = np.random.default_rng(42)
K = 200_000                      # number of quanta (skills) in the "world"
B = 1.0                          # extra loss on a sample whose quantum is not learned

def zipf_p(s):                   # frequency of quantum k is proportional to k^-s
    p = np.arange(1, K + 1, dtype=float) ** -s
    return p / p.sum()

def loss_params(p, n):           # model with capacity for n quanta learns the n most frequent
    return B * p[n:].sum()

def loss_data(p, D, tau=5):      # infinite capacity; a quantum is learned once seen >= tau times
    counts = rng.multinomial(D, p)
    return B * p[counts < tau].sum()

def slope(x, y):                 # least-squares exponent on log-log axes
    return np.polyfit(np.log(x), np.log(y), 1)[0]

ns = np.unique(np.logspace(1, 3.7, 12).astype(int))     # 10 .. ~5,000 quanta
Ds = np.logspace(3, 6.5, 12).astype(int)                # 1e3 .. ~3e6 samples
print("s    alpha  fit_N  pred_N  fit_D  pred_D")
for s in [1.25, 1.5, 2.0, 2.5, 3.0]:
    p = zipf_p(s); a = s - 1
    LN = [loss_params(p, n) for n in ns]
    LD = [np.mean([loss_data(p, D) for _ in range(5)]) for D in Ds]
    print(f"{s:<5}{a:<7.2f}{-slope(ns, LN):<7.3f}{a:<8.3f}"
          f"{-slope(Ds, LD):<7.3f}{a/(a+1):.3f}")

# Control: frequencies that fall off exponentially instead of as a power law
p = np.exp(-np.arange(K) / 300.0); p /= p.sum()
LN = np.array([loss_params(p, n) for n in ns])
local = np.diff(np.log(LN)) / np.diff(np.log(ns))
print("exp control: local log-log slopes", np.round(local, 2))

# Finite world: Zipf s=1.5 but only 2,000 quanta exist -> curve leaves the power law
K2 = 2000; p = np.arange(1, K2 + 1) ** -1.5; p /= p.sum()
LN = np.array([B * p[n:].sum() + 1e-12 for n in ns[ns < K2]])
local = np.diff(np.log(LN)) / np.diff(np.log(ns[ns < K2]))
print("finite K=2000: local slopes", np.round(local, 2))

Four results came out of the run.

• For moderate and steep tails (s from 2 to 3), the fitted exponents match the analytic predictions to within about 0.01 on both axes. A Zipf tail of skills produces a power law in loss, with the exponent set by the tail rather than the learner.
• For shallow tails the fit drifts high. At s = 1.25 the capacity exponent came out at 0.312 instead of 0.25. Rerunning with more quanta pulled it down to 0.280 for K = 2 million and 0.265 for K = 20 million. The bias comes from truncating the unlearned tail at a finite number of skills.
• With exponentially decaying frequencies instead of Zipf (a control), there is no power law at all. Local log-log slopes ran from −0.04 at small n to −12.8 at large n. The learner passes power laws through; it does not manufacture them.
• With only 2,000 quanta in total, the local slope steepened from −0.52 to −2.56 as capacity approached the number of skills. A finite skill inventory predicts a curve that bends downward off the straight line.

The second finding matters most for language models. Kaplan’s α_N ≈ 0.076 corresponds, under this model, to s ≈ 1.08, an even shallower tail than the hardest case I ran. In that regime the measured exponent is hypersensitive to how many quanta exist and where the fitting window sits. So comparisons such as Michaud’s measured −1.24 against an implied −1.08 deserve caution: finite-size effects like those seen here could move either number by that much.

A piece of arithmetic points the same way. The simple quantization relation says the data exponent should equal α_N/(α_N+1). Plugging in 0.076 gives about 0.071, against Kaplan’s fitted 0.095. Kaplan’s data curves came from early-stopped training, not the repeated-exposure setting the formula assumes, so this is a prompt for a better test, not a refutation.

What the simulation does not show is equally clear. It assumes quanta exist, are independent, cost equal parameters and are learned strictly in frequency order. Real networks share features across skills, learn some rare things early, and follow optimizer dynamics this toy ignores. The run shows the quantization story is internally coherent and where it is fragile. It is no evidence that language models actually work this way.

The author’s reading, and what would settle it

What follows is opinion, labeled as such. My reading is that the power law is inherited from heavy-tailed structure in natural data, and that the geometric, spectral and quanta accounts are three descriptions of that inheritance at different levels of grain. The control run above is the cleanest small argument for this: change the frequency tail from a power law to an exponential and the power law in loss disappears. I also think the skeptics are right that the exponents for frontier language models are known less precisely than the field’s habits suggest, and that extrapolations two or three orders of magnitude past the data deserve wide error bars.

The accounts do make different predictions, and that is where the question can be picked up. Five tests, each falsifiable:

• Swap the tail. Train identical small transformers on synthetic corpora whose generative process has controlled Zipf exponents. Data-centric theories predict the loss exponent moves with the tail; if it stays fixed, they are in trouble.
• Measure dimension independently. The manifold view predicts α ≈ 4/d. Estimate intrinsic dimension of representations for several modalities with methods fixed in advance, then check whether 4/d predicts the exponent better than chance across tasks.
• Check the exponent link. Quantization predicts α_D = α_N/(α_N+1) under multi-epoch training; resolution-limited theory predicts α_D ≈ α_N. For small exponents these differ by roughly 7 percent, which a controlled multi-epoch sweep with enough runs could detect.
• Look for the bend. Both the spectral model and my finite-quanta run predict departures from the straight line as a model exhausts the data’s structure. On small, closed domains (a fixed formal grammar, a finite game) the curve should leave the power law at a predictable point.
• Prune and watch. If curated data beats the power law on language models at scale, as Sorscher et al. found for image classifiers, the exponent is a property of random sampling rather than of learning.

For the mechanics underneath, start with how gradient descent and backpropagation train a network, since every theory here assumes that optimizer works.

Why the mechanism changes engineering decisions

If loss tracks the tail of a fixed data distribution, the cheapest gains come from changing that distribution: deduplication, curation, synthetic data aimed at rare skills. If the manifold account holds, gains come from architectures and tokenizations that lower the effective dimension, which is one way to read work on byte-level models like the one covered in our piece on Bolmo and byteification. Sparse architectures complicate the parameter axis too, since a mixture-of-experts model does not use all of its parameters per token, a point that comes up in our explainer on Mistral Large 4’s sparse design.

Pretraining loss is also only one axis. Inference-time compute follows its own trade-off curves, and fixed token budgets can misstate how models compare, as discussed in our article on inference compute and benchmark scores. Pretraining theories say nothing yet about those.

Common questions

What is a power law in AI scaling?

It is a relationship of the form loss = a × N^-α, where N is parameters, data or compute. On a log-log plot it is a straight line, and the exponent α sets how much each tenfold increase buys; Kaplan et al. measured α under 0.1 for each resource.

Do scaling laws mean AI will keep improving forever?

No. A power law in loss approaches an irreducible floor set by the entropy of the data, and several theories predict the curve bends or plateaus once a model exhausts the structure in its data. Lower loss also does not translate one-for-one into new abilities.

Is the quanta model proven?

It is a hypothesis with supporting evidence from toy tasks and a tentative measurement on one family of language models. The measured skill-frequency slope was close to the predicted value, within what the authors call the margin of error, and the measurement depends on how skills are clustered.

Can better data beat a power law?

For image classifiers, Sorscher et al. reported better-than-power-law scaling when training data was pruned with a good metric. Whether it holds for large language models is unsettled.

References

1. Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2021; 2024). Explaining neural scaling laws. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.2311878121. arXiv:2102.06701
2. Besiroglu, T., Erdil, E., Barnett, M., & You, J. (2024). Chinchilla scaling: A replication attempt. Preprint, arXiv:2404.10102
3. Brill, A. (2024). Neural scaling laws rooted in the data distribution. Preprint, arXiv:2412.07942
4. Caballero, E., Gupta, K., Rish, I., & Krueger, D. (2022). Broken neural scaling laws. Preprint, arXiv:2210.14891
5. Dębowski, Ł. (2025). From Zipf’s law to neural scaling through Heaps’ law and Hilberg’s hypothesis. Preprint, arXiv:2512.13491
6. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y. (2017). Deep learning scaling is predictable, empirically. Preprint, arXiv:1712.00409
7. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., et al. (2022). Training compute-optimal large language models. Preprint, arXiv:2203.15556
8. Hutter, M. (2021). Learning curve theory. Preprint, arXiv:2102.04074
9. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models. Preprint, arXiv:2001.08361
10. Maloney, A., Roberts, D. A., & Sully, J. (2022). A solvable model of neural scaling laws. Preprint, arXiv:2210.16859
11. Michaud, E. J., Liu, Z., Girit, U., & Tegmark, M. (2023). The quantization model of neural scaling. Preprint, arXiv:2303.13506
12. Sharma, U., & Kaplan, J. (2020). A neural scaling law from the dimension of the data manifold. Preprint, arXiv:2004.10802
13. Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., & Morcos, A. S. (2022). Beyond neural scaling laws: Beating power law scaling via data pruning. Preprint, arXiv:2206.14486
14. CarolusRenniusVitellius (2026). Power laws are not enough. LessWrong (essay, not peer reviewed). lesswrong.com
15. ICLR 2023. Scaling laws for deep neural networks: Driving theory and understanding through experimental insights (workshop page). iclr.cc