Model Collapse: What Happens When AI Trains on AI-Generated Data

Model collapse is the name researchers gave to a specific failure: a generative model trained on output from earlier models slowly forgets the rare parts of the original data, and after enough rounds it produces a narrower, blander, sometimes outright wrong picture of the world. The effect is real in the lab. Whether it will happen to the large language models people actually use, now that the open web contains a growing share of machine-written text, is a different and unsettled question.

This article separates what has been shown from what has only been argued, walks through the four main positions, and reports a small simulation built for it that compares three ways of handling synthetic data. The short version: throwing away real data is what causes the damage; keeping it, or curating what replaces it, appears to stop most of it. The open part is how much of that protection survives contact with a messy, unlabeled internet.

The question, stated precisely

Suppose model generation 1 is trained on human-produced data. Its outputs are published and mixed into the corpus used to train generation 2, whose outputs feed generation 3, and so on. Two questions follow. First, does quality or diversity decay as the generations pile up? Second, if it does, which practical choices about data (how much real data is kept, whether synthetic samples are filtered, whether provenance is tracked) prevent the decay, and at what cost?

People clearly want an answer. A search for the question turns up press coverage such as Bloomberg’s piece on why researchers are raising alarms about model collapse and blog posts with titles like “Is AI Model Collapse Inevitable?”. The phrasing in that title is the right one, because the research literature has moved from “does it happen?” to “under which conditions?”

Stakes are practical. If collapse is the default outcome, pre-2023 human text becomes a scarce asset, and labs with large archives of it gain a lasting advantage. If it is easily avoided, synthetic data is just another ingredient to be measured. There is also a measurement worry: machine output is already common online. One preprint found that a large share of web text in lower-resource languages appears to be low-quality machine translation (Thompson et al. 2024, preprint), which shows the contamination problem is not hypothetical, even if it says nothing direct about collapse.

What is established

The landmark paper is Shumailov et al. (2024) in Nature, first circulated as a 2023 preprint. The authors fine-tuned a small language model, OPT-125m, on the wikitext2 dataset, generated text from it, trained the next model on that text, and repeated. They also analysed simpler cases, including Gaussian distributions, and found the same pattern. Their summary of the mechanism:

“indiscriminate use of model-generated content in training causes irreversible defects in the resulting models”

Shumailov et al., 2024, Nature

In that paper the tails of the original distribution disappear first. Rare events get sampled less often, so the next model sees fewer of them, assigns them still lower probability, and eventually drops them entirely. For the Gaussian case the authors prove that the fitted covariance shrinks to zero almost surely as generations accumulate. They attribute the drift to three sources: statistical approximation error (finite samples), functional expressivity error (the model family cannot represent the truth exactly) and functional approximation error (training procedures are imperfect).

Two details from the same paper matter for everything that follows. In the language-model runs with no original data retained, perplexity on the original test set worsened, from about 20 to 28 points by the paper’s account. When 10% of the original data was preserved in each round, the authors report only minor degradation. So even the founding result contains the seed of the mitigation debate.

Image models show a parallel effect. Alemohammad et al. (2023), a preprint, called the loop “self-consuming” and coined the term Model Autophagy Disorder. Their conclusion is stated bluntly:

“without enough fresh real data in each generation of an autophagous loop, future generative models are doomed to have their quality (precision) or diversity (recall) progressively decrease.”

Alemohammad et al., 2023, arXiv preprint

Bertrand et al. (2023), another preprint, gave stability conditions for iterative retraining: the loop stays stable when the first model approximates the data well enough and when the share of clean real data is large enough relative to synthetic data. Taken together, these results establish a fact and a boundary. The fact: pure self-training on finite samples degrades a model. The boundary: that result assumes real data gets replaced rather than kept.

Four answers to whether model collapse can be avoided

1. Collapse is the default, and the web is already contaminated

The strongest form of this view says that the internet is becoming a closed loop, that nobody can reliably filter out machine text, and that later models will therefore inherit the narrowing. Evidence for it: the original experiments, the image-model results, and the general difficulty of detecting machine text after the fact. Evidence against: it assumes labs discard old corpora, which no published account of frontier training suggests, and it treats all synthetic data as uncurated. Much of the alarming press coverage implicitly rests on that replacement assumption.

2. Accumulating data instead of replacing it solves the problem

Gerstgrasser et al. (2024), a preprint, ran the same kind of loop on language models, diffusion models and variational autoencoders, but let each generation’s synthetic data pile on top of the original rather than displace it. Collapse disappeared in their experiments. For a linear model they proved the point:

“if data instead accumulate, the test error has a finite upper bound independent of the number of iterations, meaning model collapse no longer occurs.”

Gerstgrasser et al., 2024, arXiv preprint

A follow-up by Kazdan et al. (2024), presented at NeurIPS 2024 workshops, tested three workflows. Replacing data collapsed in every task setting they studied; accumulating stayed stable even as the real share shrank towards zero; and a fixed compute budget drawn from the growing pool degraded slowly rather than explosively. The critique of this camp is that accumulation is an idealisation. Real training runs have budgets, deduplicate aggressively and reweight sources, which pushes practice towards the fixed-budget case.

3. Even a small synthetic share changes how models scale

A third line, mostly from Dohmatob, Kempe and collaborators, argues that collapse is better described as a change in scaling laws. Dohmatob et al. (2024) model how synthetic data truncates the heavy tail of real data and show effects such as lost scaling, shifted scaling across generations and “unlearning” of skills. In “Strong Model Collapse” the same group reports, in a supervised regression setting, that even a tiny synthetic fraction (they discuss shares around 1%) can stop larger training sets from helping, and that bigger models can intensify the problem before an interpolation threshold. Both are preprints. Their force is theoretical; the open question is how closely regression on random features predicts behaviour in trillion-token language models.

4. Verified synthetic data is an asset, not a contaminant

The fourth position flips the framing. If a verifier selects which generated samples to keep, the loop is no longer blind resampling but something closer to search. Feng et al. (2024), a preprint, found that verifiers prevented collapse in transformer eigenvalue computation and in news summarization, even when the verifiers were imperfect. The catch is obvious: verification is easy for arithmetic and code, harder for prose, and close to impossible for open-ended cultural knowledge, which is precisely where rare facts live.

A small simulation built for this article

To make the competing claims concrete, this article uses an iterated-fitting simulation written in Python with NumPy (random seed fixed at 3). It does not train a neural network. It isolates the statistical mechanism that all four camps agree is at the root of the problem: each generation fits a model to a finite sample drawn from the previous generation’s model.

Two toy worlds were used. In the first, the “true” data are standard normal; each generation draws 100 points and fits a mean and standard deviation, repeated across 1,000 independent replicates for 500 generations. In the second, a vocabulary of 1,000 tokens follows a Zipf law (exponent 1.1); each generation draws 10,000 tokens and the next model is simply the observed frequency table, repeated 50 times for 200 generations. Every world was run under four regimes:

• Replace: each generation trains only on samples from the previous model.
• Mix 10% and Mix 25%: that share of each generation’s training set is fresh real data; the rest is synthetic.
• Accumulate: synthetic samples are added to everything seen so far, including the original real data, and the model is fitted on the whole pile.

For the Gaussian world I also ran the replace regime with the unbiased (n−1) variance estimator, to check whether the well-known downward bias of the maximum-likelihood estimator is the whole story. The full module is short:

import numpy as np
rng = np.random.default_rng(3)
R, n, G = 1000, 100, 500          # replicates, samples/generation, generations

def gauss(mode, alpha=0.0, ddof=0):
    mu, sd = np.zeros(R), np.ones(R)
    S1, S2, N = np.zeros(R), np.zeros(R), 0   # running sums for 'accumulate'
    out = {}
    for g in range(1, G + 1):
        k = int(round(alpha * n))              # fresh real samples this generation
        x = rng.normal(mu[:, None], sd[:, None], (R, n - k))
        if k: x = np.hstack([x, rng.normal(0, 1, (R, k))])
        if mode == "accumulate":
            if g == 1: x = rng.normal(0, 1, (R, n))
            S1 += x.sum(1); S2 += (x**2).sum(1); N += n
            mu = S1 / N; sd = np.sqrt(S2 / N - mu**2)
        else:
            mu, sd = x.mean(1), x.std(1, ddof=ddof)  # ddof=0: maximum likelihood
        if g in (1, 10, 50, 100, 200, 500): out[g] = (mu.copy(), sd.copy())
    return out

def tail(mu, sd, c=3.0):                       # model mass beyond |x|>3 (true: 0.0027)
    from math import erf, sqrt
    Phi = np.vectorize(lambda z: 0.5 * (1 + erf(z / sqrt(2))))
    return 1 - Phi((c - mu) / sd) + Phi((-c - mu) / sd)

def vocab(mode, alpha=0.0, V=1000, N=10_000, G=200, s=1.1):
    p0 = 1 / np.arange(1, V + 1) ** s; p0 /= p0.sum()
    p, C, out = p0.copy(), np.zeros(V), {}
    for g in range(1, G + 1):
        k = int(round(alpha * N))
        c = rng.multinomial(N - k, p) + (rng.multinomial(k, p0) if k else 0)
        if mode == "accumulate":
            if g == 1: c = rng.multinomial(N, p0)
            C += c; p = C / C.sum()
        else:
            p = c / N
        if g in (1, 10, 50, 100, 200):
            H = -(p[p > 0] * np.log2(p[p > 0])).sum()
            out[g] = ((p > 0).sum(), (p[500:] > 0).sum(), p[500:].sum() / p0[500:].sum(), H)
    return out

A separate driver script called these functions, summarised the replicates and wrote the numbers below; the whole run took about 15 seconds.

What the simulation produced

The Gaussian results first. The true standard deviation is 1, and the true probability of landing beyond ±3 is about 0.0027. The table gives the median fitted standard deviation across replicates, the share of replicates whose fitted spread had fallen below 0.5, and the median probability the fitted model assigns to the region beyond ±3.

Under pure replacement the distribution narrows steadily, and by generation 50 the median model already assigns essentially zero probability to the outer tails. Switching to the unbiased estimator slows the slide but does not stop it: after 500 generations 86% of replicates had still lost more than half their spread. That matches the theory in the Nature paper, where the shrinkage comes from compounding sampling noise, not merely from a biased formula. Even 10% fresh data each round was enough to hold the spread near 0.95, though not at 1; mixing left a small, stable deficit rather than a perfect recovery.

The token world tells a subtler story. At generation 1 a sample of 10,000 tokens already misses about 160 of the 1,000 types, and about 357 of the 500 rarest ranks show up at least once. The true entropy of the distribution is 6.80 bits.

One result is easy to miss. Under replacement, the total probability mass sitting in the rare half of the vocabulary stayed, on average, roughly where it started (the median ratio to the true value at generation 200 was 0.99). Frequency resampling is unbiased in expectation, so mass is conserved on average. What disappears is diversity: by generation 200 only about six of the 500 rare types survived, and they had swollen to absorb the probability of the hundreds that died. This is the population-genetics picture of neutral drift, and it means a metric like “share of rare tokens” can look healthy while the variety underneath collapses. Entropy and type counts caught it; aggregate tail mass did not.

Accumulation froze the world at its generation-1 state. That is good news and bad news. Nothing was lost after the first round, yet the 160 types missed by the first sample never came back, because a model can only resample what it has seen. Fresh real data is the only regime in which unseen types kept reappearing.

What the simulation does and does not show

It shows, in a setting simple enough to check by hand, three things the literature claims: replacement degrades spread and diversity; a modest stream of real data halts the decay at a slightly degraded equilibrium; and accumulation prevents further loss while preserving the first generation’s blind spots. It also illustrates a measurement trap worth stating plainly: aggregate tail mass can survive while the number of distinct rare items falls by two orders of magnitude.

It does not show anything about neural networks specifically. Real models smooth and generalise, which can invent plausible new rare items (helpful) or hallucinate implausible ones (harmful); neither happens in a frequency table. There is no curation, no deduplication, no instruction tuning and no human preference step. Sample sizes here are tiny compared with pretraining corpora, and larger samples slow every effect. Treat the numbers as an illustration of a mechanism, not a forecast for any product. Readers who want the neural side of this can start with how training actually updates weights, covered in our explainer on how neural networks learn through backpropagation.

The author’s reading

What follows is opinion, labelled as such. Model collapse in the strict, runaway sense looks to me like a solved problem for any lab that keeps its historical corpora, and every major lab has strong commercial reasons to do so. The live risk is quieter. It is the fixed-budget, mixed-provenance regime that Kazdan and colleagues describe, combined with the scaling-law effects Dohmatob’s group predicts: a slow tax on diversity and long-tail knowledge that never produces a dramatic failure, only gradually more homogeneous output.

Provenance, then, matters more than volume. A lab that can tell which documents are human, which are machine-made and which are machine-made but verified can choose its mixture; a lab that cannot is flying blind. That is why I think tools for labelling content at creation, such as the C2PA credentials discussed in our guide to checking whether a video is AI-generated, are more promising than after-the-fact detection, whose weaknesses are covered in our piece on why AI text detectors fail.

Studies that would settle it

Several concrete, falsifiable tests would move the question from argument to evidence:

• Matched-corpus pretraining. Train two identical mid-sized models on equal token budgets drawn from web crawls before and after the spread of chat assistants, holding filtering fixed. If collapse is underway in the wild, the later model should show lower output diversity and worse rare-fact recall at equal benchmark scores. No difference would count against the alarmist reading.
• Fixed-budget dose response. Vary the synthetic share (0%, 1%, 5%, 20%, 50%) at constant compute across several model sizes and fit scaling curves. The strong-collapse hypothesis predicts that the exponent itself bends at small shares; the benign hypothesis predicts only a constant offset.
• Diversity metrics that count types, not mass. Report distinct rare entities, n-gram type counts and entropy across generations, since the simulation above shows tail mass alone can hide collapse.
• Verifier stress tests in open domains. Extend the Feng et al. design from arithmetic and summarization to long-tail factual knowledge, where verifiers are weakest. If collapse returns there, verification is a partial cure.

Common questions

Is model collapse already happening to chatbots?

No published study has shown it in a deployed frontier model. The demonstrations are controlled experiments with small models or simplified settings. Whether a slower version is affecting current systems is exactly the open question this article covers.

Does any synthetic data cause model collapse?

Not according to the best evidence. Collapse appears reliably when synthetic data replaces real data, and largely disappears when real data is kept or synthetic samples are verified. Some theory suggests even small shares can change scaling behaviour, but that remains untested at frontier scale.

Why do rare things disappear first?

Every generation learns from a finite sample. Rare events are the most likely to be undersampled or missed entirely, and once a model assigns them zero probability it can never generate them again for the next generation to learn from.

Is model collapse the same as overfitting?

No. Overfitting is a single model memorising its training set. Collapse is a multi-generation process in which each model may fit its own data well, yet the chain as a whole drifts away from the original distribution.

References

1. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631, 755–759. https://www.nature.com/articles/s41586-024-07566-y
2. Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., & Baraniuk, R. G. (2023). Self-Consuming Generative Models Go MAD. arXiv:2307.01850 (preprint). https://arxiv.org/abs/2307.01850
3. Bertrand, Q., Bose, A. J., Duplessis, A., Jiralerspong, M., & Gidel, G. (2023). On the Stability of Iterative Retraining of Generative Models on their own Data. arXiv:2310.00429 (preprint). https://arxiv.org/abs/2310.00429
4. Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., Roberts, D. A., Yang, D., Donoho, D. L., & Koyejo, S. (2024). Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. arXiv:2404.01413 (preprint). https://arxiv.org/abs/2404.01413
5. Kazdan, J., Schaeffer, R., Dey, A., Gerstgrasser, M., Rafailov, R., Donoho, D. L., & Koyejo, S. (2024). Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World. NeurIPS 2024 Workshops (M3L, ATTRIB); arXiv:2410.16713. https://arxiv.org/abs/2410.16713
6. Dohmatob, E., Feng, Y., Yang, P., Charton, F., & Kempe, J. (2024). A Tale of Tails: Model Collapse as a Change of Scaling Laws. arXiv:2402.07043 (preprint). https://arxiv.org/abs/2402.07043
7. Dohmatob, E., Feng, Y., Subramonian, A., & Kempe, J. (2024). Strong Model Collapse. arXiv:2410.04840 (preprint). https://arxiv.org/abs/2410.04840
8. Feng, Y., Dohmatob, E., Yang, P., Charton, F., & Kempe, J. (2024). Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification. arXiv:2406.07515 (preprint). https://arxiv.org/abs/2406.07515
9. Thompson, B., Dhaliwal, M. P., Frisch, P., Domhan, T., & Federico, M. (2024). A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism. arXiv:2401.05749 (preprint). https://arxiv.org/abs/2401.05749
10. Bloomberg (2024, August 5). AI model collapse: why researchers are raising alarms [news article]. https://www.bloomberg.com/news/articles/2024-08-05/ai-model-collapse-why-researchers-are-raising-alarms
11. Vincent, W. S. Is AI Model Collapse Inevitable? [blog post]. https://wsvincent.com/is-model-collapse-inevitable/