AI models are breeding, and biology already did the maths
Almost every AI model released this year is descended from another one. Some were made by taking a big model and training it a little further on a narrow task. Some were trained on text that an earlier model wrote. Some were built by taking two existing models and averaging the millions of numbers inside them, which sounds like it should not work, and often does. Public repositories, like Hugging Face, now hold millions of these things, and their family trees have already been drawn by people using software written for tracing species.
A group whose members inherit from one another, combine, and pass the result on again is a population, in the sense a biologist means it. Biology has spent a century working out what happens to populations, and what I found is that the theory carries over to machines with its numbers still attached.
The rare things go first
Train a model on the output of the model before it, generation after generation, and it degrades. This is known, and it has a name: model collapse, the term Shumailov and colleagues introduced when they showed it across model families. What is less appreciated is the shape of the damage. Common abilities survive. Rare ones vanish, and they vanish first.
Biologists have known this pattern since the 1930s and it has nothing to do with quality. It is the accident by which rare surnames disappear from small villages: nobody is selecting against them, but each generation is a small sample of the one before, and a name held by two families can be lost by chance in a way that a name held by two hundred cannot. Geneticists call it drift. A model retrained on its own output is running exactly that process, and the maths turns out to be the same equation, not merely a similar one.
How much reality does a model need?
The obvious fix is to keep feeding the thing real human-made data. The question is how much, and here the answer surprised me.
It is not a percentage. It is a count.
What protects a rare ability is the number of real examples of it that reach each generation, and the size of everything else in the training set makes no difference whatsoever. Ten real examples per generation preserve about 95% of the variety, and that is true whether the training set holds two hundred items or two million. Conservation biologists have a rule of thumb for exactly this, worked out for small wild populations: one migrant per generation. One. Not one percent.
The practical version is a budget, and it is a demanding one. To hold on to an ability that shows up in one real example in ten thousand, you need roughly ten thousand real examples every generation, and they have to be examples of that ability. Real data about something else will not do.
A Victorian objection to Darwin, rediscovered by machine learning
Now the part I enjoyed most.
In 1867 an engineer called Fleeming Jenkin published an objection to Darwin that was, at the time, devastating. If children are an average of their parents, he argued, then any rare advantageous trait gets halved at every generation and is watered away to nothing long before natural selection can act on it. Darwin had no good answer. The answer arrived with Mendel: traits do not blend, they pass on whole or not at all. You inherit your grandmother’s eye colour intact, not a smeared average of four grandparents.
Model merging, one of the most popular techniques in AI today, is blending. When you average two models together, an ability held by only one of them gets half the weight, exactly as Jenkin described. I found that machine learning has run into this three separate times, under three different names, without anyone noticing it was the same phenomenon, and that it was a 159-year-old objection to the Origin of Species.
The fix is the one biology arrived at. Stop averaging. Keep the specialists whole and send each question to the one that knows the answer, or generate several candidate offspring and keep whichever tests best. In the experiments, blind averaging on hard tasks did no better than picking the best single specialist. Keeping them separate beat it every time.
When two models can no longer be crossed
A horse and a donkey give you a mule, and the mule is sterile. Somewhere along the way two lineages drift far enough apart that they stop producing viable offspring. That is speciation, and I wanted to know whether it happens to models.
The obvious guess is that models whose internals have drifted far apart will merge badly, and this is what the field currently measures. That guess failed every test I put it to. Two models trained separately on the same skill had almost nothing in common numerically and merged perfectly. Models trained for six times longer than usual merged fine. Language models trained twelve times longer merged better.
What broke merging, every single time, was something more human: the two models had been taught to answer the same question in two different styles. One says “yes” and “no”. The other says “1” and “2”. Both are correct. Neither is compatible with the other, and a merged model has to pick one and be wrong for the other parent forever after.
So the thing that gets inherited in a population of models is not a pile of numbers. It is a convention, a way of answering a kind of question. Most numerical differences between two models are harmless, in much the way that most of the differences between any two human genomes are harmless.
That gives a cheap test, and it works. Before merging two models, ask them both the same set of questions and count how often they disagree. That count predicted how much damage the merge would do. The standard measure, distance between their internals, predicted nothing at all.
A small society of models, over six generations
Finally I built a little society: three lineages of language models, each learning a new skill every generation for six generations, each able to merge with the others.
Lineages forced to merge every generation did well for three generations and then collapsed. Lineages allowed to refuse a merge (by checking whether the merged offspring was better than the unchanged parent, and keeping whichever won) never collapsed at all.
And the strangest result: merging a model with its own ancestor from three generations back was safer than merging it with a contemporary, in every single run. An ancestor lacks everything you have learned since, so it has something to offer, but it shares every convention you hold, so it cannot contradict you. Biologists have a name for a population that can breed with its own stored past. They call it a seed bank, and it is what a sediment full of dormant seeds does for a plant population.
Why this matters
If you train on synthetic data, budget in counts and not percentages. Almost every published guideline is stated as a fraction of the training set, and the fraction is the wrong unit. The same arithmetic turns up in work on data poisoning, where a near-constant number of documents does the job whatever the size of the model. It also means that the bigger your training set gets, the smaller the percentage you need, which is good news nobody is currently claiming.
If you merge models, averaging is the wrong default. It is the cheapest option and it destroys the specialist abilities you merged in order to keep.
Incompatibility is measurable in advance, and cheaply. Asking two models the same questions costs almost nothing compared with performing a merge and evaluating the wreckage.
And a warning about scale. These models are increasingly trained, evaluated and selected by other models. A population that is scored on how well it agrees with itself, rather than against anything real, converges confidently on a wrong answer. That is not speculation, it is what happened in my simulations whenever I removed the external check. Reality has to keep the right to say no.
Links and further material
The preprint, the code, every configuration file and every random seed are public. The entire study re-runs from one script.
