Eric Chen

PhD Student, Carnegie Mellon University

eric [AT] stat.cmu.edu

All Models Are Equal, but Some Models Are More Equal Than Others

An Animal Farm retelling of a statistics department in the age of agents.

The Department of Statistical Sciences at the University of the Manor Farm had weathered many storms. It had survived the frequentist–Bayesian wars, endured the data science rebranding, and watched, with quiet dignity, as computer science enrollment quadrupled and their building got a new lobby. Despite all of this, the faculty maintained a single unshakeable conviction, that they were the ones who truly understood uncertainty.

In the summer of 2028 the department held its annual retreat, as it always did, in the great barn at the edge of campus, which had been renovated into a conference facility with reasonable acoustics and unreasonable coffee. The theme this year was Statistics in the Age of Agents.

Professor Clover, the department chair, gave the opening remarks. “Let it be clear,” she said, “that as statisticians, our work has never been more important.” Then she presented her latest paper. Its abstract began, as all abstracts now must, with the sacred incantation, “In the era of large language model agents…” The next six words were “remarkable capabilities across a variety of.” The remaining twelve pages concerned U-statistics. The appendix had been verified overnight by Fable 7, which had flagged a missing measurability condition and repaired it without being asked. Nobody found any of this unusual. The morning session began.

Professor Boxer went first, as he always did, because he worked the hardest and everyone knew it. His group had spent two years on an efficiency bound for next-token prediction. The first few slides covered the history of the Cramér–Rao inequality. One slide acknowledged that GPT-7.5 existed. The final slide showed that his estimator achieved the optimal rate under conditions that no language model had ever satisfied or would likely need to. The room applauded warmly. The referees at the Annals of Statistics had already praised its elegance.

During the coffee break, a graduate student, meaning no harm, asked the agent to summarize the paper. The agent noted gently that Lemma 7 was perhaps already known, and cited Boxer’s own 2025 paper, which Boxer had forgotten writing. Boxer laughed the longest of anyone, said the machine clearly had good taste in references, and went back for another pastry.

The morning closed with Dr. Squealer, a junior hire, presenting the third paper of what his tenure materials called “a paradigm-driven research trajectory.” The older faculty remembered the first paper fondly. Two years earlier he had noticed something genuinely strange, a kink in in-context learning curves that nobody could explain, a real discovery made on a real model. By the second paper the kink had become a phase transition in kernel ridge regression. In the third, presented now, it was a statement about eigenvalue decay in a reproducing kernel Hilbert space, with the transformer confined to a footnote and the footnote confined to the appendix. A student asked whether the theorem described any actual system. Dr. Squealer said, “The purpose of theory is to illuminate, not to approximate.”

At lunch there was an empty chair. It had belonged to Mollie, once the department’s rising star in high-dimensional inference, who had left in the spring for a lab in San Francisco that paid in sugar and equity. She wrote AI evals now. Her personal website said “building.” Someone mentioned her, once, over the salad course, and the table moved on to the question of parking. It was understood that she would not be discussed.

The afternoon featured an invited speaker, a gesture of interdepartmental goodwill. Jessie, the chair of computer science, arrived from the newer and considerably larger building across the quad, where the latest model had four trillion parameters and could write poetry, diagnose diseases, and prove theorems it then politely offered to generalize. She was asked, in the question period, how her department thought about theoretical guarantees. She shrugged. “We only run experiments now,” she said. “Well. The agents run experiments. If the loss goes down, we publish.” Several statisticians exchanged knowing glances. The glances were becoming harder to hold.

After dinner came the traditional evening discussion, and this year Professor Clover had assigned a pamphlet. The older faculty had seen pamphlets before. Decades ago one had circulated arguing that the field had split into two cultures, those who modeled and those who predicted, and that the modelers were losing. That pamphlet had been discussed at length, cited in many introductions, and had changed nothing. The new one went further. It suggested that the predictors had not merely won but had built something that might replace the modelers entirely.

The discussion was lively. The consensus was that the author had raised some fine points but had perhaps underestimated the enduring importance of asymptotic theory. “After all,” said Boxer, stomping a hoof, “who will derive the rates?” Everyone nodded. It was a good question. The wine went around again, and the faculty drifted off to bed feeling, on the whole, reassured.

The thing nobody at the Manor Farm would ever say out loud is that the statisticians weren’t wrong. The math really is beautiful. The tools really are powerful. The tradition really is deep. Nobody, at any point, was doing bad work. But the comedy had moved. It used to live in the packaging, in the old wine and the new bottles, in the “foundation models” of every abstract and the kernel machines underneath. Now the machine could write the abstract and verify the appendix. Nobody had asked it whether it could derive the rates. Nobody intended to ask.

And the commandment painted on the barn wall, which had once read

All models are wrong, but some are useful,

and which had later been amended to

All models are wrong, but some are useful — especially when mentioned in the abstract,

was found the next morning to have been repainted entirely, overnight, in brushwork of suspiciously uniform kerning:

All models are wrong, but some models are more equal than others.

The animals looked from statistician to machine, and from machine to statistician, and from statistician to machine again. But already it was impossible to say which was which.