The Invisible Graveyard

During the Second World War, the Allies wanted to know where to add armor to their bombers. The natural approach: study the planes that returned from missions, map where the bullet holes were concentrated, and reinforce those areas.

A statistician named Abraham Wald looked at the same data and reached the opposite conclusion. The planes that came back were not a representative sample. They were the survivors. The bullet holes on a returning plane mark the places an aircraft can be hit and still make it home. The planes that took damage to the engine, the cockpit, the fuel line, never returned to be studied at all. Wald's recommendation was to armor the areas with the fewest holes on the surviving planes, because those were the spots where damage was fatal.

The data you can see is systematically the wrong data, precisely because the failures you most need to learn from are the ones that never come back to be measured.

I think academic research has the exact same blind spot, and almost nobody is looking at it.

When an institution thinks about research quality, it studies its returning planes. The papers that got published. The grants that got funded. The projects that made it across the finish line. Every quality initiative, every audit, every metric is built from this visible, survivor-biased dataset.

Nobody studies the invisible graveyard. The dataset collected eighteen months ago that never became a paper. The clinical observation a resident noticed but never wrote up because they did not know how to frame it. The project that stalled for six weeks at a methodology question nobody was around to answer, and quietly died because momentum, not ability, ran out.

This is not a quality problem. It is a survivorship problem.

Across the last four letters, I have written about methodology debt, about designing projects that compound, about authorship clarity, about statistical traps. Every one of those is a place where a project can take a hit and survive if the researcher gets help at the right moment. But the institutions and tools built around research almost universally focus on improving the planes that come home: helping a finished draft get published faster, helping a strong paper get into a better journal. Almost nothing is built to catch the project before it goes down silently, at the moment of stuck, unanswered uncertainty that never produces a record anyone can study afterward.

That invisible graveyard is, I suspect, far larger than the published record it sits beside. And because it is invisible, it has never been treated as a real institutional cost. You cannot put a line item on a budget for "projects that died of an unanswered question at 11pm." But the talent, the data, and the months of work inside that graveyard are real, and they are gone.

If that is true, the most valuable thing a research support system can do is not make the published record better. It is to be present at the exact moment a project is about to go down quietly, when a researcher is stuck on a question with nobody to ask, and the difference between finishing and abandoning is simply whether help arrives in the next ten minutes or not at all.

I do not think this is a tooling problem in the conventional sense. It is closer to an armor placement problem. The question is not "how do we help our best projects publish better." It is "where, specifically, are projects currently taking fatal hits that we cannot see, and what would it take to reinforce exactly there."

Don't let it go quiet →

We have spent a great deal of effort, as a field, studying the planes that come home. The bigger opportunity, I think, is finally looking at the ones that did not.