Predicting disaster is the single best-compensated position in technology commentary, and it is worth being precise about why — because the reason is not that the predictions are good.
I want to be careful here. There are people doing serious, technical, genuinely uncertain work on how these systems fail, and they are not who this is about. The difference between them and what I am describing is not the conclusion. It is whether the claim was ever capable of being wrong.
Why catastrophe pays
Three things stack, and they stack in the same direction.
Attention is asymmetric. A claim that something might end civilisation travels further than a claim that it will moderately improve document processing. This is not a flaw in audiences; it is a sensible heuristic about which errors are expensive. But it means the two claims face wildly different evidential burdens for the same reach.
Doom is cheap to produce. An accurate account of what a system does requires using it, testing it, finding its edges and describing them precisely. That is slow and the output is boring. A catastrophe scenario requires none of it. You can write one without touching the technology, and many are.
It cannot be falsified on any relevant timescale. This is the important one. If you predict a specific capability by a specific date, you can be checked. If you predict eventual catastrophe with no date attached, you cannot be — and every year that passes without it can be reframed as the danger drawing nearer.
That combination is unusual. Most positions that pay well carry a risk of being publicly wrong. This one has been engineered, whether deliberately or by selection pressure, to remove that risk entirely.
The tell
The distinguishing feature is not pessimism. It is unfalsifiability.
Serious safety work makes claims that could fail. This class of system will exhibit this behaviour under these conditions. This evaluation will show this result. This failure mode will appear at this scale. Those are real predictions. Some turn out wrong, which is precisely what makes the ones that hold up worth something.
The grift version never does this. It stays at a level of abstraction where no observation could contradict it. And when you press for specifics — what exactly, by when, measured how — the response is that demanding specifics is itself dangerously naive, which is a remarkable rhetorical position: the request for evidence is reframed as evidence of the problem.
The other reliable tell is the direction of the ask. Genuine concern about a technology's risks tends to produce demands for testing, transparency and access. The grift version tends to produce demands for restriction and consolidation, which happen to benefit the largest existing players. It is worth noticing when a warning about concentrated power resolves into a proposal that concentrates it.
What it costs
If this were merely annoying I would not spend words on it. It does actual damage, in three ways.
It crowds out real safety work. Attention is finite. Every cycle spent on unfalsifiable catastrophe is one not spent on the boring, tractable failures — systems that are confidently wrong, that fail unevenly across populations, that get deployed into decisions they were never evaluated for. Those are real, they are happening, and they are much harder to get anyone to care about.
It makes people worse at assessing risk. Sustained exposure to unfalsifiable alarm produces either paralysis or dismissal, and both are failures. The person who concludes it is all hysteria will ignore a genuine warning. The person who believes all of it cannot prioritise, because everything is maximally urgent.
It corrodes trust in the people doing it properly. When the loudest voices in a field are unaccountable, the field's credibility gets spent on their behalf. Researchers who make careful, checkable claims end up sharing a label with people who make unfalsifiable ones, and pay the reputational cost when the unfalsifiable ones look silly.
What I would ask for instead
Not optimism. I am not arguing these systems are harmless and I would be doing the same thing in reverse if I were.
I am arguing for claims with edges. If you think something bad is coming, say what, roughly when, and what observation would show you were wrong. Then keep the receipt and let people check it.
That is not an unreasonable standard. It is the ordinary standard in every field where being right matters — and the striking thing about this particular discourse is how successfully it has argued that the stakes are too high for it to apply.
The stakes are the reason it applies.
This piece expands on a video of the same argument. If you think I have this wrong, the counter-argument is genuinely welcome.
