The precautionary principle is a trap

The precautionary principle has an enormous advantage over its rivals: it sounds like the responsible position. Nobody has ever been called reckless for demanding more evidence before proceeding. It costs nothing to say and it makes the speaker sound like the adult in the room.

It is also, in its strong form, close to useless as a decision rule — and worse than useless when the thing you are being cautious about is itself the safer option.

The strong form is unfalsifiable

The strong version says roughly: where an action might cause serious harm, the burden of proof falls on whoever proposes it, and they must demonstrate safety before proceeding.

The problem is the word demonstrate. You cannot demonstrate the absence of a harm. You can fail to find one, across a set of tests you thought to run, over a period you were willing to wait. That is not the same thing, and it never becomes the same thing no matter how long you look.

This means the standard can always be reasserted. Whatever evidence is produced, the question "but have you ruled out the possibility that…" remains available, forever, at no cost to the person asking. A rule that can never be satisfied is not a high bar. It is a veto wearing a lab coat.

Inaction is also an action

The deeper flaw is that the principle is stated asymmetrically. It weighs the risks of doing something against an implied baseline of zero risk for not doing it.

That baseline almost never exists. If a diagnostic system would catch a fraction of cases that are currently missed, then every month it is delayed has a cost measured in missed cases. That cost is real, it is ongoing, and it is invisible in a way the alternative is not — because the people harmed by the delay are statistical, and the people harmed by a deployment failure have names.

We are extremely bad at this comparison. A visible, attributable harm caused by a new thing generates outrage, inquiries and rules. An identical quantity of harm caused by continuing to do it the old way generates nothing at all, because there is nobody to blame and no moment when it happened.

The precautionary principle does not correct for that bias. It formalises it.

Who it actually protects

Follow the incentives and the pattern is consistent.

A requirement to prove safety in advance is a fixed cost. Fixed costs are trivial for large incumbents with regulatory affairs departments and prohibitive for small entrants without them. Applied to a fast-moving field, "prove it is safe first" reliably functions as a moat, and the organisations subject to it are frequently the ones most enthusiastic about it — which should tell you something.

This is not a conspiracy. It does not need to be. It is simply what happens when you impose a cost that scales with a company's ability to absorb it, and it happens whether or not anyone intends it.

The result is a market with fewer participants, less competitive pressure, and slower iteration, defended in the language of public safety. It is possible to want careful oversight and still notice that this particular mechanism produces concentration rather than care.

What I am not arguing

I am not arguing for building whatever you like and finding out afterwards. That position is genuinely reckless and I have no interest in defending it.

Some domains genuinely warrant proving safety in advance, and they share a specific shape: the harm is severe, irreversible, and hard to detect until it is too late to act. Nuclear containment. Novel pathogens. Anything where the feedback loop is longer than the damage.

The mistake is generalising from those cases to everything. Most decisions are not irreversible. Most harms are detectable. Most systems can be rolled back. For that much larger category, a rule designed for irreversible catastrophe is simply the wrong instrument, and applying it anyway is not caution — it is a failure to distinguish between kinds of risk.

A better question

The useful version is not "is it safe?" — nothing is — but a comparison the strong form refuses to make:

Compared to what? Not compared to a hypothetical world with no risk. Compared to the status quo, which has its own failure rate, currently being paid by someone.

How fast would we find out? A system whose failures surface in days is a fundamentally different proposition from one whose failures surface in a decade, even at identical severity. Reversibility is doing more work here than probability.

Who carries the risk, and did they choose it? A risk accepted by the person who benefits is a different moral object from one imposed on a third party. This is the question the strong form is actually reaching for, and it would do better to ask it directly.

What does waiting cost, and to whom? If the answer is "nothing", wait. It is rarely nothing, and the fact that the cost is diffuse and unattributable is not a reason to score it as zero.

None of those questions are as satisfying as "prove it is safe first". They do not resolve into a slogan, they require you to estimate things you would rather not, and they can be answered wrongly.

They also have the significant advantage of being answerable at all.