A Design Is Not a Pathogen

Here is the nightmare, stated plainly.

Someone with no particular training sits down at a laptop, asks an AI the right questions, and walks away with everything they need to build a virus that could start a pandemic. No lab pedigree. No state program. A chatbot and bad intent.

It is a good nightmare. Vivid, frightening, and exactly the kind of story that makes policy.

The question is whether it’s true.

In late 2023, an executive order told the government to find out, and the National Academies convened a committee to assess what artificial intelligence actually does to biological risk. I served on it. We spent months on a single, deliberately narrow question.

Not whether AI is scary. Scary is not a finding.

The question was how much AI changes a risk that already exists.

Making a biological weapon did not become possible in 2023. It has been possible, for the wrong people with the right resources, for a long time. So the only useful question about a new tool is what it adds on the margin, beyond what a determined person could already pull from a textbook, a search engine, a graduate education. We even gave that margin a name. Delta AI. The uplift.

Measured that way, the fear gets more specific, and more manageable.

The uplift is real, and it sits almost entirely at the front of the process: ideation and design. AI is genuinely good at proposing sequences, generating hypotheses, sketching molecules that might do a thing. Some of that matters. A model can redesign a toxin to slip past the screening meant to flag dangerous DNA orders — a real problem, though a local one, not a pandemic.

But designing is not the dangerous step.

A design is a file. It sits on a screen. The risk only becomes real when someone crosses from the screen into the physical world and actually makes the thing, and that crossing is where biology stops cooperating.

No tool we examined can design a working virus from scratch. The datasets you would need to train one do not exist. And the step everyone fears most, turning a sequence into a living, spreading pathogen, runs straight into a wall that has nothing to do with how good the model is.

The report puts it in one flat sentence I keep returning to: the bottleneck in physical production is not affected by the capabilities of AI biological models.

A better model gives you a better blueprint. It does not pour the concrete.

That is the line I would attach to most AI-and-biology fears. The model is not the bottleneck. The physical world is. And the physical world is indifferent to how impressive your model got this year.

Here is where it would have been easy to write one of two essays.

The alarming one: it’s coming, ban it now, imagine the worst and legislate against it.

The dismissive one: it’s hype, the machines can’t do it, relax.

We wrote neither, on purpose.

We said, plainly, that no AI today can design a pandemic virus from nothing, and, in the same breath, that if that ever changes it is the single most important thing to watch. We insisted on separating capability from intent, because the same models that might design a toxin are the ones designing vaccines and antibodies, and you do not get to strangle the second to prevent the first. Capability is not a crime.

And we tried to build something better than a guess.

The honest worry was never today. It’s the slope. Data is the thing to watch, because models follow datasets the way protein-folding models followed the Protein Data Bank, which makes the data collected now the leading indicator of the capability that arrives later. The physical wall could get lower too, if automated labs keep making the build step cheaper. None of that is here. All of it is worth watching.

So instead of writing rules against an imagined future, we argued for a strategy that adapts to a real one. Decide the thresholds in advance. If this kind of dataset gets assembled, monitor for that capability. If that capability appears, watch for this output. Governance that moves when the evidence moves, instead of freezing on the day the report went to print.

I have said a version of this before. In 2023 I told a Senate forum that we cannot prove an absence of risk, but we can build empirical ways to measure change over time. That is the whole posture, compressed. You do not get certainty. You get instruments, and the discipline to keep reading them.

It is also, if I’m honest, the same reason I write novels.

The report is a careful measurement of the present: how far we are, today, from the edge. The fiction is the other half of the same discipline, a careful imagination of what it looks like to go over. Echoes of Tomorrow is, among other things, the story of the moment a design finally does become a pathogen, and the institutions built to catch it turn out to be a step behind. The report measures the distance to that edge. The novels jump off it, so we can watch the fall from the safety of a page.

Both are refusals of the same thing: the false comfort of certainty, in either direction.

The nightmare I opened with is not wrong. It’s imprecise. Someday the datasets may exist. Someday the build may get easy. The right answer to that future is not to panic into it or to wave it away, but to name in advance what we are watching for, and to keep measuring the distance as it closes.

A design is not a pathogen.

Not yet. That “yet” is the whole job.