What is the risk of AI at that sort of nation state level where they have a lot of capability already?

No country really openly admits having a biological weapons programs, although there are intelligence reports that some countries do.

The goal of creating something entirely new from scratch, that's going to remain quite a challenge because the AI models themselves have to train on existing data.

And so at the moment, I think the greatest concern is innovations, changes to existing viruses or pathogens.

but in ways that are somewhat unpredictable or outside the usual design space that humans would be working in or that nature has worked in before.

I can give you an example that received a lot of attention this summer, which is a group of scientists, very talented scientists and engineers, Their goal was to use very powerful biological AI models to design novel bacteriophage viruses.

These are viruses that infect bacteria.

Tom, in your view, what do we need to do? What do we need to bring in now to protect us as much as that's going to be possible?

I'm imagining you saw some of those news reports.

Yeah.

I mean, it's been described in a lot of different ways in the press.

There-- A couple of things have happened in the last month.

One was the report from Anthropic, the, one of the frontier model companies, that there had been...

They presented five cases that had occurred in the last six months, I believe is the timeframe, of different entities trying to use their model to do things that they ultimately refused, Anthropic refused to honor, and that Anthropic itself thought were quite dangerous.

I think some in the community wouldn't go all the way to say, and I think I-- it's not clear that those things were being-- those prompts were being used to make biological weapons, but what they were seeking was information that would create, in some cases, more dangerous pathogens than currently exist in nature or in laboratories, which could be used deliberately or even accidentally, even if they're doing this work for some purpose that was benign, if it got out of the lab, could cause potentially very serious problems.

First of all, what do you even mean by raising the ceiling of harm? What, what are we worried about here?