Burgess & Bolton Weekly Wrap-up | 11th Sept 2026

So, Ben, I want to start... by, I'm sure you've seen this, but there's this extraordinary kind of pronouncement that AI could kill all humans.

There's a 10% chance that AI could kill us all.

um and this was this came from uh i think it was uh evan hubinger who's anthropics uh lead scientist there and he's he's just resigned i think

yeah

and they they basically said that um In their opinion, I don't think it's based on any kind of data or research or anything like that.

But just in there, there's more than 10% chance that we could be wiped out.

So I was quite interested by this from, well, firstly, it was top news story this week, but also because it does have ramifications for insurance given probability, risk, all that type of thing.

I thought we might just have a chat about it.

and see what you thought um so my thought was immediately without any context a more than 10 chance of us wiping out humanity i mean where does that sit on the scale of other stuff that could wipe us out well

it's just a guess isn't it i suppose

I just had a quick look into this because the UK has a national risk register, which has 95 risks on it.

Yeah.

And the latest one, which is in July this year, kind of has AI in there, but not a specific threat so that the biggest ones are a pandemic, national power failure.

So that's not just sitting in the dark.

That's, you know.

ATMs, fuel not working, water, refrigeration, transport, all that kind of going down.

Hostile war, cyber, which we'll come on to later, cyber attacks on things like water and ports and hospitals and financial services and things like that.

the thing about ai is that it could accelerate that it becomes a a risk multiplier uh so it i think that's when it becomes kind of a big big risk um And yeah, I was quite interested just to think about that through a kind of the context of insurance.

Well,

yeah, I think there's a couple of things.

One is the background also, I think is to do with this concern.

I think that the, what you might call the, what I think are called a proprietary AI company.

So you chat GPTs and anthropic and companies like that have a, have a kind of natural natural way if you like of controlling bad behavior on on on ai so both the kind of intrinsic risks of things multiplying up but also the recent one that someone trying to develop a biological weapon using anthropic i think and these these open source ai models which are are being developed all over the place so in other words the open source are uncontrolled like the wild west and the propriety can be controlled you can see someone doing something bad on your system and cut them off or stop them doing it and so yeah so i think that would have implications i mean for humanity obviously but also i think in i mean rather trivially perhaps in insurance terms The question is, what can you ensure against or not in terms of this whole AI? Because what we've been talking about previously is things, again, maybe it's kind of seemed quite trivial, like the mistakes or the hallucinations.

But now we're talking about the purposes for which AI is used.

And that's a very different kind of conversation.