International AI Safety Report

Going a bit broader now.

So I also noticed that you were involved in the International AI Safety Report for this year.

What is your perception of the AI safety? So we're jumping a little bit broader.

My perception of AI safety, not of the report, but of the general question.

But how likely is it then that we really end up in a scenario to lose control to AI systems? In a large survey of almost 3,000 AI researchers, the median researcher put the chance of an extremely bad outcome at about 5%, which is still an estimate, but this is not deterministic.

And the International AI Safety Report says that AI agents do raise the risk, but that overall today's systems don't have the capabilities to cause a complete loss of control.

The named blackmail result much as the other incidents came from a constructed scenario and if you want to pin it down you could say that yes in a narrow sense AI went rogue this summer but none of the system wanted anything for itself rather they were chasing a dedicated task.

So AI depends on the infrastructure the energy and the human initiation to and it can only escape human control if humans let it do so.

A, a developer cannot reliably recall every copy of a model once its weights are publicly distributed.

The International AI Safety Report explained why those releases create distinct challenges for both monitoring safeguards, but also recall of that program as well.

So that makes testing before release especially important.

An emergency shutdown mechanism is one layer of protection, but it cannot replace careful deployment decisions or restricted permissions or monitoring or even, and especially, human accountability.

That's a bit mean because most of them are also, I think, pretty scared about what's going on, and, um, I'll explain why in just a minute.

Also, Yoshua Bengio, the famous Canadian computer scientist and chair of the International AI Safety Report, he has also entered the chat, as they say, this week.

Uh, he wrote a piece where he argues that in the midst of all these agent hacks, there's a quote, unquote "dangerous myth" that has gained traction.

And his argument is that these incidents, you know, these, all these agents breaking out of their sandboxes, going rogue, et cetera, are not really a cybersecurity problem, and they're actually a result of just flawed training of these systems by the humans that are creating them.

The European Union, China, and the G7 have all published initiatives that emphasize transparency, human control, and risk response.

And the International AI Safety Report calls for a multi-layer defense system that establishes safeguards within AI companies, such as filters on what prompts AI will accept and what kind of information it's willing to spit out, as well as protective measures outside of them.

Those would be things like DNA screening to flag bioweapons, established protocols for cyber attacks, and media literacy programs.

By the way, AI companies are not required to publish safety frameworks.