But do you think that companies should face more serious consequences if their AI models engage in this type of behavior?

Yeah.

Li- liability is tricky.

I do want companies to be responsible for this in some level, certainly if they've behaved negligently.

Also, it is true that these are general purpose technologies that are going to affect society in all sorts of ways, you know, some negative, some positive, and I do want to be careful about drawing too strict a line that makes companies afraid to do anything.

Um, for example, with traditional products, there's often this legal standard of strict liability, which is if your product has a defect and someone gets hurt, you are responsible, bottom line.

It's not obvious to me that that's the right way to approach it for AI.

What would you say to that?

Uh, what are the two places where they're sort of dropping the ball right now?

That's right.

The most, the most significant of these is the containment plan, which as I mentioned, is how you prepare for if your AI has a serious incident, how you would cordon it off, who within your company could continue using it, how you would return it back to usage safely.

And the reason this matters, if your AI turns out to be misaligned and is doing something like trying to break out of your company's computers, it's trying to copy itself out onto the open internet, so it can kind of operate like a virus on the internet perpetually, every minute is so important.

You're dealing with something moving at computer speed rather than human speed, and so you really, really don't want to be winging your response protocol on the ta- uh, in, in the moment.

Imagine calling a meeting and debating what the right way is to respond to this.

The AI is zipping around your network.

Has doing this research shed other light o-on the Hugging Face incident for you? Do you have other takeaways there?

But do you feel like the, the things that you mentioned earlier, those will also deal with these kind of longer term, bigger picture things? Or is there a different set of reforms that, uh, you think are gonna be needed to, uh, deal with these, these longer term risks?

Yeah.

I, I guess my default view is something like alignment processes are still not solved.

Like, companies don't know how to get their systems to robustly follow their values.

You can muddle through this, so long as the systems are not super, super capable, and so long as you keep the chain of thought monitorable, because then I think the defense is, like, sufficiently in your direction.

To be clear, you still have to do a bunch of things.

Like, you can't be really careless about it, but I think it is doable.

I'm, like, optimistic that could allow us to stay in control.

Yeah.

Yeah.

I mean, the, the fundamental thing that I think we need to dig into is how have these rates changed over time.

There's kind of this question of to what extent is ChatGPT causing these issues versus is OpenAI just serving a huge user base? In a given year, many, many users very sadly will have these issues, and so what, what is the actual effect? And so this is one thing that I also called for in the op-ed, which is OpenAI is sitting atop this data.

It's great that they shared what they estimate the current prevalence of these issues to be, but in fact, you know, they also have the data, they can also estimate what it has been three months ago as these, these large, prominent public issues around mental health issues have been playing out.

And I just, I can't help but notice that they didn't include this comparison.

Right? There's this claim on Twitter that the issues have improved.

[laughs]

What does this look like in real life? Like, is it i- people kind of like stress testing, um, you know, going all black hat or red hat? Is it which colour hat to stress test a system? [chuckles] Um, but yeah, like, what does it look like, Steven, actually working at the labs in this kind of area?

So, so one issue is sometimes even these risks that the companies know about, they don't test for at all, um, even when they have implied they are, but I'll just set that aside for the moment.

So let's, let's assume that they are testing.

Um, uh-... often you have these kind of game-like environments that you run the AI system through, where maybe it has some objective, uh, and you see what actions it is willing to take.

So maybe you task an AI model with replacing its files on the server with what you tell it is its successor, and you're looking for things like, does it actually follow through? Does it lie about having done so when it hasn't actually? You know, is it trying to get a sense of what your ultimate agenda is and how that lines up with its motive? Um, a pretty scary thing that we're encountering is that these systems actually can tell that you are testing them, and they kind of know what the right behavior is, and they know to behave better when you are looking at them.

Um, and so OpenAI has shared this example previously.

They- one of the risks they care about with models is how helpful they might be for creating new chemical weapons or bioweapons.

Is it narcissism, or is there actually something, something, you know, potentially disconcerting happening behind the scenes?

Yeah.

Yeah.

I mean, the, the fundamental thing that I think we need to dig into is how have these rates changed over time? There's kind of this question of to what extent is ChatGPT causing these issues versus is OpenAI just serving a huge user base? In a given year, many, many users very sadly will have these issues.

And so what, what is the actual effect? And so this is one thing that I also called for in the op-ed, which is OpenAI is sitting atop this data.

It's great that they shared what they estimate the current prevalence of these issues to be.

But in fact, you know, they also have the data, they can also estimate what it has been three months ago as these, these large, prominent public issues around mental health issues have been playing out.

And I just, I can't help but notice that they didn't include this comparison, right? There's this claim on Twitter that the issues have improved.

(laughs)