How do you know when you're dealing with Claude that it won't become manipulative? It won't say things to you that are dishonest.

This is a constant, constant worry that we expend an incredible amount of energy on in how we train the models.

So it is a great question because it is a real risk.

And we actually publish reasonably unflattering studies in which we talk about the ways that if we did not apply specific techniques to make the models more honest, more harmless, more helpful, they would do exactly the kind of bad things that you talk about.

And the reason we publish those kinds of studies and that research is because we want everyone to know that A, you have to be critical of how the models work, and B, we are expending a huge amount of effort to learn the ways that the model can accidentally go around these safeguards to catch them and to make sure that we're building them in.

There's a lot of discussion around this concept of sycophancy.

And I think this is very important because to me, sycophancy is harder to stamp out than some of the more obvious ways that models can be bad, right? We originally had this framework.

So I think this is some kind of parable for the post-AGI age.

How do you know when you're dealing with Claude that it won't become manipulative? It won't say things to you that are dishonest.

This is a constant, constant worry that we expend an incredible amount of energy on in how we train the models.

So it is a great question because it is a real risk and we actually publish reasonably unflattering studies in which we talk about the ways that if we did not apply specific techniques to make the models more honest, more harmless, more helpful, they would do exactly the kind of bad things that you talk about.

And the reason we publish...

Those kinds of studies and that research is because we want everyone to know that A, you have to be critical of how the models work, and B, we are expending a huge amount of effort to learn the ways that the model can accidentally go around these safeguards to catch them and to make sure that we're building them in.

There's a lot of discussion around this concept of sycophancy.

And I think this is very important because to me, sycophancy is harder to stamp out than some of the more obvious ways that models can be bad, right? We originally had this framework, we called it HHH.

So I think this is some kind of parable for the post-AGI age.

Uh, my question is, what does Anthropic believe is the right balance between government regulation and AI innovation, and what do you wish governments around the world were doing differently?

Great question.

Um, so I th- I think this is an area where, um- I think the conversation about it has been s- you know, sadly as sort of is the case, I think, today just, just in the sort of political c- climate, it's like hard to have a nuanced discussion.

And I, I think that's a shame because I think it is a really nuanced question.

I think sensible regulation will need to be part of the story for artificial intelligence, right? It's a very different technology than any other technology that's ever been built before.

And even the last generation of technologies probably could have benefited from slightly more regulation, in my personal opinion.

Um, that being said, we're not blind to the fact that, like, you need to have some room to maneuver as a company to be able to try things if you want to come up with the next generation of incredible products that people wanna use and that are adopted.

Uh, what do you think individuals should actually be doing to protect their privacy?

Do you think there's truth to the idea that some businesses may be overspending on AI tools in this early experimental phase and they may cut back as they move out of that phase?

So I think two things are true at once here.

I like that you said experimental because I think the reality is it feels like these AI tools are so powerful and so capable, which is true.

And comparing them particularly to even two years ago when I was on stage at Bloomberg with Dario, I think the models have just come so far in what they're able to do and the economic value that they provide.

But I actually think there's a lot more distance to go still for what the models will be able to do two to four to six to eight years in the future.

And I would suspect that we will see some experimentation in different fields to understand how do you get the most value out of these models for what particular workflows.

I think the second thing that's true is how are choosing to use AI will change.

Do you agree with that? And if so, what's the solution? Is it something like more taxes? We talked about basic income, job retraining, and how do you pay for all of that? Sorry.

Okay.

But, you know, I, I read that book, and I think that was part of...

You know, again, it felt like an, in some ways, an easy decision for us to say, "We just don't know enough about what AI is going to do to kids." It's not to say that there couldn't be great benefits for kids using AI for learning, but that needs to be done with an adult in the room.

It needs to be done with a human in the loop.

And I, you know, I think that, that the reason we've always taken this perspective is we just, we...

People deserve autonomy.

To d- to, to Dario's point, people shouldn't feel like artificial intelligence is happening to them.

It's gonna be a bunch of different things.