Tell me how that's changing right now.

Effectively, the last era of security, from my perspective, started with the Microsoft, the Bill Gates, trustworthy computing memo.

in the early 2000s.

And that is when we started designing how we do software security.

And these big companies, Microsoft and Oracle, and then later Amazon and Salesforce, and these folks started hiring ex-hackers.

And I was part of that generation of the ex-hackers who were like at DEFCON and Black Hat, got hired by the companies to go break their stuff.

And we started coming up with Responsible Disclosure, Bug Bounties, consulting companies.

So are we going to go through a period where everything is so much worse and then it's perfect forever? Or how does the cost manage itself out into balance?

And so we might as well get started with the title of this class, which is what is trust and safety engineering?

Thanks, Pax.

Really excited to be able to offer Computer Science 152 Trust and Safety Engineering to students who are not able to come to the Stanford campus.

And this is going to be an exciting hybrid class for both students on campus and off, and ability for working professionals to learn about the space and to collaborate with current Stanford computer science students.

But that's a great question.

What is trust and safety? Trust and safety is a little bit of a industry term for studying how people abuse the Internet to cause harm.

The term comes from the tech industry starting about maybe 20 years ago, it started to become in vogue, especially with payment companies, auction sites and such.

Can we dig a little bit deeper then into the learning goals of the class and what students will get out of it or what you hope students will get out of it at the end of the course?

When we hear something like that, how are we supposed to evaluate that kind of claim? What is the state of the internet right now?

Okay, so to say the internet is going to collapse, I think is hyperbole.

The fundamental infrastructure of the internet is holding up fine, right? Like the internet, the routers, the switches, there have been bugs in the fundamental systems that move traffic.

But those have been fixed.

I think what is happening, however, is we have lived through a period where you have been able to assume that if you use a service that your data is going to be safe.

That assumption was never correct, it turns out, right? These systems were never safe.

It's just that we lacked the number of skilled attackers to take advantage of the fundamental insecurity of these systems.

But what should people be doing right now in this moment on a personal level to shore up, make sure they're as ready and prepared as possible for this more insecure version of the internet?

It must be frustrating for someone such as yourself who spends all their time thinking about this stuff, right?

Yeah, it's driving me insane.

Effectively, there's a bunch of people in Western capitals who are looking for any excuse to ban the use of any model that is derived from a Chinese model.

So not just Chinese models themselves, but anybody who has distilled or utilized Chinese models.

in any kind of way to not let you use them, to not let you use code written by Chinese models.

We've seen the start of this in the United States.

I think that would be a spectacularly stupid decision.

um alex what what's your vibe here on what's actually going on because is it token theft is it distillation i mean i feel like it's going to be both right because you you steal the tokens you pump them through something some sort of llm router you keep the transcripts and that becomes useful for distillation later but i feel like the primary motivation here is probably just to resell stolen tokens

So how do you actually think so for top of mind for us is how do we actually scale the security program to deal now with the 10x, 100x, 1000x things that are attacking us things that we're finding while then also internally making sure that we're not slowing down the development of the AI product so it's it's a is go fast, but also move fast at the same time.

Yeah, so I mean, I'll try to give you a short answer.

I wrote a long thing in my substack, which is like alexthomas.com.

The open AI attacks against Huggy and Face are the start of something I talked about earlier this year, which is we are entering a valley of pain, which is almost all of the benefits from AI over the last three, four years have accrued to defenders up to this point.

And now attackers are actually getting benefit.

And they accrued to defenders because defenders have money, right? And attackers, funny enough, had, you know, especially like the financially motivated attackers, funny enough, have relatively less money, right? They don't have the billions of dollars for R&D that the huge cybersecurity industry that you see here at the RSA conference have.

And so now that we have these incredibly capable offensive models, and especially the open-weight models, the attackers are going to go hog wild.

What would you say is something that startup founders are getting it right, and what are the startup founders getting it wrong in terms of security?

It was found with AI and what this has led to and what it presages for what comes next.

Yeah, first to set the stage, we'll talk about PHP first, but this has been about the craziest 30 days or so in the history of our industry of cybersecurity.

And so PHP is what's catching the headlines, but for people to do this professionally, there's actually lots of stuff that's happened just in the last couple weeks.

So the PHP bug specifically, like you said, PHP is a very popular language for writing websites easily.

So very few kind of enterprise quality websites are written in PHP.

The interesting counterexample is Facebook, although Facebook doesn't actually use real PHP.

They use a typed version of PHP called Hack.

Goodness

[laughs]

So like most, most of, most of our economy is not the Fortune 500.

It is very important small to medium enterprises that have hundreds to sometimes thousands of employees who are very, very important.

They are worth millions to hundreds of millions and sometimes low billions of dollars.

But they are not big banks.

They're medium-sized banks.

They are the local bank of town.

Mm-hmm.

Your thoughts?

Okay, so I, I wanna first dismiss some conspiracy theories that you did not say but that come up here, um, before we get into it.

So first, this is not a marketing thing, right? Um, that has been, uh, the kind of cynical culture war, uh, and it's from both sides.

There's a horseshoe theory.

You know, there's a horseshoe thing happening here from the far left and far right.

Like, this is a real problem, uh, and it is not just marketing from these companies.

I, I'm not sure I'm ready to get into like a liability discussion a- at this point.

Your thoughts on James Coxon, or Jacob Coxon, excuse me.

And why next year?

So y- the foundation labs, the OpenAI and Anthropic especially, and, and maybe Google.

Google hasn't done a big release for a while, so we're not totally sure what, what they've got.

But at least OpenAI and Anthropic are something like three to six months ahead of their Chinese competitors.

Um, this year we have seen these releases of Chinese open weight models.

Uh, first GLM 5.2, Kimi K3.

Now we're seeing releases of DeepSeek models, uh, that are very, very close to the capabilities of, uh, the American models.

Right

Right

... a cyber-aligned model.

This could be like the f- Yo, OpenAI makes these cyber-specific models.

Like they have this 5.5 cyber.

This could be 5.6 cyber, right? So it could be something that's specifically tuned to be good at hacking things.

So we shouldn't be shocked that it's good at hacking things.

But the alignment issue is...

K3

So how did the technology manage this? My colleague Rob Young spoke to Alex Stamos.

What we found out today was that OpenAI was evaluating one of their new unreleased models.

So we don't know the name of it yet, but this is a model that they were testing internally.

They were running it through a standard security test, and they told the model, "Go do this standard test." The model figured out that the answers for this test were stored at Hugging Face.

Instead of just doing the test, what the model [chuckles] decided was it was going to cheat and go find the answers.

And so it did a couple of things.

First, it needed to break out.

And so are you wowed by the technology's capabilities or horrified at the lack of human control of it?

Where are we on the use of AI from a defensive posture trying to fight back against, uh, incoming risks?

Oh, God.

If you wanna talk about messes, that is a humongous mess.

So the White House created a huge disaster for the US AI industry a couple of weeks ago when they banned Anthropic's Fable model.

Fable was, when it came out, the most powerful model available to Americans.

It was only out for a week, at which point, uh, Amazon had some complaints that they sent over to Anthropic about some capabilities they thought Fable should not have.

Now, unfortunately, the White House got a, a hold of this report, totally overreacted, and then on a Friday afternoon, banned it with only 90 minutes of warning to Anthropic.

Is there any way for the US government to even level the playing field if it does kind of take some models offline, study them, review them, make sure that they're safe, you know, dribble them out to certain companies? I mean, China can basically do whatever it, it wants, the Chinese companies, right? There's nothing the US can do to, to prevent that.

yeah it's a bit strange um you know when it comes to like the federal response and everything i mean sisa just put out a notice like they're changing the the patch guidance for for agencies down to three days now yeah um which you know i guess in the world that we're talking about where zero days are coming out basically before anyone even has a patch i mean how does this reflect because you can sit and try to patch but it seems like we need to be thinking out at least six months or 12 months or 18 months like what does this mean for for if you're running a you know a security program at a corporation

yeah so i mean that's amazingly fast if you're thinking about running a traditional infrastructure so I think that's for the calves, right? So those are like known exploitive vulnerability.

So that's just not like normal patch Tuesday stuff.

That's stuff that's definitely being exploited.

Um, where I see what I see that meaning is like what they're really saying is you can't be hosting your own.

You can't have any state that you own yourself, right? That you have to be on containers.

You have to be on the cloud.

Yep, definitely.

What, what happened?

Right.

So, uh, just for recap, uh, for everybody, uh, last Friday, uh, what happened was the White House became aware of research that Amazon had did, had done.

Um, it is not unreasonable for Amazon to do this work, right? So, uh, Amazon hosts, uh, Anthropic's models in the, in what they call Bedrock.

Uh, this includes in all of the classified spaces, so if the NSA is using Anthropic's models, it is running in AWS secret cloud or top secret cloud.

Uh, and so Amazon is doing security testing.

I expect this was part of getting ready to host Fable in Bedrock, and, uh, found some things that they wanted to complain about with Fable's protections that make Fable different than Mythos.

[laughs]

government thinking it can regulate AI, and what does that do to the AI industry and the world?

It's a complete disaster for the American AI industry.

This is a complete and total disaster.

The other thing that happened yesterday is a model called GLM 5.2 was released by Zeta AI, which is one of the leading Chinese AI labs.

This is an open weight model.

It is MIT licensed.

I mean, anybody can download this model and use it for whatever purpose they want.

Get used to it, folks.

What do you think?

think?Uh, what do I think? So like you said, uh, we have this letter at freefable.org.

So it's not just what I think.

We have over 150 signatories, as you pointed out, uh, a lot of people who really know what they're talking about, not just academics, but CEOs, CTOs, chief scientists at AI companies, large tech companies, uh, founders of AI companies saying they do not believe there is a real risk from Fable.

Why do they say that? So a number of us have seen the reports that, uh, Anthropic have put out, uh, refuting Amazon's research.

Now, I don't think Amazon has done anything wrong here.

Uh, Amazon has a shared responsibility model with Anthropic where Amazon hosts Anthropic's models in a number of situations.

Mm-hmm

government thinking it can regulate AI, and what does that do to the AI industry and the world?

a complete disaster for the american ai industry this is a complete and total disaster the other thing that happened yesterday is a model called glm 5.2 was released by zeta ai which is one of the leading chinese uh ai labs this is an open weight model it is mit licensed i mean anybody can download this model and use it for whatever purpose they want they can retrain it they can redistribute it you can actually use this model in the commercial product if you like.

It is within a couple percentage points of Opus 4.8 in its performance and a number of coding benchmarks.

It is amazing.

So the same week that the United States government kneecaps one of probably the leading US AI lab, the Chinese release a model that is almost as good as the best model you can currently get from that lab.

I don't think you can come up with a better storyline for why this is incredibly stupid.

The idea, the whole action is based upon the premise that American labs are light years ahead of our Chinese competitors and will always be so.

what do you think of schneider's argument what do you think of schneider's argument that it's all silly because the the pandora's box is open if this model isn't the most powerful the next one will be get used to it folks

What does explain then this plead overstated reading of Amazon's findings? What does it sort of suggest to you went on behind the scenes here?

So first off, I want to say, I don't think Amazon did anything wrong in doing the research.

Amazon runs Anthropix models.

So they have a shared responsibility model here with Anthropix.

When they are hosting Anthropix models, they have to do some of the security and safety work themselves.

That includes in any situation where the US government is accessing anthropic models is my understanding that that is running on Amazon hardware.

If the NSA is using anthropics models, if they're using mythos, that is running at AWS top secret cloud, for example.

So I want you as a kind of CISO, I think you've just answered this in many ways, but in more basic terms for any listener that might not be thinking certainly about cybersecurity in the way that you are, what could a security officer at a hospital or a utility or another company do with Fable last Wednesday that they can't do today?

I'm really excited to be here and be at Socket and excited to have my inaugural podcast with both of you.

And hopefully it goes well and it's not your last.

So a shout out to the executives and board members of Socket.

socket.dev um so yeah uh what's going on the fable uh yeah so uh over the weekend uh we wrote and uh pulled together an open letter on this you can go to free fable.org uh a little bit of a failure uh by the brand management team at anthropic that that domain uh i was able to grab on cloudflare uh for eight bucks or whatever um uh but uh yes uh why uh did we do this um Basically what happened, so a quick recap is there was research that was being done by, well I'll just say it because everybody knows, by Amazon, which is not unsurprising.

Amazon is a partner of Anthropix, hosts Anthropix Models.

And I expect that this research is being done as part of kind of the shared responsibility model that Amazon has here, where they are responsible for monitoring and securing Anthropic's models in some cases, including in the classified setting, right? Famously, there are controversies around Anthropic providing their models to the government.

In the situations where Anthropic models are provided to the government, it is on Amazon hosting.

Why was Fable the one that was pulled away?

I don't know if you'd have any comment on whether Meta is training off of this Chinese model.

Yeah, I don't have any specific insight.

I'm not at Meta anymore.

What that's called is distillation.

So when you're building a model, one of the ways you can train it is you could ask another model millions or billions of questions, take that output, and then build your model to give similar kinds of output.

That is widely believed and pretty much proven that this is how DeepSeq jumped way ahead by doing this using OpenAI's models running in Microsoft's cloud.

So it is not shot talking, that Facebook would be doing the same thing.

I mean, what do you think of the president's EO? Appropriate or do we need to let states kind of move ahead in certain areas in regulating the usage of AI?

Yeah.

But systems get more complex, less understandable, and more important every year.

And so as a result, systems get less safe and there's more need for people to break them and make them safer.

And I think AI has just massively multiplied that.

I mean, this is one of the weird things about AI.

Uh, uh, just theoretically, AI systems are supposed to be what's called deterministic, right? So a deterministic system is a piece of software where if you know the inputs, you can predict what the outputs are.

In practice to human beings, modern AI systems are non-deterministic.

Yeah.

Alex, is there anything that Microsoft and OpenAI and other providers can do to perhaps prevent this from happening again? If not DeepSeek, then perhaps another competitor from China?

Well, this is the challenge that they're going to have.

Microsoft's business model is that you can go to Microsoft Azure and you can throw down a credit card, you can open up an account, and you can ask them to create a private copy of OpenAI's models for you that you can use for your own purposes.

This is how Microsoft plans on monetizing their multi-billion dollar investment in OpenAI.

That is a great thing for them.

That's a great thing in theory for both OpenAI and the customers that you can have a version of OpenAI that in theory does not expose their model weights to the customer and does not expose the customer's private enterprise data to OpenAI.

But what has happened is that people have figured out ways that you can ask a model many, many millions or billions of questions to suck all of the data out of it.

Should similar reticence be used by U.S. companies, by U.S. people, Alex?