Nancy Leveson

You're going to control that one and let the other stuff burn.

And one of the exercises there, I'm putting a link that's going to be in the show notes, I'm assuming, on Leveson, on severity, is an exercise that the Ascertain team ran multiple times that was called, like, who do we kill in space to save on Earth? And the idea comes from Nancy Leveson from Engineering a Safer World.

She has this exercise, not this exercise, but this list of requirements NASA has.

And one of them is like, there's a top level priority of a mission.

So we have never built systems that are... fully autonomous for more than short periods of time, right? Even things like the automatic pilot in aircraft require human management around them.

And I recommend that people look at Nancy Leveson's textbook, Engineering a Safer World from 2011, and her more recent textbook on safety engineering, where she really emphasizes that you need to have a controller that's basically the human organization that surrounds a system.

So if we look at, say, the self-driving cars, Waymo has been having lots

of problems.

This isn't a new idea.

Back in 1986, John Knight and Nancy Leveson ran a famous experiment.

Independently written program versions failed together far more often than independence predicts.

Gnem's paper leans on it.

Um, one of the...

There's a, there's a, a famous MIT professor, her name's Nancy Leveson.

She's, she's done work with system dynamics on this concept that's called, uh, the complac- complacency loop.

And, and what happens is that we have an accident, and everybody's like, "We're never gonna let that happen again.