Holden Karnofsky

Ms. Tuna is on its board, as is an executive at the trading firm Jane Street.

So is Daniela Amadei, who's married to GiveWell's co-founder Holden Karnofsky, now working at Anthropic.

Amadei is the co-founder and president of Anthropic and sister of Anthropic CEO Dario Amadei.

EA has had some big bang moments, like the implosion of Sam Bankman-Fried and his colleague Caroline Ellison, who met at Jane Street and, motivated by effective altruism, tried to strike it rich and do good with it, or so they claim.

Well, I will tell you, I don't think I can say this because this is a long time ago.

I remember having a long discussion in Holden Karnofsky's car because he was dropping me off to my apartment in Berkeley in 2010.

I believe it was the spring.

I'm just like thinking of the weather.

That same year, Amodi stumbled on a link on an economic blog that led him to a nonprofit charity evaluator called GiveWell.

Uh, founded the previous year by former Bridgewater Associates hedge fund analyst Holden Karnofsky and Elie Hassenfeld, uh, it aimed to help donors maximize the good done per dollar donated, usually measured in lives saved.

Uh, Amodi began leaving comments on the GiveWell blog and once guest blogged on it directly.

In the 2010 post, uh, he deployed utilit-utilitarian reasoning while weighing whether to give 10,000 to one charity rather than another.

Yes.

GiveWell was a charity evaluator founded by a man named Holden Karnofsky.

Karnofsky was basically the central hub, the chief accountant for the early effective altruism movement.

So you have this collision of titans.

You had Chris Ola, another foundational, globally recognized AI researcher.

Wow.

And of course, Holden Karnofsky, who would later marry Daniela Amidoy and become Dario's brother-in-law.

It was a closed loop of the highest IQ individuals in the Bay Area.

And you have a Princeton physics PhD student named Dario Amadei, who is heavily absorbing Yudkowsky's probability models back in 2008.

And she eventually connected with Holden Karnofsky, who co-founded a charity evaluator called GiveWell.

Yeah, and by 2013, Karnofsky had literally moved into Amodei's group house near Glen Park in San Francisco.

So that Glen Park house was effectively the incubator for the modern AI safety movement.

Yeah.

Is it basically all AI all, all the time these days? I think o- one of your colleagues, Holden Karnofsky, who's also been on the show before, uh, used to work on a wider range of issues, but recent, recent events have prompted him to narrow his focus somewhat.

Has, has the same happened to you?

Certainly for me, it's all AI all the time, and it has been for three years or so.

... that's working on practical tests for whether a model is safe that potentially could be applied to all kinds of models, uh, b- before they get deployed.

And I think the, the hope that they have is that AI labs would, you know, voluntarily sign on to follow a kind of standards and certification program where models above any particular size would have to be stress-tested, I guess, in, in all these kinds of different ways to ensure that they're not gonna engage in behaviors that, that we don't like.

I think there, there's a decent amount of buzz around this effort and people, uh, who are helping with include Beth Barnes, who I'm hoping to interview later in the year, and Holden Karnofsky and Paul Christiano, who've been on the show before.

I, I guess we should say that, uh, I think, uh, you and Paul are, are, are an item, so.

Daniela Amodei, now Anthropic's president.

Holden Karnofsky, her husband now.

Um, Chris Olah, Paul Christiano, like all these sort of names from, uh, AI-

All lived in one house

I don't know.

I think about it in the orbit of, of like, uh, effective altruism, like sort of people who are interested in like money ball for charity sort of ideas of like, you know, what, what's like the most effective use of this dollar? So, um, uh, Daniela Almada's husband, Holden Karnofsky, was previously, you know, the founder of, of GiveWell, you know, sort of this platform for, you know, ch- channeling donations towards the most effective causes.

Um, uh, their early...

You know, Dario and Daniela themselves have been very involved with GiveWell and kind of philanthropic type of organizations for, for a long time.

Right.

The head of it is married to his sister, Holden Karnofsky.

He works at Anthropic.

Um, Dario was roommates with him, um, early on in, in his, uh, like, stint in the tech industry.