Is there anything more to say about what is the challenge? And I guess, what is the urgent challenge that we face with space governance?

Okay, great.

There's two challenges with space governance.

So that's a couple of starting intuition pumps for why this would become important.

One is just, okay, well, if the industrial base is growing a lot, well, we're going to need more resources.

And you've got enormous, in fact, a billion fold as many resources in space as there are on Earth.

And then a second is, once you've got AGI, it is much easier to do things in space, to have industry.

you want to make the case that sort of by default, assuming that we navigate through upcoming times, we don't go extinct, that in practice we're likely to do something that from an ethical or moral perspective is very mediocre, that only achieves quite a small fraction of the total value that in principle could be generated by the universe?

T- tell us about that debate.

Yeah, I mean, this has been...

It's like I often call it, like, the horror of effective altruism in some way.

But it's just the horror of the world that we need to make incredibly hard trade-offs between different groups we can help, but also between different ways of helping.

There's Give Directly, which is extraordinarily evidence-based.

You just basically guarantee that you're gonna massively improve the lives of many people versus things that, uh, just are more speculative.

And so one of the things that I think we've learned, or certainly I've learned or changed my views on over the years, is that a certain mode of reasoning that seems very speculative has actually now got quite a remarkably powerful track record where those of us in effective altruism were warning about funding, working on, um, issues of pandemics and AI for many, many years before COVID, before the ChatGPT moment.

And military.

Yeah.

These protect against regular, normal pandemics like we've seen throughout history, but they also protect against novel pandemics, where we have the ability now to create and build new viruses, new pathogens.

At the moment, that ability is constrained to people with sufficient skills in, you know, a handful of labs.

But the equipment needed to do so is getting, is not that expensive, and it's getting cheaper all the time.

The knowledge needed to do so is becoming more and more democratized.

Mm.

Where there are gains in how much computing power are just being thrown at AI for training, for experimentation, for inference.

There are gains in algorithmic efficiency, so how much of a punch can you get from that computing power? And then when you look at how does AI perform, whether that's on benchmarks or in terms of the time horizon of human equivalent tasks, so a task that might take a human three minutes or thirty minutes or three hours, that just occupies this relatively smooth exponential trend-

And their values today might be that I don't really need a fancy car and I don't need a house in the Hamptons and I don't need to keep up with the Joneses and I'm really motivated by the set of ethical considerations, but if I go to Wall Street at 21 and I'm surrounded by people who make a ton of money from such an early age and I start to think, "Well, you know, I want the next promotion and to get the next promotion, I have to go to dinner with my boss, and so I need to learn to spend quite liberally because they're not gonna think I'm a good cultural fit if I don't, I mean, I'm not gonna earn the money that I need in order to be able to donate as much as I want." And then perhaps you meet a partner and you start thinking, "Well, they have certain expectations for how to live because they're from that social milieu." And when you have kids who are going to certain kind of private schools that are very expensive and, y- you know, by the time that you've made all of this money, by the time that you're actually in a position to, you know, fulfill the ambition that set you off on that path in the first place, are you going to be the same person? Are you (laughs) actually going to be making those donations or will you have any number of rationalizations for why, uh, yes, you'll make some donations and so on, but really, you should be spending the bulk of that money on yourself and your family and the lifestyle to which you've grown accustomed?

Yeah.

Okay, so, uh, great objections, um, and, uh, all very important ones that, you know, are things that we thought about a lot and, uh, we're concerned about from, um, uh, from the early stages.

Like, I actually talked about all three of these in, um, my original article on the topic, now 12 years ago.

Um, so going through them kind of briefly, like, on the harm question, I just think, don't do things that are, you know, seriously common sense unethical that, um, involve inflicting large amounts of harm.

Now, from some perspectives, maybe you've got a kind of, uh, kind of radical anti-capitalist perspective.

Maybe that's just many areas of the world.

Are there technological solutions to that, that AI labs just currently don't have the incentives or the regulatory, uh, imperative to, to deal with? Or are they just inherently things that if we build sufficient intelligent systems, we're not gonna be able to control?