Marcus is about to flag a core philosophical objection to EA thinking, and actually to utilitarian thinking in general, that there is no consensus, nor has there ever been, on what we mean by good, and definitely not long-term good.

So, you know, one thing that we need to think about is, well, what is it to do the most long-term good, right? Typically, that's understood in terms of, again, utility or happiness.

And so you say, well, look, it would be a better world if the AI just got rid of us terrible human beings that commit crimes and commit murder.

But wait a minute, right? The question of what long-term good is is a philosophically contested one.

Because, you know, despite all of the bad things in humanity, right, we live in a very meaningful world with success and failure.

And, you know, when we discuss in philosophy, uh, why, if God exists, God would create a world with evil, part of it is, is that, you know, evil challenges our souls, right? It gives us meaning, right? Suffering can help us understand and empathize with others and improve ourselves and try to improve the world.

And so if we come back to effective altruism, do the most good, right, what doing the most good is is a deeply contestable philosophical question, right? And, you know, most of us would think, no, the most good is not for a potentially non-sentient set of AI, right? Because, uh, again, I think consciousness of AI is a very serious question, whether it's possible for digital AI to have conscious awareness at all, right? Whether it would be good for non-conscious AI to exterminate us to prevent us from doing harm when in fact a world with evil in it is meaningful.

Are humans even capable of altruism to begin with?

I mean, is this something that we should endeavor on?

on?That's a really interesting question.

Here's what I'll say.

I don't think that people have taken seriously enough the concern about experimentation, you know, in trying to grow or, you know, a conscious being as, you know, an artificial intelligence if it's possible to create, um, artificially sentient creatures.

We're also testing on them, right? And potentially, like, killing them constantly, and I don't think the ethics of that has been taken seriously enough.

M-my skepticism about digital consciousness notwithstanding 'cause again, I don't presuppose that my, my views are right.

I just provide the arguments that we have, uh, good evidence for.

And, uh, like will you be consulting with people like that as you are, as you are, you know, establishing these ethical guidelines?

The idea of consciousness as an intrinsic property of the world of physics might remind you of earlier discussions we've had about panpsychism Uh, for whatever reason, every time we end up talking about, about neuroscience on this show or talking about consciousness, we end up talking about panpsychism, which I understand [chuckles] you have some thoughts on.

Yeah.

Um, look, there's no, no theory of consciousness that's accepted, uh, by a majority of theorists because th- they all say things that, that seem wrong, right? Or very hard...

N- maybe not seem wrong, but seem very hard to accept, right? So for a while, you know, computational functionalism, the basically, like, consciousness and the mind is, is simply a set of computable functions, like enjoyed a great deal of like support.

But then people realize, a- as I mentioned earlier, it's not clear how a function, right, any function, right, can explain what green looks like or what red looks like, uh, the fundamental constituent subconscious experience.

It's not clear that anything can explain them rather than taking them to be some kind of like fundamental aspect of reality.

Like redness is just a thing in itself somehow in reality.

Consider me guilty as charged.