Superintelligence

How's AI going to kill us? How actually is it going to do that? So I think that this is something that's been kind of confused quite a lot in the reporting.

The concern is not with the intelligence itself, but the concern is with superintelligence and the threshold that that kind of change represents.

Superintelligence is just the word for when intelligence is greater than human capability.

And so I'll give a scenario.

Say I decided that I wanted to wipe out humanity.

I also think that the arguments are old.

I think that Superintelligence by Nick Bostrom were essentially rehashing the same points he made for the past decade.

I think the big problem here is that there's no consistent messaging.

So I think on the one hand, you know, they surface the risks and at some point they're saying, well, this is the most dangerous thing ever.

What I'm worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought, end quote.

So the concern from both of these individuals is what's called self-improving superintelligence.

Superintelligence is the idea of an AI system becoming better than humans at almost every major thinking task.

So things like writing software, doing scientific research, solving complex math problems, analyzing very large amounts of information, making strategic decisions, even helping design new technology.

And when we talk about self-improvement and superintelligence, the concern is what happens if AI also becomes better than humans at improving AI itself? If that happens, it could help build a smarter version of itself.

Hard to say.

I recall when I first read Nick Bostrom's Superintelligence, and I was a bit more skeptical about the general idea of AI safety.

And my thought at the time was, when it gets here, we'll figure it out.

And now it's kind of here.

Recursive Intelligence raised $335 million at a $4 billion valuation back in February.

Three months later, Recursive Superintelligence raised $650 million at a $4 billion valuation.

And former Google DeepMind veteran Jeff Dean launched Discovery Loop last month.

Connor Leahy, U.S. Executive Director of AI Safety Nonprofit Control AI, told TechCrunch, The creation of recursive, self-improving loops, so an AI system that can build the next generation of AI system, which itself can build an even more powerful AI, which can build a more powerful AI, etc., etc., is the most likely candidate for the point we lose control.

The major political news is that U.S.

Senators Bernie Sanders and Greg Cesar introduced a bill this week called the Ban Artificial Superintelligence Act.

The BSA bill.

Yes.

They're gonna get us.

He says, "I believe Anthropoci- Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

Superintelligence is the still theoretical notion of an AI agent that is smarter than even the sharpest human minds." Great.

Okay.

What's the fair gonna be like next year?

Okay.

Superintelligence is a still theoretical notion of what AI could become.

But I think this is where people who work in the industry are starting to place their concerns.

Evan Hubinger is part of a company that is based in San Francisco called Alignment Science.