GPT-3

People didn't really expect that.

We were pointing towards, for example, GPT-3 by OpenAI.

We were arguing that, hey, it looks like large language models could actually become quite a big thing.

But lots of people were not persuaded.

Yes, this is a big one.

In the era of GPT-3 and 4, the meta strategy was to craft massive, intricately detailed system prompts or custom instruction files.

Users would write 2000 words dictating exactly how the AI should format its text, what tone it should adopt, and explicitly forbidding it from using certain cliches or attempting specific tasks without permission.

It was like putting massive bumper rails in the gutters of a bowling alley because the early AI models constantly threw gutter balls.

So if we just think, you know, back to what we think of AI as, you know, chat GPT, there's a lot more to AI than these things.

Just that, four years ago, when chat GPT-3 was first released, Who would have thought then, four years ago, that the same particular AI could be making wonderful pictures for us, talking to us, answering questions about anything we have, making videos, helping us deal with problems? Who would have thought that? And now, you know, because things are speeding up, I don't think any of us have any idea where things, what's going to be happening in two years' time.

It's just going faster and faster.

And the human mind cannot think in exponential terms.

Um, a-actually, I don't know if he talks about it much anymore, but initially, uh, uh, Clem, you know, sort of, you know, he got co-founder, but he's sort of the main guy, uh, so to speak.

He wanted to build a conversational agent, but, like, pre-GPT-3, pre-ChatGPT, you know, um, in, in, in, in sort of the LSTM days.

And then that wasn't really working, so he's like, "Hey, you know, can we basically just build a central dep- repository for, for AI models?" You know? And the reason was at the time, people were releasing models in open source, but every, every GitHub package was different.

So the definitions weren't the same.

Why were the Chinese companies a bit behind on LLMs? I don't really know the answer to this, but what the, that same person who pointed me to Super Glue said was the Chinese companies just didn't really see the point in spending so much money to create a giant LLM.

They looked at GPT-2, they looked at GPT-3, and they were kinda like, "That's an awful lot of money to burn to get an AI to write bad poetry." [chuckles] That's the way he put it.

Why would we spend all this money to create something that has no use case? Again, reflecting maybe a bit more practical and a bit less ideological view of AI in China than we have in the United States, or at least in, in Silicon Valley.

Now, as it became clear that, oh, instruction following works, now you have an AI assistant, and then obviously everything from there from reasoning to agents, like they've clearly seen the use cases and they've clearly been making the investments to try to be neck and neck with the United States at the frontier.

A lot of their talent has found its way to the United States, which is definitely to our credit as a society that welcomes people, knock on wood, from around the world and allows them to put their talents to their highest and best use.

But the competition was already fairly tight as of 2022, and it was m- m- maybe a mistake by the Chinese companies to take a pass on that initial scaling push in language models.

But it's like a fairly reasonable one because indeed, like as I can tell you as a user of GPT-3, like it wasn't really that useful.

We were able to get it to write basic marketing copy and that was about it.

It's also smarter in logic.

Here's an example of GPT-3 versus GPT-4 on the same logic problem.

User: In a room there are 100 murderers.

You kill one of them.

Its reasoning capabilities surprised even the OpenAI team.

During tests, they discovered that it was able to solve hindsight neglect, a decision-making problem where it had struggled greatly in the past.

In this figure, you can see GPT-3.5 scored almost zero and GPT-4 a perfect score of a hundred.

Improved reasoning also boosted a sore spot for previous models, the factualness of the model.

Yeah.

GPT-3, uh, 2N, and so on, right?

Oh, oh.

Amazing.