GPT-4

I tried multiple times and every time it failed on word counts.

I'm not trying to pick on one GPT, but very often using one of the GPTs wasn't really an improvement on just using GPT-4.

There was, however, one notable exception to that, which was the consensus GPT.

In the world of expense management, SAP Concur is here to help.

Terms and conditions apply.

Hiring now? Then this is a job for Indeed sponsored jobs.

It's not sponsored, but by giving relevant links that I could then follow up with, it was genuinely helpful and better than GPT-4 base.

This is for the specific task of looking up scientific research on a given topic.

So I kind of went to all my AI research friends and was telling them, hey, I'm thinking about working on this.

And essentially the consensus from everybody was that this seems impossible to you know, why don't you just, you know, come join me at OpenAI and like work on making GPT-4 better or something like that.

It's like, no, no, I think there's something here.

I really do.

And yeah, OpenAI were the leaders in that initially and just scaling this thing up by orders of magnitude, having more and more transformers on more and more NVIDIA GPUs.

And every time they did that, GPT-2, GPT-3, GPT-4, they were shocked.

The people developing these systems were shocked at the heightened levels of intellectual ability, of generalizability that came out of that scaling.

What I find, I think that's incredible.

From 2018 to almost 2024, this graph is a little bit outdated, but from 2018 to 2024, as the model size has gone up almost consistently, we have been increasing the size of the model.

So we started with something like BERT was 340 million parameters, GPT-2 was 1.5 billion parameters, and then GPT-3 was 175 billion parameters, and then say Palm was 540 billion and then there was a bunch of subsequent models and GPT-4 is estimated to be there which is like trillions of parameters so you can imagine that there has been an exponential growth in just the model size for large language models.

The large stands for that they're growing in size and that has been one of the key areas in which the models have been getting better.

So why do we care about the models getting bigger? What does this lead to? So as I was explaining that as the models get bigger, they continuously improve in performance on national language benchmarks, on various other benchmarks, reasoning benchmarks, and so on.

For a bunch of tasks like math and coding benchmarks, we increase the number of samples per problem from one, which is the normal one, to 10,000.

And here we are showing the coverage or the fraction of problems that are solved by at least one of these samples.

And what we are seeing here is that, so the red line, the red dash line was the GPT-4-0 model.

And while these blue and green lines, which were the LAMA-8B, LAMA-3-8B, and 70B, were worse than GPT-4-0 model with one sample, but if we increase the number of samples from these models, in all of these cases, they do better than the GPT-4-0 model.

And finally, and I still don't fully understand or believe this, a miracle seems to have happened, deep inside the weights of the frontier models.

Quoting Davidad, Quote, Somewhere between the capability profile of GPT-4 and the capability profile of OPUS 4.5, there seems to have been a phase transition where frontier LLMs have grokked the natural abstraction of what it means to be good, rather than merely mirroring human values.

End quote.

Quoting Yanis.

In a randomized controlled trial published in JAMA in 2024, Go and colleagues, G-O-H, Go and colleagues gave physicians tricky diagnostic cases.

So half of these physicians got GPT-4, which is the machine that powers ChatGPT-4 to help.

The other half used conventional resources such as books and internet sources.

physicians who had the AI barely improved.

So the AI barely helped physicians at all.

However, when GPT-4 in this experiment was working alone, it scored well above both groups.

So listen to that again.

GPT-4 working alone scored well above the physician using AI and the physician using conventional resources such as books and internet sources.

The concepts are abstract enough that they would of course never have been seen in the training data.

The original GPT-4 flopped hard at this challenge in the paper In Nature, and 01 Pro Mode gets close, but still can't do it.

After thinking for nine minutes, it successfully translates who as double, but doesn't quite understand moreau.

It thinks it's something about symmetry, but doesn't grasp that it means between.

I just suspect it's not in the training data of O1 at least.

likely not in O3, but we don't know.

Is the base model for O3 Orion, or what would have been GPT-4.5, GPT-5? OpenAI never mentioned a shift in what the base model was, but they haven't denied it either.

Someone could make the argument that O3 is so good at something like physics that it can- UKG is

genuinely it's either that none of this is about to happen we're just wrong about what the like future of develop of ai looks like maybe we don't hit asi as fast as we thought we don't even hit age like who knows that could be plausible so there's some probability you have to place on that like obviously you would agree like there's some probability that like gp8 isn't even good right like that's theoretically possible of course yeah there's some

probability that something hits a wall, but I will say this, my personal probability of thinking, you know, when GPT-4 came out, I was like, hey, there's like a 30% chance that we, remember this concept of AI winter, right? We had like five AI winters in a row followed by short AI summers.

And, you know, Robin Hanson was on the show a couple of years ago saying, yeah, we should expect more AI winters.

I haven't seriously considered AI winter in a long time.