Orca: The Model Few Saw Coming

Do you remember this paper, less than two weeks old? It made waves by concluding that open source models can mimic the style but not the factuality of ChatGPT.

Overall, we can conclude, they say, that model imitation is a false promise.

Well, 48 hours ago we have this, a 51 page report on Orca based on a small 13 billion parameter model.

I don't often comment on open source models because they're simply not competitive with OpenAI's models.

But Orca is not just competitive with GPT-3.5, it beats it in quite a few well-established benchmarks and even matches GPT-4 in a couple of tests of reasoning.

As always, I've read both papers in full and can also bring in just released comments from Sam Altman and Ilya Sutskova on competition from open source models.

But let's start with Orca, named presumably because Orcas or killer whales are frequent visitors to South American coastlines.

And South America is of course the land of llamas and vicunas.

But all the research was done by Microsoft which I find interesting and I'll come back to that at the end.

But why did they make Orca and why does it perform better than models like llama, alpaca and vicuna? Well, they say here in the abstract that those other models lack rigorous evaluation, resulting in overestimating the small model's capability as they tend to learn to imitate the style, but not the reasoning of LFMs, large foundation models.

To address these challenges, we develop Orca, a 13 billion parameter model that learns to imitate the reasoning process of the larger models.

Orca learns by looking at GPT-4's step-by-step thought processes and is guided by teacher assistance from ChatGPT, which is GPT 3.5. And to give you a taste of what's to come, Orca surpasses conventional state-of-the-art models, such as Vicuna, by more than 100% in complex zero-shot reasoning benchmarks, like the Big Bench Hard, which I'll talk about, and by 42% on AGI eval.

It goes on...

Orca reaches parity with ChatGPT on the big bench hard and shows competitive performance in professional and academic examinations like the SAT, LSAT, GRE, and GMAT.

And I know many of you will be interested in this footnote.

We are working with our legal team to publicly release a diff of the model weights in accordance with Lama's release policy.

So if this is anything like Lama, it's going to be leaked across the internet imminently.

I'm gonna show you so many tests and benchmarks in a moment, but just to give you a sample, here is Orca outperforming ChatGPT in the Vicuna evaluation set and matching Text DaVinci 3 in the SAT, LSAT, GRE and GMAT.

And as I'll touch on later, this was zero shot without chain of thought or any advanced methods.

You can watch pretty much any of my other videos to see how advanced prompt engineering would probably boost those results still further.

For those who didn't know, 13 billion parameters is about 7% the size of GPT-3, which is 175 billion parameters, and possibly around 1 or 2% of GPT-4's size.

That gives you an indication of the difference in size between Orca and these models that it's competing with.

And if that doesn't make any sense, a smaller size means it can be run on much smaller devices, like a desktop or even possibly a laptop.

The authors start off by giving a little slap to the other paper, you know, that one that said model imitation is a false promise.

And they continue that contrary to this assertion, it is possible to reduce the gap with proprietary LLMs on multiple zero-shot benchmarks that require sophisticated reasoning.