'Show Your Working': ChatGPT Performance Doubled w/ Process Rewards (+Synthetic Data Event Horizon)

They trained two reward models for GPT-4.

One which gave positive feedback for a final result.

The final answer to a mathematics problem for example.

And another model where they gave positive feedback to GPT-4 or ChatGPT based on each intermediate reasoning step in the mathematical solution.

Basically a show you're working out kind of approach.

And the result they got by rewarding good working out surprised even them.

It was able to solve 78% of problems from a subset of the math test set, which I'll get onto in a second.

Not only is that almost double GPT-4's raw performance of 42.5%, which by the way is about double GPT-3's performance of 23%, it also outperformed just rewarding correct answers.

The blue line represents using a model that rewarded correct answers only.

And then you have the reasoning or process supervised RM at the top.

So even when you explicitly reward correct answers, you get fewer correct answers than rewarding good working out.

And yes, that did surprise OpenAI.

I can hear some of you wondering about Palm 2, the latest model behind Bard.

Well, the raw model gets 34.3% and even the model with self-consistency and chain of thought only gets 48.8% on this math dataset.

The previous state of the art, by the way, was 50.3%.

So 78.2% is quite a big leap.

And later on, I'm gonna show you why that's not even the cap.

Just for interest, here is the rather ugly title page that OpenAI put out.

They call it improving mathematical reasoning with process supervision.

Maybe if someone had supervised the color scheme of this release page, it might have looked better.

But my point wasn't just to diss a color scheme, it was to point out something that they also said down here.

They say, in addition to boosting performance relative to just looking at outcomes or correct answers, this form of process supervision also has an important alignment benefit.

It directly trains the model to produce a chain of thought that is endorsed by humans.

Indeed, Ilya Sutskova retweeted this from the head of alignment at OpenAI, calling it a really interesting result.

But let's leave alignment for later.