Reinforcement learning

But this sort of brings up a natural question.

You know, what can we actually use as a reward to encourage the behaviors we want in this text generation system? That's really up to you as you design this, your generation pipeline, a common practice in the early days of using RL for text generation was to set the reward to be the final evaluation metric that you were going to evaluate on.

And so here, instead of having a unique reward for each generated token at every time step, you would just take the final sequence score that you get and reward every token in the generated sequence with that value.

And this was absolutely magical.

you would set your evaluation metric as the reward, and you'd end up learning to get more reward, because that's what RL algorithms do, which in turn means that you were learning to generate sequences that do better on your evaluation metric.

So NLG benchmark scores were shooting through the roof.

We were making real progress.

And it's not always clear that optimizing to those approximations is going to lead to more and better coherent text generation.

This is the second of the post-training lectures.

We're going to talk about the exciting developments in RLVR, or reinforcement learning from verifiable rewards.

And I think this is, in some ways, very timely.

I didn't get to update the slide, but there was just an announcement by OpenAI folks that they had solved one of the major problems.

So that's an open math problem using one of their thinking models.

How does that thing work? That's exactly this thinking model stuff, or RLVR stuff.

And so thus far, where we're at is we are kind of at the left side here.

After we finished last lecture, we've got instruction tuning.

When we think about training a model for reasoning, one thing that immediately jumps to mind is you could have humans write out their thought process and train on that.

One aha moment for me was when we saw that if you train the model using RL to generate and hone its own chain of thought, it can do even better than having humans write chains of thought for it.

And that was an aha moment that you could really scale this and explore models' reasoning that way.

Wait, so it's better to train on model-generated chains of thought? But how come they're so often wrong? And what does he mean by reinforcement learning in this context? Well, how about this? And here, clearly, I'm going to get slightly metaphorical.

Order on the Sweetgreen app.

the model coming up with solutions with reinforcement learning.

We do not discuss any supervision the generator would receive from the reward model if trained with RL.

And here's the key sentence, although fine tuning the generator with reinforcement learning is a natural next step It is intentionally not the focus of this work.

And so human kind of fills out some kind of a... trace that like comes to the solution but like some parts of that are trivial to the model and some parts of that are a massive leap that the model doesn't understand you're kind of just like losing it and then everything else is polluted by that later and so like fundamentally what you need is the model my the model needs to practice itself uh how to solve these problems it needs to figure out what works for it or does not work for it but it needs to learn that for itself based on its own capability and its own knowledge so that's number one is like that's totally broken i think it's a good initializer though for something agent-like it And then the other thing is we're doing reinforcement learning from human feedback, but that's a super weak form of reinforcement learning.

It doesn't even count as reinforcement learning, I think.

So RLHF is nowhere near, I would say, RL.

It's silly.

And the other thing is imitation learning is super silly.

RLHF is a nice improvement, but it's still silly.

And I think people need to look for better ways of training these models so that it's in the loop with itself and in some psychology.

And I think there will probably be unlocks in that direction.