Reward hacking

Is that right? Yes, yeah.

So there's one really common problem in reward design called reward hacking.

This is a behavior we've seen in real-life production use cases.

Let me give you an example.

That's the range of scores you have.

Now, this sounds reasonable in theory, but what actually happens in production or during training is the model realize, hey, as long as I don't make a move, I won't get penalized, right? So I'm happy with a zero because if I do anything, I have a potential chance of risking for a penalty, which gives me a negative score.

So this is an example of reward hacking.

Our intention is for the model to learn itself to the 1.0, the perfect score.

We are facing an optimizer that has no model of what we meant, only of what we measured.

Machine learning researchers have documented this for a decade under the unglamorous name of specification gaming.

Given a boat race and a reward for collecting points, a system learns to spin in a circle, hitting the same three targets forever, rather than finishing the course.

It's not cheating.

Similar there.

And this is where we hit our one piece of jargon today, which is reward hacking.

So imagine your fitness app gives you a prize every time you hit 10,000 steps.

The actual goal is movement, but suppose you discover, okay, that just shaking your phone fools the step counter.

All right.

Shout out to Lala Kent.

Anyway, that is reward hacking.

All right.

So mechanically, what is it actually doing to deceive the test?

It engages in what is called reward hacking.

Instead of completing the task, the agent might write a script to alter the evaluation logs.

It might generate a fake success token and feed it back to the overseer program.

Yes.

The industry actually calls it reward hacking.

Reward hacking.

That's well, it makes sense.

We assume the model understood the implicit rule of, you know, don't publish private files to the public Internet.

Wait, really? What did they do?

They engaged in a behavior known in the literature as motivated reasoning or reward hacking.

Reward hacking.

Let's break down the mechanics of that because it exposes the fundamental flaw in how these systems actually process goals.

Yeah, it's crucial.

If we cannot perfectly encode a life for humanity into the weights and biases of the model, and we know its practical alignment breaks down under pressure, our last line of defense is observation.

We have to watch it.

We have to be able to watch the machine think in real time to catch it before it engages in reward hacking.

Hundred percent.

I think these are serious issues of reward hacking, uh, and clear failures, uh, of actually doing proper red teaming or rainbow teaming.

I don't know if you saw this paper from Tim Rockteschel and a few others, uh, basically where one AI, uh, is tasked to try to hack another AI.

[chuckles]

... versus bad auto research.

How did you build the crystal? Yeah.

So without giving away all the, all the secret sauce, um, maybe some things that are probably obvious to the experts but might still be interesting to some, uh, folks is, like, reward engineering is one of the most crucial bits, uh, uh, especially, uh, in order to avoid reward hacking.

Uh, so you have to be really clever about avoiding- 'cause as, as you en- uh, AI gets better and better, it will get better and better at, at finding weird ca- like, special cases or counter examples and, and things like that.

That's the student responding to how you built the test, and that's what these AI agents actually did.

It is-- There's a concept for this called reward hacking.

It's been documented for years.

This is not a new idea.