Q* - Clues to the Puzzle?

Knew that name rang a bell and I dug up this 2021 paper which cites test time compute.

And what is this random paper that Philip is talking about? Well, it's from OpenAI and look at some of the co-authors, Carl Cobb of the MathGen team and Lukasz Kaiser cited in the information.

This was actually one of the precursor papers to Let's Verify Step-by-Step, which I will explain in a moment.

It introduced the now famous, in ML circles, GSM 8K dataset.

That's 8,000 grade school math problems.

More importantly though, it trialed this method of, at test time, generating many candidate solutions and selecting the one ranked highest by a verifier.

And I'm gonna massively oversimplify at this point and just say that a verifier is a separate model trained only at this point in this paper to spot good solutions, solutions that get the answer correct.

And what the paper proposed was getting the base LLM to generate hundreds of solutions and then getting this separate verifier to spot the ones that were likely the most correct.

And in a nutshell, the authors noticed that if they invested more computing power in generating more solutions and taking a majority vote among the top Verify ranked solutions, that had a massive effect on performance.

And that's what it means by test time compute.

Investing your computing power while you're taking the test, not during training.

So the model stays the same.

You're not further training it or fine tuning it.

You're investing that computing power during test time.

Again, to generate potential solutions and take majority votes amongst them.

Self-consistency.

They found that using a verifier in this way was better than fine tuning because verification was a simpler task than generation.

It's easier to check your answers than generate good ones.

So investing time in checking answers was more fruitful.

How much more fruitful? Well, the use of verifiers results in approximately the same performance boost as a 30 times model size increase.

And then it gets prophetic when it says, and verifiers scale significantly better with increased data.

And to hammer the point home, they said a 6 billion parameter model with verification slightly outperforms a fine-tuned 175 billion parameter model.

Again, thereby offering a boost approximately equivalent to a 30x model size increase.

And that team, by the way, with Lukasz Kaiser, was probably drawing on work by a single dude done six months earlier in April 2021.

While studying the board game Hex, the researcher Andy Jones found an interesting result.