Recommender Systems Optimization Goals

Welcome back to the Data Skeptic Recommender Systems season finale.

In part one, we traced those systems from a 1992 research team at Xerox PARC all the way up until the Netflix Prize.

As you'll recall, the Netflix Prize has a 1 to 5 rating.

At least that was their original rating system.

They had a whole lot of data on it they released.

And they used RSME, Root Mean Square Error, between your prediction of 1 through 5 and their holdout data of actual ratings given by actual users to actual movies.

And the criticism that I had for it was that your ground truth data and the thing you'd like to predict are not the same thing.

The ratings provided by Netflix represented how someone felt about the movie after watching it.

This is not the same question as what movie would you most like to watch next.

Now maybe you find that to be a pedantic distinction, but in pedantry we find the story.

The rating was a good proxy, but is not precisely the same thing as the outcome you're seeking.

And even if, as machine learning people do, we just kind of incrementally inch up and up and up until we've overfit a data set, it seems that root mean square error is, what do they say, necessary but not sufficient.

So that leads me to the question I want to tackle today.

What should we be actually optimizing for in recommender systems? And that's where we pick up today.

Thanks again to Run and Punch for use of their song, We've Never Met.