Recommender Systems Origin Story

There's this story that every data science course tells.

It goes like this: A man storms into a Target outside Minneapolis, coupons in hand, and demands to speak to the manager.

His teenage daughter has been getting mailers for cribs and baby clothes.

She's still a teenager.

She hasn't even graduated high school.

Why are you sending her this? The manager apologizes, says he'll look into it.

A few days later, the father calls back, but this time it's to apologize rather than seek an apology.

It turns out his daughter really was pregnant, and Target knew before he did.

Now, it's not as though there's an is_pregnant column in some database somewhere.

How did this happen? It's kind of a perfect story, and a statistician named Andrew Pole did some analysis and was able to build his own pregnancy prediction score based on Target data by looking at just 25 products.

He could use those 25 to estimate the due date of the pregnancy.

But that isn't a miracle, it's just data.

Purchasing a pregnancy test is a decent predictor of having a baby eight to nine months in the future.

Buying gender reveal party invitations probably indicates you're going to have a baby in one to three months, I would guess.

But what do I know? Andrew Pole had the data.

He found things like supplements are often bought early in a pregnancy.

Items like unscented lotion start showing up on the receipts in about the second trimester.

So there's this emotional hook that Target knew before the father knew.

And depending on your definition of knew, of course it did.

In fact, it wouldn't surprise me to find out that the father is almost always the last to find out about a teenage pregnancy.

A spreadsheet beating him to it is not a sign of an advanced machine making a startling prediction.

It's a machine that can find a probabilistic needle in a haystack of data.

The Target story is really a story about inference, a machine noticing a pattern that you did not know you were giving off, and it gets more unsettling as the signals get more intimate.