Most Measurement Models Are Wrong

So, um, let's let you jump in and, uh, kick us off on this first topic, why measurement models fail without real data validation.

Yeah, for sure.

So this idea of real data validation, I think, is sometimes, um, a little confusing to people because they're like, "Well, I, you know, I log into the Google Ad platform, and I see real data there," right? But what do we mean when we say real data validation? The sort of philosophical concept that I think is really important to understand when it comes to marketing measurement is that the true incremental performance, like, the true return on investment of investment in some marketing channel is unknown and unknowable, right? So, like, if I invest a million dollars in Facebook, that will generate some amount of revenue for me ranging from zero to $100 million.

There is some true amount of additional revenue that that's driving for us, but we don't know what it is, and we can't know what it is.

Um, and so when we think about the right way to do marketing measurement, there's sort of two points that, that I think are important to keep in mind.

One is that you need to be thinking about this idea of incrementality, right? If not for this additional marketing investment, what would have happened otherwise, right? So you don't just wanna say, "Oh, you know, a bunch of people clicked on our ads and then purchased, and that means those ads caused the purchase." That's not necessarily true.

A lot of people could click on an ad who were going to purchase anyway.

So we care about this idea of incrementality.

Um, and then the second piece is that we need to recognize that no measure of marketing is going to be perfect.

They're all going to fall short in lots of different ways because we don't have access to the truth.

Um, and so different measurement methods are, you know, all going to be limited in important ways, and it's really important for us to think about exactly in what way is this measurement wrong and how can we sort of test the assumptions of these different measurement methods against reality, against the, the things that actually matter to the business, like the amount of profit that we're generating, in order to make sure that we're not fooling ourselves.

Yeah.

I, I think you made a really good point when it comes to incrementality, 'cause a lot of times people overlook that, you know, that thought process.

There's a certain amount of business that's going to come in, so what is the actual incremental lift that this campaign is actually generating, right? Um, it's something that we've-- we have some experience with in, in some of the clients that we have in, um, in the TV and CTV space, where we've talked about, like, they know that they're gonna reach so many households already.

So when we run a campaign and we apply the data that we have to that campaign, what is the incremental lift, and how do they measure that so they can see at, at, you know, that they actually reached X number of new households that they hadn't reached before, um, you know, with that same program? And so I, I think that's, like, one example, and it's sometimes something that's very easily overlooked.

People look at-

But how-

... you know, they run a campaign and look at what was the total revenue generated from that campaign, but, you know, you have to break that down and take a look at what would you have gen- generated already, and that's a great way of doing that.

Yeah, I mean, and, and this shows up in, like, one really common way.

If you go to all of your different ad platforms, and you look at, like, the amount of revenue generated from each of those different ad platforms, and you sum it up, you put it in a spreadsheet and you sum it up, you'll often get a number that's, like, much higher than your actual revenue.

Yeah.

Yeah.

So, like, there's double counting happening here in, like, very meaningful ways, uh, that can really lead you astray if you aren't really just, like, constantly thinking about it.

What would have happened otherwise, I think, is, like, the question that we always need to be asking ourselves as we're thinking about trying to do marketing measurement well.

Yeah.