Engagement with safety: Headspace's guardrails for AI tests

Wow, that is amazing.

So I would imagine with a business like Headspace, it's a little bit of a different business in terms of what it means to do optimization versus a prime video.

That's a great question.

Yes, and I think that comes down to the different ways that the company is built.

So I mentioned we have like a B2B business and we have a D2C business.

So when I think about optimization, there's two different kinds of customers that we're looking for, first of all.

And then secondly, which this will come up when we're talking about actually writing experiments, we are not just trying to drive engagement at all costs.

So at a place like HBO or Prime Video, yeah, like putting costs aside, more engagement is generally better.

You want people to watch the next show, read the next book, watch the next movie.

You can look at things like how many minutes someone's spending with your product and But that's not the case for Headspace because more is not always better.

We want people to get better if they are coming to us with actual symptoms.

And so the optimization is health outcomes, not just more and more time in the

app.

Right.

So now we have this like dual mandate that we're really trying to drive.

Yeah.

And how do you measure outcomes?

Great question.

There are lots of different ways to measure outcomes.

Some are at a population health level.

Some are using third-party data sets or match data sets.

But one of the ways that apps like ours really try to hone in at the individual level is by self-reported data and taking snapshots of a user's self-reported scales of how they feel in terms of anxiety or depression, for example, and take those snapshots over time and be able to kind of correlate that with any interventions, treatments, or content that they've engaged with over that time.

Yeah, that makes a lot of sense.

So I'd imagine you're, you know, for a typical experiment, you're looking at both kind of traditional metrics as well as these kind of more outcome based.

Exactly.

Wow, that is amazing.

So I would imagine with a business like Headspace, it's a little bit of a different business in terms of what it means to do optimization versus a prime video.

That's a great question.

Yes, and I think that comes down to the different ways that the company is built.

So I mentioned we have like a B2B business and we have a D2C business.

So when I think about optimization, there's two different kinds of customers that we're looking for, first of all.

And then secondly, which this will come up when we're talking about actually writing experiments, we are not just trying to drive engagement at all costs.

So at a place like HBO or Prime Video, yeah, like putting costs aside, more engagement is generally better.

You want people to watch the next show, read the next book, watch the next movie.

You can look at things like how many minutes someone's spending with your product and But that's not the case for Headspace because more is not always better.

We want people to get better if they are coming to us with actual symptoms.

And so the optimization is health outcomes, not just more and more time in the

app.

Right.

So now we have this like dual mandate that we're really trying to drive.

Yeah.

And how do you measure outcomes?

Great question.

There are lots of different ways to measure outcomes.

Some are at a population health level.

Some are using third-party data sets or match data sets.

But one of the ways that apps like ours really try to hone in at the individual level is by self-reported data and taking snapshots of a user's self-reported scales of how they feel in terms of anxiety or depression, for example, and take those snapshots over time and be able to kind of correlate that with any interventions, treatments, or content that they've engaged with over that time.

Yeah, that makes a lot of sense.

So I'd imagine you're, you know, for a typical experiment, you're looking at both kind of traditional metrics as well as these kind of more outcome based.

Exactly.