Sample size determination

Yeah.

And you, and you mentioned there kind of the sample size that we're, we're treating these players with.

And on the one hand, people are so fast to kind of jump on the positives with players and, um, but then so patient with some other players.

So obviously you mentioned Ronald Araujo there, kind of the, the loan player that everyone's fallen in love with.

That's why analysts call it .5.

You can only see trends.

You can't look at seven games or whatever it is and say that's a big enough sample size to say conclusion.

You're looking at trends and what might be happening.

So if we have 15 studies, we'd have 15 data points in the meta-analysis, right? Right.

And the longer term that the study was and the larger the sample size for each one of those dots, the more precision you have on the true mean of what that dot is.

I guess one thing I will mention that I didn't really expect to talk about today, but I think folks would probably be interested to hear is, well, if we really need to combine data across such a number of studies before we can really start forming valid opinions, then maybe it's a reasonable question to ask, why do we publish these one-offs on monitoring turkeys or something else for several years on one or two sites? Um, and, and part of that gets into the logistics of the job, right.

Of, of being a scientist, you know, one of the ways that you're evaluated is by how many, by how many publications you're able to produce per unit of time.

Yeah, after we've just had the worst episode we've ever recorded, please rate us.

Before you leave us a rating and review, go back and listen to a sample.

What would be a valid sample size, Marcus? Ten episodes? I would say ten at minimum.

Listen to ten at minimum so that you have a valid sample size to base your inferences off of.

This decision shapes how you manage the collection phase.

If your goal is to hit a precise sample size, participant count is your anchor.

If your timeline is rigid, the deadline becomes your hard stop.

This choice defines the boundary of your data collection.

It feels like a binary decision, but in practice it's often a negotiation with stakeholders.

Exactly.

I've seen clients want a hard deadline, but then panic when the sample size is too small to draw conclusions.

That's the trap.

Okay, how many millions of people are maybe going to nike.com to buy shoes? A lot.

So you can quickly get a sample size, which means y- you know, maybe you run it for a, a week and there's millions of people coming, and you can very quickly detect there's a difference between the performance of two versions of pages.

On the B2B side, how many people are shopping for a new, uh, cybersecurity software today? Some, okay, and in fact, there's quite a few, but not as many are shopping for shoes.

So one way to get results quicker is to make big changes.

And then you gotta follow your exit rules.

And then you have to be, uh, in a place where you have repeatability and a sufficient sample size.

We'll talk more about sample size and statistics in a few minutes.

And the trader exactly, right? Invalidation comes over to here.

The trader knows, what happens if I'm wrong? They know where they're gonna get out, they know where they're gonna get out, they know what size they got in with and what size they'll get out with, and they understand that no matter what, they can't control the outcome.

In theory, they would all get you to the same expectancy, but they don't arrive in neat, easy order.

They arrive randomly.

Now, sample size matters, right? When you're doing your backtesting, when you're putting on your trades, one trade at a time tells us almost nothing.

And let's say that you did two more, right? And you hit one on heads, another on heads, now you're 50/50 win rate.

And then you hit another on tails, now you're back under 50% win rate, right? Can your edge go from positive to negative? It can, but only if I have a small sample size.

The law of large numbers kind of negates that.

So that's why we wanna have as big and as many frequency as possible.

That's not going to work for my studio.

I understand the sentiment, but they're often drawing a conclusion from a sample size of effectively one.

It's them.

And that is their world.

If they say no, well, you have a data point.

And here's the thing.

Make sure your sample size is big enough that it's not one.

Do it 50 times, 100 times, and see then what works.

So you just need to learn how to think and then you can take your time and read how did they get this information and does the information that they observed actually lead you to certain conclusions or not?

And how important would you say this idea of so sample size in science and in research publications is something that's like this highly sought after value, a large sample size.

What is your thought on the idea of N equals one and just personal anecdotal experience of how does something affect you or make you feel?

Right.

Well, you're entering us into a very slippery slope kind of a world here bringing up statistics because sample size has to do with statistical power, right? Do you have enough subjects in your model that you can say that your findings are not just by chance alone? And Of course, when we apply statistics at all, we're saying that the effect is not so powerful as it's going to happen 100% of the time.

It's only going to happen part of the time.

So we're looking at more subtle effects and we're looking probably at things that have multiple causal factors.

And we're gonna need that control experiment comparison group.

Mm-hmm.

And then, so the sample size was 206 and that was planned to detect a sub-distribution hazard ratio of 1.75. And so they ended up enrolling 205 of the planned 206 patients into the study.

And I think that's, yeah, that's the standard hazard ratio that kind of gets everybody into, you know, discussion because it really That's the number that they've generated based on a really quite a significant clinical difference between the two groups.

And possibly it would have been more subtle.

That

probably makes sense as how to present it.

So essentially, as I said, they'd set it up to have this standard hazard ratio of 1.75 and then they worked backwards to get a sample size of 206, of which they got 205.

And so that was based on getting an increase of successful weaning from 50% to 70% with 80% power.