Skip to main content
David Talby

David Talby

Oct 6, 2026

6:21
Now I can make it, yes.
6:23
Exactly.
6:24
Exactly.
6:24
It kind of says, okay, well, you know, this is going to be like, you know, 50 to 100, you know, person thing.
6:28
It's going to take, you know, several years to do.
6:30
And you go to marketing, you evolve it.
6:32
Now you can kind of say, yeah, I can just do that, basically.
10:09
So maybe we could talk a little bit about how that's so important, not for just technical teams, but just people that are using AI and their businesses, right?
12:42
and
12:43
very often if you have a benchmark like public qa that's been around for six years at this point all the public llms are basically trained on this data set as well right it has seen the data set it has seen papers about the data set it's in paper that say what's wrong with this data set so a lot of what it's doing is memorization Data combination is also a big issue with leaderboards.
13:05
And here's an example.
13:06
This is from the now defunct Hanging Face Open Medical LM leaderboard.
13:11
But you can see this is from kind of the top contender.
13:15
But basically, every time you see a model that claims 98%, 98.9% accuracy on a specific benchmark, with all likelihood, it really is just memorization right on that specific benchmark, and it's not going to generalize.
13:31
Other than being another form of cheating, the main problem here is that once you have data contamination, right, and basically kind of, you know, you've seen the, you know, you've seen the answers to the question and the answer to the test before you took the test, it leads to non-generalizable models, right? So it means that, yes, you know how to answer those specific questions, but even if you change the question slightly, you're going to fail.

7 MINS LATER

20:50
to

We value your privacy

We use cookies to understand how you use our platform and to improve your experience. Click “Accept All” to consent, or “Decline non-essential” to opt out of non-essential cookies. Read our Privacy Policy.