Making Sense of PISA 2025 with Montserrat Gomendio (Ep 81)

So do you have anything you wanna add about PISA 2025 or PISA in general before we close off this discussion?

Yeah.

I want to be a bit more precise about my criticisms about PISA interpretations.

I think PISA data are very valuable, and there are many, many criticisms that I won't go into.

You know, whether the countries are comparable, whether you should measure math, science, and reading or other things, and there are many, many criticisms, uh, that most people are familiar with.

Despite all those criticisms, I think the data that PISA provides are very, very ba-valuable and probably the most reliable source of information that we have on the quality of education systems.

The issue that I have is with the way PISA analyzes and interprets the data.

Now, it's been said a thousand times, but for reasons that I fail to understand, it doesn't change.

PISA only runs many, many, many correlations in every cycle.

And as most people know, you cannot draw causation from a correlation.

In some cases, it's even worse because the correlation, like the one I mentioned about the use of AI and science scores, is not significant, clearly not significant, almost a flat line, and still PISA develops a narrative around that particular result, which is very confusing for people because most people don't have the time that I spend [chuckles] looking in detail at the graphs, at whether it's statistically significant, what the conclusions are, and so on.

So I think the issue with PISA for me is the analysis are not the right ones.

I don't know why they don't move from correlations to more complex analysis that would give us more reliable information on what's going on.

But also, then they have this priority, which is to have media impact, to have a narrative that is catchy, that everybody would understand, "Oh, yeah, it's the mobiles, it's the screens, and those are doing a lot of damage." But what are the conclusions from PISA for policymakers? Because the OECD was created to give advice to governments, as I said, and that's what I did during five years at the OECD.

And I think most of the advice provided to the governments, it's very, very valuable.

And if the analysis were more elaborate and the interpretations more accurate, then I think the kind of advice that it would give to governments would be really valuable.

Whether governments follow the advice or not, that's another issue.

But the OECD has often argued when it's become clear that countries don't improve, before COVID and all of this, when countries were still not improving and even declines were starting to become apparent, that the problem was governments, that they were not listening to the OECD.

Well, I think that's very unfair.

I think most governments do listen to the OECD, and one of the examples that most people understand is most governments have followed the advice that the policies implemented by Finland should be copy-pasted all over the world, and that has been a disaster.

So what I'm trying to say is the data are incredibly valuable.

There are other surveys, PIRLS and TIMSS, but none of them, I think, have the amount of information and also the fact that it's at the 15-year-olds, it's at the end of compulsory education.

It's like a summary of everything that they have learned over compulsory education, that it's really important.

I'm not saying that it's not important to understand what students in primary learn.

It's just that this is like a summary of the whole education system during the compulsory stage.