What did these studies actually find, and why do you think these studies spread so quickly?

Yeah, so one of the major claims made in both the meta-analyses that we analyzed for this paper was that ChatGPT enhances academic performance for learners, and both meta-analyses found this with pretty large effect sizes.

And what they were looking at specifically was, uh, research that compared a condition where participants learned with ChatGPT to a condition where students learned without ChatGPT.

And so from, you know, looking across all of these different studies, they were able to make this claim.

One thing I do wanna note is that one of the papers that we looked at in our critical analysis did end up being retracted, so the, the Wang and Fang paper, it was retracted back in April due to concerns related to discrepancies in the meta-analysis, so one of them is no longer being used, uh, as evidence.

But that being said, the research that they looked at, uh, they still are in the, the zeitgeist, but through our investigation, it seems like these findings were based on studies that compared conditions that differed in more than just whether ChatGPT was being implemented in these conditions.

Uh, which makes it very difficult to know whether ChatGPT is actually useful or beneficial in learning.

What tips would you have around how we should do these media comparisons, how we should do these meta-analyses going forward?