Head Start needs reforming, not dismantling | Episode 1032 of The Education Gadfly Show

Well, we are kinda in that same neighborhood.

We're gonna look at a new RCT, love these, and ask a question whether AI tutoring actually improves math achievement for struggling kids.

And then they wanna know, okay, if it does, then what might be driving those gains? The authors kind of, you know, excitedly say it's the largest and longest experimental evaluation of an I- AI tutoring program to date.

However, as we get into it, we realize they kinda walk back that a little bit because it's actually very programmatic, so it shouldn't be seen as AI tutoring versus no AI tutoring because there's a lot of other programmatic stuff going on in the control condition.

So I'll talk about that in a second.

Anyway, it's a two-year cluster randomized trial across 18 Tennessee middle schools.

They're randomizing the classroom to either Khan Academy with a AI tutor-embedded Khanmigo, which is one of the more popular ones and one of the first ones out of the gate, I think, that schools started adopting, or what they call business as usual remediation math instruction.

Of course, we wanna know what that is.

But first, just a little bit about Khanmigo.

I think most of our, our listeners are aware it's designed to help kids conversationally, again, not giving them the answer, but sort of lead them to water.

That's the big thing.

But on this treatment condition, we're told that the also, the... and this is the other programmatic piece, those kids also get their existing MAP scores are used to place them in a specific point on this Khan Academy Khanmigo learning trajectory, which is kind of another pu- piece of the puzzle, again, that sort of clouds the waters a little bit.

The business as usual is a mix of teacher-led practice, other math software without AI tutoring.

So it's not no technology, it's this, they can actually use computer-assisted learning, but it doesn't have the platform of the AI tutoring trying to talk to them, and so on and so forth.

And then they actually looked into the control classrooms, and a lot of them were using Waggle, which was new to me.

But Waggle is an adaptive practice program.

Again, it's from their district curriculum provider, and it, again, doesn't have the AI component to it.

All right.

The participants were all mandated to attend the district's existing daily remedial math sessions.

This was 25 to 40 minutes in small groups that all below grade level students had to attend.

So not optional, it's mandated, uh, every day, 20 to 45 minutes, uh, 40 minutes.

And then they looked at it for two years, and they had about 6,900 student by term records.

So in other words, if a kid appeared in two terms, that was two records.

All right.

What did, what happened? On achievement, the program produced modest but real gains.