What would you say to those two points?

So I think there are ways in which, not just in education but generally, we are all different and we're all unique individuals and we've got our different preferences and different interests.

And then I think there are ways in which, as I've been saying all along, there are things that we share and I would say there are some aspects of cognitive architecture that we do all share.

And I would say that working memory, long-term memory distinction is a good one that holds constant.

And that doesn't mean we have to be one size fits all.

It does mean there can be kind of flexibility.

But I think similarly in the way that you would want a medical treatment to be aware of the aspects of biology that we all share.

Are we in the right place there? Most people say not.

expectations, lots of

testing.

High expectations, focus on standards.

removing the sort of element of teacher assessment and coursework, focusing on traditional subjects, all of those kinds of things.

And around about the same time, actually probably starting a few years before, Scotland went down a very, very different route and basically the opposite route.

And they had a programme and a curriculum called Curriculum for Excellence, which you would say in many ways was the opposite of what Gove did.

So it was more about things like 21st century skills, a lot of the skills-based things I talked about, not being explicit about the knowledge that had to be learned, much more of a focus on project-based learning, not as much of a focus on the sort of terminal exams.

Is it the case that humans are getting dumber?

Right.

And it is something that is not just developed in school.

It's also developed out of school and at home, just in the everyday language that you hear.

So it's something that also I think schools can justify if you think this is really hard for us 'cause it isn't just about what we do in school.

But that is what we need to be doing.

[laughs] We need to be retooling the whole of education around developing the powerful knowledge, the powerful background knowledge, the powerful vocabulary that students n- need to be able to read.

I also think, in parallel with that, we need to be very cautious about what can be achieved with reading strategies.

Yeah

Okay.

We've now dropped that down to two, 10%, and the AI does the other 18.

So every piece is getting 20 judgments and being seen 20 times, but now two of them are a human.

Okay.

But why that's so important is... two of them being human, that's still twice as many as with traditional marking.

And it's still a lot more than other AI systems where it's just the AI does everything.

So it's very important for us to have human oversight.

Why that's a preference for you?

And again, Daisy Christodoulou says, before you leap to conclusions, look at the data from Scotland and Wales.

It's the World Health Organisation's health behaviour in school-aged children.

And it has a whole battery of questions on mental health and well-being.

And it reports England, Scotland and Wales separately.

And it's been running for quite a long time.

It does ask about kind of mean life satisfaction.

And what it does seem to show is that mean life satisfaction for teens in England, Wales and Scotland does seem to have fallen, you know, across this century.

And this natural experiment comparing England with Scotland and Wales, when Scotland and Wales basically moved in the opposite direction in terms of the education systems, any sign of any differences there?

Uh, what-- uh, why is he interesting?

So Hong Xiuquan is really interesting because he's probably the instigator of the worst civil war, maybe the worst war in history.

It's not very well known in the West, and in fact, one of the leading American historians of this, of this war, this civil war, has said that he didn't even read about it till he was an undergraduate.

So it's remarkably kind of not, not very well known in, in the West, given how, how significant it is and how, how major a factor it is in Chinese history.

And Hong Xiuquan, he's basically a failed civil service candidate.

So, and this, this all takes place in, in the mid-nineteenth century in China, and at that time, China has these very rigorous exams to become a civil servant.

So if you want to become a civil servant in nineteenth century China, you sit these very, very tough exams.

What's wrong with that argument?

And West Ham are at home to Manchester United.

Yeah, absolutely.

So West Ham have been promoted.

We are back in the Premier League.

We've been back for a few seasons and we're going to leave.

So again, 2012.

The other thing I said about 2012 is obviously as well as West Ham getting promoted, it was the big Olympics year.

But

And we're gonna start in the World Cup in 1994, Ray Houghton for Ireland against Italy.

Yeah, absolutely.

So this was my first World Cup.

I was nine years old, so I was too young for the 1994 World Cup.

I don't really have any memories of that.

Um, and this is obviously a World Cup where England didn't qualify, but the Republic of Ireland did, and their manager was Jack Charlton.

Jack Charlton was a England World Cup winner, so I think a lot of people in England had a lot of time for him.

Yeah

And

when I say inconsistent, people often think when I talk about inconsistency with traditional marking, they think the problem is, oh, you've got like one marker who's just not very good and the other marker is very good.

And what we have to try and do is get the less good markers to agree with the good marker.

We've got some expert, and our aim is to get everybody in line with the expert good marker.

Now, that is part of the problem, but actually the problem is even harder than that.

The problem is not just that different markers disagree with each other.

The problem is that individual markers disagree with themselves.

so people who haven't read daisy's thing about the germany analogy tell people what you're saying there because it's very it's very small

If the evidence is so strong for this systematic way of teaching children how the alphabet works, why has it been so controversial? I'm Daisy Christodoulou.

I'm the Director of Education at No More Marking.

We're a provider of online comparative judgment writing assessments.

We do a lot of work on literacy.

So I think the really deep-rooted idea that's really misleading on this is the idea that reading and writing are natural.

And actually, that's quite an easy mistake to make because reading and writing are very similar to speaking and listening.

And speaking and listening are natural.

But if it... could interpret it would you have been confident the reason i ask is what i found fascinating about your journey you've been very open about this is you you were quite skeptical right and and what i love about your post now is you always refer back to like you said you know three years ago this is what i said here so back in 2022 were the kind of two separate issues whether the it couldn't read the writing and even if it could you wouldn't have been confident that it could do anything useful with it

Definitely, and I think those are two still separate issues.

So you're right, handwriting transcription is one thing, and then a marking is another.

And actually, to begin with, ChatGPT 3 didn't solve handwriting recognition, and actually, if we want to talk more about handwriting recognition, it is still a difficult problem.

It isn't actually completely solved, but it's way, way further than we thought in November 2022.

So actually, our initial trials with marking, we weren't getting it to transcribe.

We were using... you know, handwritten pieces that we'd had typed up by humans.

Yes.

Yeah,

pre-LLM, exactly.

And so when I said, you know, we've assessed nearly three million pieces of writing, whatever, the vast majority of what we've done, of what I'm talking about, has been human comparative judgment.

So human teachers making judgments about what they think is the better piece of writing.

In the last couple of years, we have experimented with adding LLM judgments into the mix.

And we've got to the point now where we think the LLMs are very good at comparative judgment.

They're not perfect.

Could LLMs do that in theory, though? Is that the magic of it? So

Uh, explain that, uh, that phrase.

A- absolutely.

So when it comes to education, I think anyone working in education will talk about the importance of feedback.

It's, uh, a very popular topic, I think, with teachers a-a-and with students, the idea that you need to get some idea of how you're doing to improve.

But feedback is obviously a very, very important concept beyond education, and, you know, the origins of that, it, it's not a term that's used a lot now, but maybe cy-cybernetics, uh, you know, the origins of these sort of control systems, uh, the origins of information technology, a, a lot of the origins of, of those fields of study are in the concept of feedback, of how you change a system based on, on inputs.

And so a, a really nice th- way of thinking about feedback in that more general sense is to think about a thermometer and a thermostat.

So a thermometer is a measurement tool.

[laughs]

Why don't you talk about comparative judgment and how you use it in assessing writing, and then I wanna talk about how one might, even though I think it's a bit far-fetched, but extremely interesting, how you might apply it to, say, enforcing, uh, football regulations.

Absolutely.

So yeah, comparative judgment is what I do as my, my day job.

So the organization I work for, Nomore Marketing, we use it to assess writing.

So I'll just say a bit about what it is.

Um, so comparative judgment, it rests on the psychological principle that as human beings, we are not very good at making absolute judgments.

We are much better at comparative judgments.

How would you describe it? A, a landscape, uh, uh, something of a continuum of what is a handball.

So therefore, you have to go to referees call, surely.

Yeah.

So you're absolutely right.

And I think that you're right to pick up on some of the deep issues around this, which is the faith we place in technology in our society.

And the way we almost, we just assume that it can't be wrong.

When in fact, anyone involved in technology and in measurement will tell you that there is error in every measurement.

There is uncertainty in all use of any kind of measurement technology.

Do we say that is or isn't handball or are we showing two instances that we judge which is more handball? Right.