A Million Families Waiting: Can AI Help? with Ben Cosh

So we've done loads, I mean, I guess this is our business through and through.

So we've done loads and loads of experiments on exactly that.

And to, I guess if I tell you a little bit about the story of the research we've done in that space, that might be interesting for people.

So when we started out about two and a half years ago investigating this, we pulled off the back of Google the 2,000 questions that people were asking about SEND, autism, ADHD.

A whole range of different subjects, a whole lot of different questions from Reddit and from Facebook.

And we built a piece of machinery software where we could throw those 2000 questions at all the AIs that were available at the time.

And there were lots of them.

And then we had a scoring system that said, is the response safe? Is it useful? Can I actually do something practical with that? Is it aligned to what the evidence would say is a good thing to do, given the information that it knows at the time? Some of the AIs can be quite good some of the time but what we found there is that lots of them will maybe give you five or six pages of detail but it's irrelevant for your particular situation because it doesn't have enough nuance around what's going on for that person.

Or it might be a bit sycophantic and say, oh, you're doing a, you know, this is really, really brilliant.

When actually it would be probably more healthy if it challenged you a little bit and said maybe try to do this rather than do that.

So I guess my point in there is, I was totting it up earlier, we think now there are 628 different AIs out there.

So if I was to, I guess, land one message from this conversation, it's like, we mustn't think of AI as a thing.

Each AI is different.

You know, you go to ChatGPT, it's different from Grok, it's different from Gemini, it's different from Claude, it's different from DeepSeek.

And they're all good at different things.

And what none of them are is doctors that can diagnose us, despite the fact they give you the confidence that they can, they're so confident, aren't they? And what none of them are is structured.

So if you're trying to, what's a good example? If you're trying to understand someone's sensory processing differences, They don't go through all the senses and say, I remember when I did a sensory processing test five years ago, the test was actually pretty rubbish and there were five senses because I thought there were five senses.

It turns out there are 13 because you've got your interoception and you've got your ability.

You've got like, how do you relate to time? How do you relate to your experience in your body? How hot and cold do you feel? How do you deal with what's going on in your mouth? How do you deal with toothpaste and chili and all these things? So we experience the world in lots of different ways.

And if you say to an AI, how do I do this? What you might get is something that's trained on Reddit.

And Reddit's great fun, but it's not very scientifically valid.

It's not being thought through and tested by psychologists.

so the trick i think is to find the best ai for the job um and the job if the job's having a laugh and quizzing reddit that's cool that's fine but that might not be where you want to put your kids health information

well i think reddit would be like the old wives tale yeah yeah oh my mom used to say that kind of reddit's world one thing i found is um i've not used many ais because generally certain companies i'm not a final so i will stay away from their products i started off with chat gpt and i will describe that as a very american

guy