We're Overestimating Medical AI — and Underestimating the Harm (Jessica Morley, Yale)

And also in your series, you mentioned the case from years ago when DeepMind was working in the UK and using hospital data.

There was a, it was a sco- it was a whole scandal, the fact that they basically got access to the data, but the argument was that basically the hospital gave them the data in order to provide direct care.

So that was the justification that felt okay to everyone.

So what made me wonder here is, to which extent do you think that ethics can be bended so you find the right conviction for yourself that you're doing the right thing? It's a very tricky thing, especially because rules also change.

Our perception as society changes in terms of what's acceptable and what isn't.

Yeah, absolutely, which is why I think you need both ethics and the law.

So in that DeepMind and the Royal Free, it was the Royal Free Hospital in this particular instance.

To be clear, nobody did this on purpose.

It was a sort of misunderstanding, I think, of what, what had happened.

But the law requires you to determine whether or not, in the UK anyway, to determine whether or not data is being used for one of three purposes, which is direct care, which is essentially I am being directly looked after.

It's something to do with your prescription, or you're being seen by a clinician.

Or there is research, or there's what's called service analytics or sort of service performance.

Now, each of those different categories has different requirements with regards to consent, and this is where things went wrong because for direct care, you don't need direct patient consent to share the record, and that's because it's predicated on this idea that if I am in a hospital and I need to see both a head doctor and a foot doctor, it would be inefficient to ask the patient every time you need to hand over the clipboard.

Whereas with in the data space, that starts to break down, and the fact is really that those three categories in the AI space are collapsed.

We don't have a really clear boundary between direct care, service analytics, and research.

They're all sort of one, which is essentially why you need ethics, because ethics allows you to do contextualization and to keep pace with that societal expectation changes in the way that you just described, in the way that the law often can't.

The law is pretty slow to move.

It's often left open to some degree of interpretation because there needs to be a degree of flexibility, and creating, as you say, hard rules often makes things more risky rather than less risky 'cause it allows people to jump through gaps slightly more easily, and hence the Swiss cheese model of risk.

Whereas that sort of ethical context, which is really about asking should we rather than could we, is what helps there.

So had there been more, I think, ethical discussion in that particular scenario, and asking patients, for example-Does this feel like something you're happy with? Does it feel as though it's something that's socially acceptable? I think that probably would have got you a better answer than the hospital misinterpreting the law.

So there is a really definitely a case where people can bend things to their own will.

We talk about things, so we've talked about this problem, ethics shopping, which is essentially when people go looking for the lowest common denominator, or what's your interpretation of transparency versus your interpretation of transparency.

Yours is much easier to be compliant with, so I'm gonna just pick yours.

That's not really how ethics should work.

And so the...