AI Hallucinations in Court The Sullivan & Cromwell Lesson

Some big news involving an AM Law 50 firm, Sullivan & Cromwell, global law firm, and they filed a brief in court with some citations to cases that didn't exist or some citations that were just plain inaccurate.

The case didn't stand for the proposition which they claimed.

And they got caught and had to apologize to the lawyers on the other side and throw themselves on the mercy of the court, and what's gonna happen to it remains to be seen.

But I wonder, as many of us have wondered over the past couple of years, why does this keep happening? What's the...

By, by now everybody knows that this is a, a flaw in how these large language models work, and yet people keep doing it.

Jennifer, why?

Hi.

[laughs]

I find it fascinating that how quickly th- this all happened because normally when a new technology's introduced, it takes a little while before someone gets in trouble.

But as you guys know, and you out there probably know, generative AI was introduced in 2022 with ChatGPT, and very quickly after, we had the Mata case in which someone got very publicly in trouble for dabbling and using AI to dabble.

So when you dabble, certainly you are more likely to make a mistake.

Of course, in this case, we don't have dabbling.

We have some, a firm that had a very clear AI use policy, and if you read the articles about it, they make it clear that the policy was ignored.

So when policies are created but ignored, that leads to problems, and I think every firm that is using AI should have an AI use policy.

But why does it happen? It could be laziness.

It could be confusion.

It could be a failure to understand the limitations of the tools.

Time pressure.

I think time pressure's a big one.

I know one case happened because a woman had just lost her husband, my sympathies to her of course, and she was rushing to get the work done.

We all have so many pressures on us.

I think it's very easy to, with the confidence, AI is so confident sounding, right? It, it, it gives wrong information very confidently.

[laughs]

Some big news involving an AM Law 50 firm, Sullivan & Cromwell, global law firm, and they filed a brief in court with some citations to cases that didn't exist or some citations that were just plain inaccurate.

The case didn't stand for the proposition which they claimed.

And they got caught and had to apologize to the lawyers on the other side and throw themselves on the mercy of the court, and what's gonna happen to it remains to be seen.

But I wonder, as many of us have wondered over the past couple of years, why does this keep happening? What's the...

By, by now everybody knows that this is a, a flaw in how these large language models work, and yet people keep doing it.

Jennifer, why?

Hi.

[laughs]

I find it fascinating that how quickly th- this all happened because normally when a new technology's introduced, it takes a little while before someone gets in trouble.

But as you guys know, and you out there probably know, generative AI was introduced in 2022 with ChatGPT, and very quickly after, we had the Mata case in which someone got very publicly in trouble for dabbling and using AI to dabble.

So when you dabble, certainly you are more likely to make a mistake.

Of course, in this case, we don't have dabbling.

We have some, a firm that had a very clear AI use policy, and if you read the articles about it, they make it clear that the policy was ignored.

So when policies are created but ignored, that leads to problems, and I think every firm that is using AI should have an AI use policy.

But why does it happen? It could be laziness.

It could be confusion.

It could be a failure to understand the limitations of the tools.

Time pressure.

I think time pressure's a big one.

I know one case happened because a woman had just lost her husband, my sympathies to her of course, and she was rushing to get the work done.

We all have so many pressures on us.

I think it's very easy to, with the confidence, AI is so confident sounding, right? It, it, it gives wrong information very confidently.

[laughs]