XX

How ChatGPT Can Fail Lawyers – and Everyone Else

Using the latest and most powerful technology tool for business can have its drawbacks

Artificial intelligence is all the rage. Watch the news, skim your social media sites, and that may be all you see. AI — specifically ChatGPT — can create marketing plans, write our kids’ college essays, and even compose music. But how well it performs can vary wildly and is very much up for debate.

Recently, I researched how a fellow lawyer learned that relying on ChatGPT could have disastrous results.

Background of the Case

On Aug. 27, 2019, Roberto Mata was on an Avianca airline flight from El Salvador to New York’s Kennedy International Airport when he was allegedly injured by a metal serving cart that hit his knee. Mata filed suit against the airline in February 2022.

Several months later, Avianca requested that the Hon. P. Kevin Castel, a Manhattan federal judge, dismiss the case since the statute of limitations expired. Mata’s attorneys responded in April 2023 by providing a 10-page brief that cited several court cases relevant to the lawsuit and the statute of limitations issue. The problem? The airline’s attorneys could not verify any of these cases, nor could Judge Castel. The court cases in the brief were fictitious.

How It Happened

It seems that the attorney had used ChatGPT when he performed his legal research. The AI program produced six court cases, including the courts’ locations, docket numbers, and filing dates. Yet not a single one of them was real.

After being caught, the attorney explained that he never meant to deceive anyone and threw himself at “the mercy of the court.” His affidavit claimed he had not used ChatGPT before and did not realize it could produce inaccurate information.

However, it is interesting to note that on ChatGPT’s website, under the category of “Limitations,” the platform admits to the following:

• It may occasionally generate incorrect information;
• It may occasionally produce harmful instructions or biased content;
• It has limited knowledge of the world and events after 2021.

You might think that a lawyer would pause before proceeding.

The lawyer told the judge he used ChatGPT to supplement his research but now regrets doing so. However, he tried to be careful and even asked the AI platform, “If all the cases were real?”; ChatGPT assured him they were.

The lawyers for Avianca suspected that the cases were bogus since they did recognize any of them, and they are familiar with airline-related court decisions. Judge Castel also did some digging. He contacted the 11th Circuit clerk, who confirmed a docket number was from a completely different case.

The Ruling

On June 22, Judge Castel imposed $5,000 fines on two attorneys from the firm that used ChatGPT. Although a nominal penalty, the fines are intended as a warning to other lawyers about using AI for legal purposes.

He explained: “Many harms flow from the submission of fake opinions. The opposing party wastes time and money in exposing the deception. The Court’s time is taken from other important endeavors. The client may be deprived of arguments based on authentic judicial precedents. There is potential harm to the reputation of judges and courts whose names are falsely invoked as authors of the bogus opinions and to the reputation of a party attributed with fictional conduct.”

In addition, he cautioned that such careless practice “… promotes cynicism about the legal profession and the American judicial system. And a future litigant may be tempted to defy a judicial ruling by disingenuously claiming doubt about its authenticity.”

How ChatGPT Works

ChatGPT — as well as Google’s Bard and Microsoft’s Bing — can have conversations that seem human. These AI platforms can write code, draft emails, and even help plan parties. It works by scanning billions of pages of content from the internet, including blog posts, news stories, tweets, and public libraries. After ingesting all this information, it produces responses based on guesses of what text fragments should follow one another.

ChatGPT likely consumed enough information in this legal case to create a brief — but riddled it with a fascinating collection of facts and names from myriad other cases.

Where AI Might Lead

For some time now, the roofing industry has been discussing the hopes and the hazards that software like ChatGPT can offer. So this case is a cautionary tale. You cannot merely search and paste the output into an important document. Even before AI, believing information on the internet required a certain amount of discernment. And there is an increasing worry that the rapid growth of AI could be used to spread propaganda and other misinformation.

We clearly cannot rely on ChatGPT to do our thinking for us. On May 30, 2023, a group of industry leaders issued a one-sentence statement warning of the dangers of AI: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks, such as pandemics and nuclear war.”

Released by the Center for AI Safety, the statement was signed by more than 350 AI executives, engineers, and scientists — the people who are building the technology in question.

Does AI bring global risks that may impact all of us? Could be.

Lawyers and other professionals should use caution when relying on AI. Although it can be a helpful tool in some situations, it clearly has its risks and limitations.

Disclaimer: The information contained in this article is for general educational information only. This information does not constitute, nor intend to offer, legal advice. Neither should it be relied upon as legal advice for your specific factual pattern or situation.