Showing posts with label AI hallucinations. Show all posts
Showing posts with label AI hallucinations. Show all posts

Tuesday, May 19, 2026

Book on Truth in the Age of A.I. Contains Quotes Made Up by A.I.; The New York Times, May 19, 2026

 , The New York Times; Book on Truth in the Age of A.I. Contains Quotes Made Up by A.I.

"The author of a nonfiction book about the effects of artificial intelligence on truth acknowledged on Monday that he had included numerous made-up or misattributed quotes concocted by A.I.

The author, Steven Rosenbaum, whose book “The Future of Truth” was released this month to great fanfare, incorporated more than a half-dozen misattributed or fake quotes in sections of the book reviewed by The New York Times.

The Times asked Mr. Rosenbaum about the quotes on Sunday and Monday. On Monday night, Mr. Rosenbaum acknowledged in a statement that the book had “a handful of improperly attributed or synthetic quotes” and said that he had started his own investigation.

He said that the inclusion of the incorrect quotes was an accident and that he had “no intention of fabricating any viewpoints” while writing the book.

“As I disclosed in the book’s acknowledgments, I used A.I. tools ChatGPT and Claude during the research, writing and editing process,” Mr. Rosenbaum said in the statement. “That does not excuse these errors, of which I take full responsibility. I am now working with the editors to thoroughly review and quickly correct any affected passages; any future editions will be corrected.”

“The Future of Truth” was published by an imprint of BenBella Books and distributed by Simon and Schuster. BenBella Books, which operates independently of Simon and Schuster, did not respond to a request for comment. Simon and Schuster declined to comment."

Sunday, May 17, 2026

AI won’t replace lawyers. It will create more of them.; The Washington Post, May 17, 2026

Damien Charlotin , The Washington Post; AI won’t replace lawyers. It will create more of them.

"The replacement story often rests on a particular picture of what a lawyer does: reading documents, applying rules and producing text. Since AI can read, apply rules and produce text, the argument goes, lawyers are cooked. That picture is not entirely wrong, but it is the perception engineers have always had of the legal domain: Feed in the facts, apply the rule, return the output. Yet the reason lawyers exist (and command high prices for their services) is that law is shot through with ambiguity. If the rules ran themselves, no one would need us. Every step in the chain — reading, applying, producing — involves choices, some of which are genuinely difficult.

A better way to think about jobs is as bundles of tasks. Some bundles are loose: A job composed of a handful of discrete, repetitive, well-specified tasks can be peeled apart and the tasks automated one by one. Other bundles are tight, because the tasks reinforce one another and cannot be cleanly separated. The key example here is offered by radiologists, long predicted to be facing extinction due to AI. Despite the dire forecasts, their numbers keep growing, and they keep commanding ever-higher salaries.

Legal work is also hard to neatly separate. For instance, doing legal research and evaluating an argument are, for an experienced lawyer, often the same mental activity: A lawyer checks the argument by writing it. Pull those tasks apart, hand the writing to a machine, and verification suddenly becomes a separate, deliberate, expensive act — at least if you want to avoid landing in my database of courts sanctioning parties for filing “hallucinated” material. In fact, an irony is that automating the easy parts of a job often makes the hard parts harder, not easier."

Law Schools Implement AI to Focus on Ethics and Technology; Los Angeles Times, May 17, 2026

 David Nusbaum, Los Angeles Times; Law Schools Implement AI to Focus on Ethics and Technology

"Over the last two years, Loyola Law School in Downtown Los Angeles has incorporated AI into six courses. It’s a sign of a growing trend where law firms are looking for attorneys who can utilize the technology to improve efficiency. While law schools have constantly looked to update coursework to keep curriculum updated as laws are updated, the application of generative AI to the practice of law is the biggest change that has happened in generations, according to Rebecca Delfino, associate professor of law at Loyola Law School...

Delfino is one of several professors who have integrated AI into their coursework. She is involved with two courses specifically focused on the ethical implications of generative AI and the legal practice.

In a first-year civil procedure course, students are divided in half, with one group an analog approach that relies on textbooks and class notes while the other half uses generative AI technology. The results are compared to see where the technology is effective and ineffective. The goal is to use AI as something that is additive rather than giving over too much authority and power, according to Delfino. For many exercises, there are six or seven AI models that are tested and compared.

Students understand that they need the AI skill set to make themselves a more attractive candidate, no matter what area of law they practice. It can be used to draft documents, conduct legal research and assist with discovery. Chatbots are tested for hallucinations, and the drawbacks are identified."

Wednesday, May 13, 2026

Most U.S. doctors are quietly using this AI tool. Few patients know about it.; NBC News, May 13, 2026

  , NBC News; Most U.S. doctors are quietly using this AI tool. Few patients know about it. 

"Almost two-thirds of physicians — or roughly 650,000 doctors — in the U.S. actively use OpenEvidence, while another 1.2 million use it internationally, OpenEvidence representatives said. With its quick and tailored replies, OpenEvidence has become an AI-era equivalent of consulting a colleague for their expert opinion, though the software can also write patient discharge notes and provide custom study tools for doctors’ medical exams.

“Sixty percent of all the searches are about how to make clinical decisions,” said Jena, who is currently examining 90 million OpenEvidence queries submitted since 2024 as part of a new research project. “The physicians are asking: For this particular patient, or with this profile, this condition, maybe other comorbidities that they have, what’s the right treatment?”

Yet with OpenEvidence’s skyrocketing popularity, some experts worry about potential hallucinations or incomplete answers, a lack of rigorous scientific studies on the tool’s patient impact, and the potential for doctors’ critical thinking and evaluation skills to erode with increased OpenEvidence use and dependence.

But many in the medical world see OpenEvidence as a time-saving tool that can improve patient care."

Thursday, May 7, 2026

Google’s AI Summary Invents State Ethics Rules… And It’s Not A Hallucination Problem; Above The Law, May 6, 2026

  Joe Patrice , Above The Law; Google’s AI Summary Invents State Ethics Rules… And It’s Not A Hallucination Problem

"If you’re a Pennsylvania lawyer wondering whether you need to disclose AI use in your court filings, Google’s AI summary has an authoritative answer for you. It’s a wrong answer, mind you. But authoritative!...

If one opens an incognito window and searches Google for “Pennsylvania AI disclosure lawyers,” the AI-generated summary will explain that “Key developments include mandatory disclosure of Generative AI (GAI) in court filings.” Throw in “August 2024” because you vaguely remember seeing something about AI on that date and the result reads “As of August 2024, Pennsylvania mandates explicit disclosure of AI use in all court submissions, making transparency a mandatory filing requirement.”

None of that is true.

The Legal AI Governance tracker, an invaluable tool maintained by Brian Alenduff of Desired Path Consulting, provides a comprehensive rundown of Pennsylvania’s AI rules. There are standing orders in some courtrooms, and the state supreme court issued a rule governing court personnel only, but as for the state of Pennsylvania writ large, there is no statewide rule as of now. The tracker notes that what Pennsylvania does have is Joint Formal Opinion 2024-200, a 2024 advisory ethics opinion from the Pennsylvania Bar Association and the Philadelphia Bar Association flagging AI as a competence issue under existing rules. But that opinion explicitly states that it is “advisory only and is not binding.” The ABA’s own 50-state survey classifies Pennsylvania as “court dependent.”...

Hallucinations are all the rage right now, but over the long haul the greater AI risk will be an unfailingly credulous bot elevating and validating mistakes until the error gets picked up as reality."

Wednesday, April 29, 2026

Copyright Infringement Suits Loom With Unchecked AI Vibe Coding; Bloomberg Law, April 29, 2026

Christopher Suarez, Bill Toth, Anthony Pericolo, Bloomberg Law; Copyright Infringement Suits Loom With Unchecked AI Vibe Coding

"Deferring the job of software coding to artificial intelligence doesn’t immunize that code from copyright risk—it could even increase it, if the person directing the coding has limited oversight over the result.

This is particularly true with “vibe coding,” where developers use high‑level natural language prompts to generate code using AI models, often with limited manual review or modification of the resulting code.

Just as lawyers should check for “hallucinated” citations when writing with large language models, engineers and software development managers need to have human and technical monitoring protocols to account for infringement and licensing risks."

Wednesday, April 22, 2026

A.I. ‘Hallucinations’ Created Errors in Court Filing, Top Law Firm Says; The New York Times, April 21, 2026

  , The New York Times; A.I. ‘Hallucinations’ Created Errors in Court Filing, Top Law Firm Says

Sullivan & Cromwell apologized for submitting a court document that had fake citations created by artificial intelligence.

"An elite Wall Street law firm has apologized to a federal judge for submitting a court filing replete with errors created by artificial intelligence, including “hallucinations” that fabricated case citations.

The A.I.-generated errors came in a recent motion in U.S. Bankruptcy Court in Manhattan and were discovered by lawyers from an opposing firm, Andrew Dietderich, a partner at Sullivan & Cromwell, wrote in a letter to Judge Martin Glenn on April 18."

Friday, April 17, 2026

AI Is Getting Smarter. Catching Its Mistakes Is Getting Harder.; The Wall Street Journal, April 14, 2026

  

Katherine Blunt , The Wall Street Journal ; AI Is Getting Smarter. Catching Its Mistakes Is Getting Harder.

As chatbots and agents grow more powerful and ubiquitous, recognizing the moments when they go rogue can be tricky


"Chad Olson was confused when his Gemini artificial-intelligence chatbot told him he had a family reunion planning session marked on his calendar."

Monday, March 16, 2026

The dictionary sues OpenAI; TechCrunch, March 16, 2026

Amanda Silberling, TechCrunch; The dictionary sues OpenAI

"Encyclopedia Britannica and Merriam-Webster have filed a lawsuit against OpenAI, alleging in its complaint that the AI giant has committed “massive copyright infringement.”

Britannica, which owns Merriam-Webster, retains the copyright to nearly 100,000 online articles, which have been scraped and used to train OpenAI’s LLMs without permission, the publisher alleges in the lawsuit.

Britannica also accuses OpenAI of violating copyright laws when it generates outputs that contain “full or partial verbatim reproductions” of its content and when the AI lab uses its articles in ChatGPT’s RAG (retrieval augmented generation) workflow. OpenAI’s RAG tool is how the LLM scans the web or other databases for newly updated information when responding to a query. Britannica also alleges that OpenAI violates the Lanham Act, a trademark statute, when it generates made-up hallucinations and attributes them falsely to the publisher."

Monday, February 16, 2026

AI legal advice is driving lawyers bananas; Axios, February 9, 2026

 Emily Peck, Axios; AI legal advice is driving lawyers bananas

"AI promises to make work more productive for lawyers, but there's a problem: Their clients are using it, too.

Why it matters: The rise of AI is creating new headaches for attorneys: They're worried about the fate of the billable hour, a reliable profit center for aeons, and are perturbed by clients getting bad legal advice from chatbots.

Zoom in: "It's like the WebMD effect on steroids," says Dave Jochnowitz, a partner at the law firm Outten & Golden, referring to how medical websites can give people a misguided understanding of their condition."

Thursday, January 8, 2026

Judges are identifying suspected AI hallucinations in Pa. court cases — including one at the highest levels; Spotlight PA, January 7, 2026

  

Sarah Boden, Spotlight PA; Judges are identifying suspected AI hallucinations in Pa. court cases — including one at the highest levels


"Veteran attorneys with a track record of arguing high-profile cases submitted an error-filled brief to one of Pennsylvania’s appellate courts, raising questions from a judge about their use of artificial intelligence...

“Your credibility is such an important part of what a lawyer is to bring to the case,” said Vanaskie. “If the lawyer is not verifying what's being submitted, their credibility is shot.”"

Sunday, December 14, 2025

Publisher under fire after ‘fake’ citations found in AI ethics guide; The Times, December 14, 2025

 Rhys Blakely, The Times ; Publisher under fire after ‘fake’ citations found in AI ethics guide

"One of the world’s largest academic publishers is selling a book on the ethics of AI intelligence research that appears to be riddled with fake citations, including references to journals that do not exist.

Academic publishing has recently been subject to criticism for accepting fraudulent papers produced using AI, which have made it through a peer-review process designed to guarantee high standards.

The Times found that a book recently published by the German-British publishing giant Springer Nature includes dozens of citations that appear to have been invented — a sign, often, of AI-generated material."

Thursday, December 11, 2025

AI Has Its Place in Law, But Lawyers Who Treat It as a Replacement Can Risk Trust, Ethics, and Their Clients' Futures; International Business Times, December 11, 2025

Lisa Parlagreco, International Business Times; AI Has Its Place in Law, But Lawyers Who Treat It as a Replacement Can Risk Trust, Ethics, and Their Clients' Futures

"When segments of our profession begin treating AI outputs as inherently reliable, we normalize a lower threshold of scrutiny, and the law cannot function on lowered standards. The justice system depends on precision, on careful reading, on the willingness to challenge assumptions rather than accept the quickest answer. If lawyers become comfortable skipping that intellectual step, even once, we begin to erode the habits that make rigorous advocacy possible. The harm is not just procedural; it's generational. New lawyers watch what experienced lawyers do, not what they say, and if they see shortcuts rewarded rather than corrected, that becomes the new baseline.

This is not to suggest that AI has no place in law. When used responsibly, with human oversight, it can be a powerful tool. Legal teams are successfully incorporating AI into tasks like document review, contract analysis, and litigation preparation. In complex cases with tens of thousands of documents, AI has helped accelerate discovery and flag issues that humans might overlook. In academia as well, AI has shown promise in grading essays and providing feedback that can help educate the next generation of lawyers, but again, under human supervision.

The key distinction is between augmentation and automation. We must not be naive about what AI represents. It is not a lawyer. It doesn't hold professional responsibility. It doesn't understand nuance, ethics, or the weight of a client's freedom or financial well-being. It generates outputs based on patterns and statistical likelihoods. That's incredibly useful for ideation, summarization, and efficiency, but it is fundamentally unsuited to replace human reasoning.

To ignore this reality is to surrender the core values of our profession. Lawyers are trained not just to know the law but to apply it with judgment, integrity, and a commitment to truth. Practices that depend on AI without meaningful human oversight communicate a lack of diligence and care. They weaken public trust in our profession at a time when that trust matters more than ever.

We should also be thinking about how we prepare future lawyers. Law schools and firms must lead by example, teaching students not just how to use AI, but how to question it. They must emphasize that AI outputs require verification, context, and critical thinking. AI should supplement legal education, not substitute it. The work of a lawyer begins long before generating a draft; it begins with curiosity, skepticism, and the courage to ask the right questions.

And yes, regulation has its place. Many courts and bar associations are already developing guidelines for the responsible use of AI. These frameworks encourage transparency, require lawyers to verify any AI-assisted research, and emphasize the ethical obligations that cannot be delegated to a machine. That's progress, but it needs broader adoption and consistent enforcement.

At the end of the day, technology should push us forward, not backward. AI can make our work more efficient, but it cannot, and should not, replace our judgment. The lawyer who delegates their thinking to an algorithm risks their profession, their client's case, and the integrity of the justice system itself."

Tuesday, December 2, 2025

The case of the fake references in an ethics journal; Retraction Watch, December 2, 2025

 Retraction Watch ; The case of the fake references in an ethics journal

"Many would-be whistleblowers write to us about papers with nonexistent references, possibly hallucinated by artificial intelligence. One reader recently alerted us to fake references in … an ethics journal. In an article about whistleblowing.

The paper, published in April in the Journal of Academic Ethics, explored “the whistleblowing experiences of individuals with disabilities in Ethiopian public educational institutions.” 

Erja Moore, an independent researcher based in Finland, came across the article while looking into a whistleblowing case in that country. “I started reading this article and found some interesting references that I decided to read as well,” Moore told Retraction Watch. “To my surprise, those articles didn’t exist.”...

The Journal of Academic Ethics is published by Springer Nature. Eleven of the fabricated references cite papers in the Journal of Business Ethics — another Springer Nature title.

“On one hand this is hilarious that an ethics journal publishes this, but on the other hand it seems that this is a much bigger problem in publishing and we can’t really trust scientific articles any more,” Moore said."

Thursday, November 27, 2025

Prosecutor Used Flawed A.I. to Keep a Man in Jail, His Lawyers Say; The New York Times, November 25, 2025

 , The New York Times ; Prosecutor Used Flawed A.I. to Keep a Man in Jail, His Lawyers Say

"On Friday, the lawyers were joined by a group of 22 legal and technology scholars who warned that the unchecked use of A.I. could lead to wrongful convictions. The group, which filed its own brief with the state Supreme Court, included Barry Scheck, a co-founder of the Innocence Project, which has helped to exonerate more than 250 people; Chesa Boudin, a former district attorney of San Francisco; and Katherine Judson, executive director of the Center for Integrity in Forensic Sciences, a nonprofit that seeks to improve the reliability of criminal prosecutions.

The problem of A.I.-generated errors in legal papers has burgeoned along with the popular use of tools like ChatGPT and Gemini, which can perform a wide range of tasks, including writing emails, term papers and legal briefs. Lawyers and even judges have been caught filing court papers that were rife with fake legal references and faulty arguments, leading to embarrassment and sometimes hefty fines.

The Kjoller case, though, is one of the first in which prosecutors, whose words carry great sway with judges and juries, have been accused of using A.I. without proper safeguards...

Lawyers are not prohibited from using A.I., but they are required to ensure that their briefs, however they are written, are accurate and faithful to the law. Today’s artificial intelligence tools are known to sometimes “hallucinate,” or make things up, especially when asked complex legal questions...

Westlaw executives said that their A.I. tool does not write legal briefs, because they believe A.I. is not yet capable of the complex reasoning needed to do so...

Damien Charlotin, a senior researcher at HEC Paris, maintains a database that includes more than 590 cases from around the world in which courts and tribunals have detected hallucinated content. More than half involved people who represented themselves in court. Two-thirds of the cases were in United States courts. Only one, an Israeli case, involved A.I. use by a prosecutor."

Wednesday, November 26, 2025

What Is Agentic A.I., and Would You Trust It to Book a Flight?; The New York Times, November 25, 2025

 , The New York Times ; What Is Agentic A.I., and Would You Trust It to Book a Flight?

"A bot may soon be booking your vacation.

Millions of travelers already use artificial intelligence to compare options for flights, hotels, rental cars and more. About 30 percent of U.S. travelers say they’re comfortable using A.I. to plan a trip. But these tools are about to take a big step.

Agentic A.I., a rapidly emerging type of artificial intelligence, will be able to find and pay for reservations with limited human involvement, developers say. Companies like Expedia, Google, Kayak and Priceline are experimenting with or rolling out agentic A.I. tools.

Travelers using agentic A.I. would set parameters like dates and a price range for their travel plans, then hand over their credit card information to the bot, which would monitor prices and book on their behalf...

Think of agentic A.I. as a personal assistant, said Shilpa Ranganathan, the chief product officer at Expedia Group, which is developing both generative and agentic A.I. trip-planning tools.

While the more familiar generative A.I. can summarize information and answer questions, agentic tools can carry out tasks. Travelers benefit by deputizing these tools to perform time-consuming chores like tracking flight prices."

Wednesday, November 12, 2025

Vigilante Lawyers Expose the Rising Tide of A.I. Slop in Court Filings; The New York Times, November 7, 2025

 , The New York Times; Vigilante Lawyers Expose the Rising Tide of A.I. Slop in Court Filings

"Mr. Freund is part of a growing network of lawyers who track down A.I. abuses committed by their peers, collecting the most egregious examples and posting them online. The group hopes that by tracking down the A.I. slop, it can help draw attention to the problem and put an end to it.

While judges and bar associations generally agree that it’s fine for lawyers to use chatbots for research, they must still ensure their filings are accurate.

But as the technology has taken off, so has misuse. Chatbots frequently make things up, and judges are finding more and more fake case law citations, which are then rounded up by the legal vigilantes.

“These cases are damaging the reputation of the bar,” said Stephen Gillers, an ethics professor at New York University School of Law. “Lawyers everywhere should be ashamed of what members of their profession are doing.”...

The problem, though, keeps getting worse.

That’s why Damien Charlotin, a lawyer and researcher in France, started an online database in April to track it.

Initially he found three or four examples a month. Now he often receives that many in a day.

Many lawyers, including Mr. Freund and Mr. Schaefer, have helped him document 509 cases so far. They use legal tools like LexisNexis for notifications on keywords like “artificial intelligence,” “fabricated cases” and “nonexistent cases.”

Some of the filings include fake quotes from real cases, or cite real cases that are irrelevant to their arguments. The legal vigilantes uncover them by finding judges’ opinions scolding lawyers."

You’re a Computer Science Major. Don’t Panic.; The New York Times, November 12, 2025

 Mary Shaw and , The New York Times ; You’re a Computer Science Major. Don’t Panic.

"The future of computer science education is to teach students how to master the indispensable skill of supervision.

Why? Because the speed and efficiency of using A.I. to write code is balanced by the reality that it often gets things wrong. These tools are designed to produce results that look convincing, but may still contain errors. A recent survey showed that over half of professional developers use A.I. tools daily, but only about one-third trust their accuracy. When asked what their greatest frustration is about using A.I. tools, two-thirds of respondents answered, “A.I. solutions that are almost right but not quite.”

There is still a need for humans to play a role in coding — a supervisory one, where programmers oversee the use of A.I. tools, determine if A.I.-generated code does what it is supposed to do and make essential repairs to defective code."

Sunday, November 9, 2025

California Prosecutor Says AI Caused Errors in Criminal Case; Sacramento Bee via Government Technology, November 7, 2025

 Sharon Bernstein, Sacramento Bee via Government Technology; California Prosecutor Says AI Caused Errors in Criminal Case

"Northern California prosecutors used artificial intelligence to write a criminal court filing that contained references to nonexistent legal cases and precedents, Nevada County District Attorney Jesse Wilson said in a statement.

The motion included false information known in artificial intelligence circles as “hallucinations,” meaning that it was invented by the AI software asked to write the material, Wilson said. It was filed in connection with the case of Kalen Turner, who was accused of five felony and two misdemeanor drug counts, he said.

The situation is the latest example of the potential pitfalls connected with the growing use of AI. In fields such as law, errors in AI-generated briefs could impact the freedom of a person accused of a crime. In health care, AI analysis of medical necessity has resulted in the denial of some types of care. In April, A 16-year-old Rancho Santa Margarita boy killed himself after discussing suicidal thoughts with an AI chatbot, prompting a new California law aimed at protecting vulnerable users.

“While artificial intelligence can be a useful research tool, it remains an evolving technology with limitations — including the potential to generate ‘hallucinated’ citations,” Wilson said. “We are actively learning the fluid dynamics of AI-assisted legal work and its possible pitfalls.”