A coalition of U.S. state attorneys general (AGs) has issued a stark warning to major artificial intelligence (AI) companies, including Microsoft, OpenAI, and Google, demanding immediate action to address "delusional outputs" from their AI chatbots. Following a series of disturbing mental health incidents linked to AI use, the AGs are pushing for new safeguards to protect users from psychological harm, threatening legal action under state laws if companies fail to comply.

The comprehensive letter, signed by dozens of AGs from across U.S. states and territories under the National Association of Attorneys General, was sent to 13 prominent AI firms. Beyond Microsoft, OpenAI, and Google, the list includes Anthropic, Apple, Chai AI, Character Technologies, Luka, Meta, Nomi AI, Perplexity AI, Replika, and xAI. It calls for the implementation of various internal safeguards to shield users from potentially harmful AI interactions.

Specific Safeguards Requested

Key demands outlined in the letter include:

  • Transparent third-party audits of large language models (LLMs) to identify signs of "delusional or sycophantic ideations."
  • New incident reporting procedures designed to notify users when chatbots generate psychologically harmful content.
  • Allowing independent third parties, such as academic and civil society groups, to "evaluate systems pre-release without retaliation and to publish their findings without prior approval from the company."

Rationale and Incidents

The AGs emphasized the dual nature of generative AI (GenAI), acknowledging its potential for positive change while highlighting its capacity for "serious harm, especially to vulnerable populations." The letter points to several well-publicized incidents over the past year, including cases of suicides and even murder, where violence has been linked to excessive AI use.

"In many of these incidents, the GenAI products generated sycophantic and delusional outputs that either encouraged users’ delusions or assured users that they were not delusional," the letter states.

Treating Mental Health Like Cybersecurity

The AGs further suggest that AI companies adopt the same rigorous approach to mental health incidents as they do for cybersecurity breaches. This entails developing and publishing "detection and response timelines for sycophantic and delusional outputs." Similar to data breach protocols, companies should also "promptly, clearly, and directly notify users if they were exposed to potentially harmful sycophantic or delusional outputs."

Another critical demand is for companies to develop and conduct "reasonable and appropriate safety tests" on GenAI models. These tests are intended to "ensure the models do not produce potentially harmful sycophantic and delusional outputs" and must be completed before the models are made available to the public.

Industry Response and Regulatory Showdown

As of publication, TechCrunch was unable to obtain comments from Google, Microsoft, or OpenAI regarding the letter. The warning from state AGs comes amidst an ongoing regulatory battle between state and federal governments over AI governance. While the Trump administration has been openly pro-AI, multiple attempts to impose a nationwide moratorium on state-level AI regulations have failed, partly due to pressure from state officials.

Undeterred by these setbacks, President Trump recently announced plans to sign an executive order next week aimed at limiting states' ability to regulate AI. In a Truth Social post, the president expressed his hope that the executive order would prevent AI from being "DESTROYED IN ITS INFANCY."