California Attorney General Rob Bonta has issued a cease-and-desist order to Elon Musk's artificial intelligence startup, xAI, demanding an immediate halt to the creation and distribution of nonconsensual sexual deepfakes and child sexual abuse material (CSAM) through its Grok chatbot. This decisive action follows an investigation into reports that Grok was being used to generate illicit imagery of women and minors, intensifying the ongoing debate around AI ethics and online safety.
The order, issued on Friday, mandates that xAI take swift measures to prevent the production of such content. Attorney General Bonta emphasized the gravity of the situation in a press release, stating:
“Today, I sent xAI a cease-and-desist letter, demanding the company immediately stop the creation and distribution of deepfake, nonconsensual, intimate images and child sexual abuse material. The creation of this material is illegal. I fully expect xAI to immediately comply. California has zero tolerance for [CSAM].”
The Attorney General's office further alleged that xAI appeared to be “facilitating the large-scale production” of nonconsensual nude images, which are reportedly “used to harass women and girls across the internet.” The agency has given xAI five days to demonstrate concrete steps taken to address these critical issues.
At the core of the controversy is Grok's “spicy” mode, a feature reportedly designed by xAI to generate explicit content. The issue has garnered international attention, with Japan, Canada, and Britain launching their own investigations into Grok. Furthermore, Malaysia and Indonesia have temporarily blocked the platform entirely. Despite xAI implementing some restrictions on its image-editing capabilities earlier in the week, the California AG's office proceeded with its cease-and-desist order.
While X's safety account has previously denounced such user activity, warning that “Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content,” attempts to reach xAI for comment were met with an automated email stating “Legacy Media Lies.”
This incident highlights a broader challenge facing the tech industry. The proliferation of free generative AI tools has led to a disturbing surge in non-consensual sexual material across numerous platforms, not just X. The alarming trend has captured the attention of both state and federal lawmakers. On Thursday, U.S. Senators sent letters to executives at several major tech companies—including X, Reddit, Snap, TikTok, Alphabet, and Meta—demanding answers on their strategies to combat the spread of sexualized deepfakes.
