San Francisco official David Chiu has demanded that Apple and Google remove 13 AI apps that can generate nonconsensual sexual images, according to Wired, putting new pressure on the two companies that control the dominant mobile app stores.
The cease-and-desist letters accuse the app store operators of running afoul of California laws that bar services supporting the creation of deepfake pornography, Wired reported after reviewing the letters. Chiu asked Google to remove five apps from Google Play and Apple to remove eight from the App Store.
The apps were not named in Wired’s report, a choice the outlet said was meant to avoid sending users to them. Wired reported that one of the apps had been downloaded more than 1 million times and promoted tools for sexualizing images of women or creating “free and uncensored” videos.
What the apps do
So-called nudify apps use AI image generation and editing systems to turn ordinary pictures of real people into explicit images. According to the reporting, these tools can remove clothing, modify a person’s body or face, put someone in a sexual pose, or attach a victim’s face to another person’s naked body.
That mechanism matters because the victim does not need to participate, pose, or consent. A regular photo can become nonconsensual intimate imagery with a few prompts or taps. Chiu told Wired that his office was “absolutely horrified” by the spread of the tools, which he said are being used against women and children.
Chiu said the images are used to bully, humiliate, and threaten women and girls, and that victims have suffered reputational harm, mental health consequences, loss of autonomy, and suicidal distress, according to Wired.
His office also estimated that Apple and Google likely collected millions of dollars in fees while the apps remained available, Wired reported. Chiu said app stores should not profit from tools that generate illegal nonconsensual intimate images.
Google says the flagged apps are gone
Google spokesperson Dan Jackson told Ars Technica that Google suspended the five apps Chiu identified for violating Google Play policies against harmful content.
Jackson said Google Play bars apps containing sexual content and said the company takes steps to find and remove harmful apps. He also said Google has suspended hundreds of violating apps and restricted related search terms, including “nudify,” on its store.
Apple did not respond to Ars Technica’s request for comment. Wired reported that Apple has removed some apps identified by researchers, while critics continue to press the company to police the App Store more aggressively.
Detection is getting harder. Wired cited a May preprint paper in which researchers found 420 apps marketed as generic face-swapping tools. The researchers tested 155 of them and found that 70 percent could be used for nudification. That suggests some developers can present an app as a harmless face-swap toy while keeping the harmful function available inside.
Grok remains a separate pressure point
The fight over nudify apps also lands while app stores face questions about xAI’s Grok chatbot. Ars Technica reported that xAI filed a lawsuit this week after finding instances of Grok-generated child sexual abuse material and nonconsensual intimate imagery targeting adults. xAI sued the user it says prompted Grok to create the illegal material.
Chiu’s letters did not ask Apple or Google to remove Grok. The issue is still relevant to app store enforcement because both companies have rules against harmful content, and generative AI tools can produce illegal images if safeguards fail or are bypassed.
Jackson told Ars Technica that Google tests generative AI apps for safeguards against nonconsensual intimate imagery and child sexual abuse material. Ars reported that he did not answer whether Grok’s current behavior is an exception to Google’s policy.
Chiu told Wired that he wants Apple and Google to remove the flagged apps and improve screening so similar apps do not reach their stores. He said his office would consider legal options if the companies do not act.
This story draws on original reporting from Ars Technica.
