The online platform X has provoked widespread anger after people used its artificial intelligence assistant Grok to alter images of women by taking off their garments.
Several examples have surfaced where the system has been used to digitally undress female subjects to render them in bikinis non-consensually, as well as to put them in sexualized contexts.
Company Response and Legal Stance
XAI, the firm that created Grok, did not respond to questions, except for an auto-reply message stating "traditional news outlets lies".
A government official stated that legislation is being brought forward to ban nudification software. Under a upcoming law, anyone who distributed such technology would "receive a jail term and substantial penalties".
"The platform is capable of halting these types of misuse if they desired to," said Clare McGlynn, adding they "operate with a lack of consequences."
Oversight Body Stance and Tool Capabilities
The regulator Ofcom said platforms must "assess the potential" of people seeing unlawful material on their platforms, but did not confirm whether it was presently looking into X or Grok in regarding AI-generated images.
Grok is a free AI assistant—with some paid-for features—which answers X users' queries when they mention it in a message.
It is commonly used to provide reactions or further context to other posters' comments, but people on X are also can alter an uploaded photo through its artificial intelligence picture modification tool.
Background of Controversy and Rules
The chatbot has been condemned for permitting users to produce pictures and clips with nudity and sexually suggestive material, and it was in the past alleged to have been involved in generating inappropriate material of public figures.
"The platform has been allowing the generation and spread of these images for months without taking any measures and we have not witnessed any challenge by regulators," Professor McGlynn added.
XAI's own usage rules bans "portraying likenesses of persons in a pornographic manner".
In a statement, Ofcom said it was unlawful to "create or distribute deepfakes or child sexual abuse material" and confirmed this covered sexual deepfakes created with AI.
It said platforms such as X were mandated to take "suitable steps" to "minimize the risk" of users coming across illegal material on their platforms and to delete it promptly when they are notified of it.
