By Maryam Yaqub

Maryam Yaqub is a leading gender equality and AI expert working with the Fawcett Society to ensure governments and employers promote the responsible, safe and equitable use of artificial intelligence. In an era where emerging technologies shape our everyday lives, it’s crucial that AI serves women’s rights, not harms them.

In this blog, Maryam reflects on the harmful misuse of Grok from early January, where an AI chatbot was used to generate non-consensual and sexualised images of women. She explores what this incident reveals about the shortcomings of current UK law, the need for stronger regulation and platform accountability, and how AI legislation must evolve to protect women’s safety and dignity online.

---

What the Grok incident tells us about AI, abuse, and the law

You may have seen recent headlines about Grok, an AI tool that was used to digitally “nudify” images of women without their consent. Since then, many people have asked the same question: surely this is already illegal, so how did this happen, and what can actually be done about it?

The confusion is understandable. The harm feels obvious, but the law is not always clear. We wanted to take a moment to reflect on what the Grok incident really shows us about AI, online abuse, and the gaps that still exist in the UK’s approach.

What happened, and what changed

In early January, users discovered that Grok could generate sexualised images of real women, including public figures and minors, without consent. These images spread quickly, prompting public outrage and intense media attention.

In response, the UK government said this kind of use was unacceptable and suggested it could breach the Online Safety Act. Ofcom confirmed it was engaging with X to understand what safeguards were in place and whether the law had been broken. Shortly afterwards, X announced that it had restricted Grok so it could no longer generate sexualised images of real people. Other companies, including OpenAI, also made changes to image generation features, with some tools paused or limited.

These steps matter. They show that public pressure and regulatory scrutiny can lead to rapid action. But they also raise an important question. If this harm could happen so easily in the first place, what stops it from happening again? 

This is not just about one bad app

The UK government has said it wants to ban “nudify apps”, and that is an important step. These tools are designed specifically to sexually manipulate images, usually of women, without consent. But Grok was not marketed as a nudify app. It is a general-purpose AI system built into a large platform, with many different features. The problem is that some of those features make sexual abuse easy, fast, and scalable, even if that is not the tool’s official purpose.

This is significant because under current rules, companies can often say they provide neutral technology and it is users who misuse it. That argument becomes much harder to defend when harm is predictable and widespread. 

Why current law struggles to keep up

Most UK law focuses on what happens after harm occurs.The Online Safety Act, for example, is mainly about removing abusive content once it has been shared and taking action against individuals who break the rules. That approach struggles with AI systems that can create harmful material in seconds. Once an image has been generated, the damage is already done. Endless takedowns do not stop the next image from being created. 

Other countries are going further

The UK is not alone in facing this problem, but some countries are already moving faster.

Australia and France have criminalised the creation of non-consensual sexual deepfakes, not just their sharing. At a European level, the EU has recognised that some AI uses are so risky they require strict safeguards, transparency, and limits on how they are deployed.

Responsibility should not sit only with victims or individual users. Companies that build and run powerful AI systems must also be accountable for foreseeable harm.

Moving from reaction to prevention

Regulators could treat AI image generation tools as high-risk features, requiring companies to build in strong protections before they are allowed to operate. This could include blocking certain prompts, filtering outputs, and acting quickly when harm is reported. 

There are practical steps that could help prevent repeat harm rather than constantly playing catch up. These include clearer labelling of AI generated images, and shared systems to stop the same abusive content from being recreated and re-uploaded again and again.

One practical solution is a shared cross platform reporting and blocking system. Under this model, platforms would be required to subscribe to a common service that flags abusive AI generated images, including non-consensual sexual images and deepfakes. Once an image is reported and verified on one platform, it is automatically blocked across all participating platforms, preventing it from being uploaded elsewhere. This removes the burden from victims of having to report the same image repeatedly and focuses responsibility on platforms to stop harm at the system level.

There is also growing recognition internationally that a person’s face, voice, and body are not free raw material for AI systems. Some countries are exploring stronger rights over personal likeness and identity, which would make it easier to challenge abusive uses at their source.

A change in approach is needed

If regulation focuses only on cleaning up the mess after harm has occurred, it will lag behind the technology. Preventing AI enabled violence against women requires acting earlier, on system design, company responsibility, and structural safeguards.

We will continue to follow this issue closely and push for an approach that centres prevention, accountability, and women’s safety from the start.

Thank you, as always, for standing with us.