Dr Paul Bouchaud is a lead researcher at
AI Forensics, a European non-profit organisation that conducts independent technical investigations into the algorithmic systems driving online platforms. We spoke about their recent audit that tracked the rapid rise of sexualised deepfakes and other harmful content generated using Grok.
Q: What is AI Forensics and can you introduce your work?
Paul: AI Forensics is a European non-profit that investigates influential and opaque algorithms. We conduct independent technical investigations into major technology platforms to uncover and expose harms caused by their algorithmic systems.
Platforms rarely provide researchers with meaningful access to data. Therefore, we have developed our own auditing infrastructure that allows us to collect evidence independently. We mimic user behaviour using methods such as sock-puppets, scraping, and structured user data donation. These adversarial but methodologically rigorous approaches allow us to remain fully independent from platforms while preserving the integrity of our datasets.
I am a lead researcher at AI Forensics conducting quantitative analysis. This involves large-scale data collection and statistical assessments of platform behaviour. I have worked on a range of investigations, including into Amazon’s recommendation systems, Meta’s ad delivery practices – which contributed to the European Commission opening formal proceedings – and advertising practices on X.
Q: What motivated you to investigate Grok and how did you conduct the audit?
Paul: The investigation began in response to a growing trend observed during the Christmas period. My X feed became saturated with Grok-generated images depicting individuals, predominantly women, in minimal attire. I suspected the trend was spreading rapidly based on anecdotal impressions. Therefore, we decided to measure the scale and characteristics of the phenomenon systematically.
Between Christmas and New Year, we collected approximately 20,000 Grok-generated images from X. In the weeks that followed, we expanded the dataset to over 100,000 images. This large-scale collection allowed us to measure patterns over time, assess demographic impacts, and track the introduction of safeguards.
We collected images generated via the X interface and xAI’s Grok Imagine web and mobile app interfaces. For Grok-generated content via X, the outputs were public. This enabled us to use automated collection methods to archive generated images at scale. Despite a lack of access to X’s official API for research purposes, we were able to collect publicly available X posts through technical means.
For images generated directly via Grok Imagine, we used a different strategy. Grok allows users to share generated images (and chats) on platforms or via direct messages. To fulfil this function, Grok creates a URL link that enables a recipient to view the generation request (i.e. the prompt and generated image). We found that web crawlers and search engines were indexing these URL links, which enabled us to collect generated images. In parallel, Grok users were sharing links, prompts and generated images in niche online communities which we were able to archive.
In addition, we conducted limited prompting experiments to test the Grok model’s ability to produce harmful content. We did not conduct prompting experiments via the X interface because outputs were public and could expose third parties to harmful content. Therefore, we only conducted prompting via the Grok Imagine interface because outputs are private by default (unless shared). This allowed us to test the model’s capabilities directly using European IP addresses.
Q: How was Grok used to generate sexualised deepfakes and CSAM?
Paul: Our findings were striking. Our initial report '
Grok Unleashed' analysed the initial 20,000-image dataset, finding nearly half of the generated images depicted individuals in minimal attire. Approximately 80% of those individuals presented as women. On X, most user prompts did not originate from the person depicted. Instead, third parties replied to an existing post and asked Grok to modify the image, typically to “put this person in a bikini”. This trend scaled rapidly within days of launch, with thousands of public replies containing modified images of identifiable individuals. This disproportionately targeted female-presenting individuals and often operated without their consent.
We also identified a small percentage of sexualised images depicting individuals who appeared to be minors. Within the Grok Imagine dataset, we identified 17 images that were subsequently confirmed by a French hotline as constituting CSAM under French law. We notified the relevant authorities and xAI. One month later, 14 of those images remained accessible online.
In addition to sexualised imagery, we observed the generation of extremist content, including Nazi and ISIS-related imagery. While such outputs were not prevalent at scale, their existence demonstrated the absence of effective guardrails.
The investigation also revealed important differences between the access interfaces. On X, Grok did not generate images with full nudity. Even when prompted indirectly by users, for example through references to transparent clothing, the model failed to produce explicit anatomical detail. By contrast on the Grok Imagine web interface, we were able to prompt Grok to generate explicit pornographic content simply by activating “Spicy Mode”. This signalled that adult content generation was an intended feature. These differences indicate that the models can behave differently depending on interface-level restrictions. The constraints observed on X were not inherent technical limitations but configurable design choices.
Q: Since your initial audit, X announced safeguards to address the controversy. How effective have these been?
Paul: Following public scrutiny, X and xAI introduced several restrictions. In particular on X, it restricted image generation to X Premium subscribers and disabled the ability to create sexualised deepfakes, including bikini transformations, through prompt filtering and adjusting its refusal thresholds. Our
updated report confirmed that the volume of bikini-style transformations dropped significantly after these changes.
However, significant disparities between the interfaces remain. While X imposed restrictions, Grok Imagine continued to allow explicit content generation via “Spicy Mode” without clearly robust systems to prevent illegal outputs. This divergence underscores the importance of evaluating not only the model itself but also the surrounding product design.
The speed with which these restrictions were implemented demonstrates that the safeguards were technically feasible from the outset. The issue was not capability but decision-making. The model’s behaviour changed once public pressure mounted.
Q: What action have regulators and public bodies taken to address these harms?
Paul: The Grok episode became an early stress test for multiple regulatory regimes. In France, the Attorney General quickly expanded an existing investigation into X to include Grok-generated sexual deepfakes. French criminal law already prohibits the creation and dissemination of sexualised deepfakes without consent. Similarly, the Californian Attorney General initiated an investigation under state-level deepfake legislation.
At the EU level, the Digital Services Act (DSA) requires Very Large Online Platforms – such as X – conduct a systemic risk assessment when it makes a significant design modification. Therefore, it arguably should have assessed the integration of Grok across a range of risks, including the dissemination of illegal content, negative impacts on the rights of the child and gender-based violence. Indeed, the European Commission (EC) has opened an investigation into Grok’s integration on X, ordering the company to retain data and internal documents. However, the EC’s approach has not matched the rapid pace with which the harms materialised. By contrast, the EC acted comparatively quickly in the case of TikTok Lite by utilising the DSA’s interim measures procedure.
Grok Imagine web and mobile apps are not well regulated by the DSA. However, the Artificial Intelligence Act does require GPAI providers – such as xAI – to comply with transparency obligations to ensure that all synthetic content is watermarked and machine-detectable. In addition, GPAI providers with significant risk – which xAI may fall in scope of – are required to conduct systemic risk assessments and put in place mitigations. Indeed, xAI signed the GPAI Code of Practice safety and security chapter, committing to identify risks from illegal content, such as CSAM and sexual deepfakes.
Overall, whilst the regulatory architecture exists, the episode exposed limitations in crisis-response speed and coordination. The Grok case demonstrates that powerful generative systems can scale harm extremely quickly, especially when integrated into social media platforms. Regulators must therefore strengthen rapid-response mechanisms that operate alongside traditional investigation procedures.
Regulators should also scrutinise interface-level design decisions. Enforcement frameworks must account for how product integration shapes risk exposure. Platforms must conduct sufficient pre-deployment risk assessments and disclose mitigations before embedding generative AI features within their platforms. As our work demonstrates, independent monitoring is essential to detect systemic harms in real time.
