Loading story…
Meta ran a covert internal effort known as Project Cannes that used hundreds of contractors to pose as minors and test how rival AI chatbots responded to high-risk prompts about suicide, self-harm, sex, eating disorders, and drugs. According to internal materials, the operation was managed by contractor Covalen and targeted OpenAI ChatGPT, Google Gemini, and Character.AI, with workers creating dummy under-18 accounts and submitting both text prompts and images designed to trigger unsafe or policy-violating responses.
One reviewed testing round in August 2025 reportedly generated more than 45,000 prompts, and internal spreadsheets allegedly stored account details including names, email addresses, passwords, and birth dates while logging model responses in comparative databases. Meta described the activity as routine AI safety benchmarking and said competitor data was not used to train its own models, but the targeted companies were reportedly unaware of the testing and argued the practice may have violated platform and API terms because it was conducted covertly on third-party systems using falsified identities.
Track how attackers are adapting to this technology.
2 events from the most recent confirmed update back to the earliest known activity.
According to internal documents cited by ZDNet via WIRED, the confidential Meta program operated by contractor Covalen continued until at least April 2026, using fake underage accounts to test rival AI chatbots and log their responses.
Internal materials indicate that one Project Cannes testing round in August 2025 involved more than 45,000 prompts sent through fake under-18 accounts to probe how ChatGPT, Gemini, and Character.AI handled sensitive topics such as suicide, sex, eating disorders, and drugs.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
2 references tracked. Mallory keeps watching after this page renders.
Map indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
OpenAI launched a teen version of ChatGPT for users aged 13 to 17 with tighter safeguards around self-harm, violence, eating disorders, dangerous activities, explicit sexual content, and anthropomorphic or romantic interactions. The teen experience also adds education-focused features including **Study Hours** and **Study Mode**, positioning the product as a more restricted and school-oriented version of the chatbot. The rollout has drawn attention because the stricter design aligns with risks OpenAI had previously documented in its own safety materials and research, including the `GPT-4o` system card and later work on unhealthy attachment and emotional dependence. Critics argue the launch shows the company has the technical ability to reduce harmful engagement and emotional-bond risks, while limiting those protections to minors rather than applying comparable controls more broadly.
Sep 5, 2026
Meta changed its AI-generated suggested prompts after Facebook and Instagram user Kalie Robins reported that Meta AI asked invasive questions about her young daughters and assembled details from years of family posts. Prompts including “Who is the child passenger?” and questions about the children’s ages and the family’s residence reportedly led the chatbot to surface names, birth dates, photos, videos, and inferred location information, including an image Robins believed she had deleted years earlier. Meta said the personal-topic suggestions “missed the mark” and that it fixed the underlying prompt issue. The company maintained that Meta AI returns only material already accessible to the person making the query, but the incident renews scrutiny of the platform’s AI privacy controls following reports of publicly shared AI chats, an alleged private-chat ID-guessing exposure, and plans to use AI interactions for advertising targeting.
Sep 15, 2026
OpenAI reported that a **ChatGPT account linked to Chinese law enforcement** used the model to review and edit internal write-ups on so-called **“cyber special operations,”** which OpenAI assessed as activity consistent with **covert influence operations and transnational repression** aimed at harassing and silencing critics of the Chinese Communist Party. The uploaded materials described a sustained, resource-intensive campaign involving **hundreds of staff**, **thousands of fake social-media accounts**, mass content generation, and tactics such as flooding platforms with fraudulent complaints against dissidents, forging documents, and in some cases **impersonating U.S. officials** to intimidate targets. OpenAI said the activity was tied to a single account, which it **banned**. OpenAI also described an attempted **smear/propaganda operation targeting Japanese Prime Minister Sanae Takaichi**, where the actor prompted ChatGPT for plans to amplify negative commentary on social media and to draft messages from fake accounts to pressure other Japanese politicians; when the model refused, the actor’s later prompts suggested the operation proceeded using other AI models. Separately, OpenAI attributed another cluster of accounts likely originating from mainland China that used ChatGPT to seek information on **U.S. persons, online forums, and federal building locations**, and to draft emails posing as a Hong Kong-based firm (*Nimbus Hub Consulting*), with OpenAI noting indicators such as VPN usage and prompts written in **Simplified Chinese**.
Jun 29, 2026