Loading story…
Leaked internal materials indicate that OpenAI's Project Lily employs hundreds of contractors to manually review anonymized ChatGPT prompts, transcripts, and user-memory summaries to evaluate response quality, style, and model behavior. Reviewers reportedly cannot see account names, but may receive enough conversational or location-related context to infer sensitive details.
OpenAI said its Privacy Filter removes personal information before conversations reach reviewers, while acknowledging that unusual, indirect, context-dependent, or short-chat disclosures can evade filtering. Use of consumer chats for model improvement is enabled by default for Free, Plus, and Pro accounts, whereas Enterprise, Business, and Edu accounts default to disabled; opting out applies only to new conversations, and previously collected or deleted chats may remain in datasets.
See the reporting duties and controls this puts on the clock.
3 events from the most recent confirmed update back to the earliest known activity.
Following 404 Media's inquiry, OpenAI directed the outlet to an FAQ stating that employees and contractors may review user content to improve models. The reporting also states that OpenAI updated its opt-out help page after the exposé, though the page did not mention human operators.
OpenAI told 404 Media that its Privacy Filter processes chats before contractor review but can miss rare, unusually formatted, indirect, or context-dependent personal information. Reviewed conversations may also include memory summaries that contain contextual or location-related user information.
404 Media reported that OpenAI's Project Lily uses hundreds of contractors to review anonymized real-user ChatGPT prompts, conversations, memory summaries, and response ratings to improve model quality and style. The materials reportedly included reviewer instructions, Slack information, real prompts, and scoring-system details.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
See what this changes for your reporting obligations and which controls it puts on the clock.
2 references tracked. Mallory keeps watching after this page renders.
tomshardware.com
Open sourcexakep.ru
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Meta ran a covert internal effort known as **Project Cannes** that used hundreds of contractors to pose as minors and test how rival AI chatbots responded to high-risk prompts about suicide, self-harm, sex, eating disorders, and drugs. According to internal materials, the operation was managed by contractor **Covalen** and targeted **OpenAI ChatGPT**, **Google Gemini**, and **Character.AI**, with workers creating dummy under-18 accounts and submitting both text prompts and images designed to trigger unsafe or policy-violating responses. One reviewed testing round in August 2025 reportedly generated more than **45,000 prompts**, and internal spreadsheets allegedly stored account details including names, email addresses, passwords, and birth dates while logging model responses in comparative databases. Meta described the activity as routine AI safety benchmarking and said competitor data was not used to train its own models, but the targeted companies were reportedly unaware of the testing and argued the practice may have violated platform and API terms because it was conducted covertly on third-party systems using falsified identities.
Jun 30, 2026
OpenAI introduced **Lockdown Mode** and **Elevated Risk** labels in *ChatGPT* to reduce exposure to **prompt injection** and related data-exfiltration risks when AI features interact with external systems. Lockdown Mode is positioned as an optional, advanced setting for higher-risk users and environments (notably *ChatGPT Enterprise*, *Edu*, *for Healthcare*, and *for Teachers*) that restricts tool access and limits how ChatGPT can reach outside systems; reported controls include disabling or constraining capabilities attackers could abuse via conversations or connected apps, and limiting browsing so that no live network requests leave OpenAI-controlled infrastructure (with browsing constrained to cached content). Admins can enable the setting via workspace controls and apply additional restrictions through dedicated roles, while Elevated Risk labels provide in-product warnings and guidance for features that increase risk when connecting to apps or the web, including across *ChatGPT*, *ChatGPT Atlas*, and *Codex*. Separate research highlighted how AI assistants with web-browsing/URL-fetching features can be abused as stealthy **command-and-control (C2) relays**, demonstrating a technique against **Microsoft Copilot** and **xAI Grok** that tunnels operator commands and victim data through legitimate AI web interfaces and can work without an API key or registered account. In parallel, the **European Parliament** reportedly disabled built-in AI tools on lawmakers’ work devices due to cybersecurity and privacy concerns about uploading sensitive correspondence to third-party cloud AI providers and uncertainty about what data is shared and retained. Other referenced material focused on general productivity customization of ChatGPT via “Custom Instructions,” rather than a specific security event or disclosure.
Jun 29, 2026
Google is rolling out *AI Mode* personalization that can **connect Google Search to Gmail and Google Photos** for opt-in users, aiming to deliver more tailored results based on personal context. The feature is positioned as “secure” and is initially available via Labs for Google AI Pro and AI Ultra subscribers (with limited account eligibility), with Google stating the system processes data for specific prompts and does not directly train on a user’s inbox or photo library; the change nonetheless increases the amount of sensitive personal data that can be accessed during AI-assisted search workflows. OpenAI is testing an upgrade to **ChatGPT Temporary Chat** that keeps the session from being saved to history or used for model improvement, while still allowing **personalization signals** (e.g., memory/style preferences) to apply—alongside a stated retention window where OpenAI may keep a copy for up to **30 days** for safety. Separately, researchers and commentators warned about an “**Ouroboros effect**” where ChatGPT may cite AI-generated repositories such as xAI’s Grokipedia, increasing the risk of **misinformation loops** and “content traps” if AI systems do not rigorously vet sources, potentially degrading trust and decision-making even without direct training on the cited content.
Jun 29, 2026