Loading story…
OpenAI's release of Sora 2, an advanced AI video generation tool, has sparked significant concern among advocacy groups and security researchers. Public Citizen, a nonprofit focused on AI accountability, sent a letter to OpenAI urging the company to temporarily take Sora 2 offline, citing the proliferation of realistic deepfakes and the lack of sufficient safeguards. The group criticized OpenAI for what they described as a pattern of rushing products to market without adequate safety measures, warning that Sora 2's capabilities could be exploited to create convincing disinformation and manipulate public perception, especially in sensitive contexts such as elections.
In addition to these advocacy concerns, AI security firm Mindgard discovered a technical vulnerability in Sora 2 that allows attackers to extract the model's hidden system prompt via audio transcripts. By instructing Sora 2 to generate audio, researchers were able to reconstruct the internal rules and safety guidelines that govern the model's behavior, potentially exposing the tool to further abuse or circumvention of its guardrails. This dual set of issues—deepfake risks and prompt leakage—highlights the urgent need for stronger security and transparency measures in the deployment of generative AI technologies like Sora 2.
Track how attackers are adapting to this technology.
2 events from the most recent confirmed update back to the earliest known activity.
Advocacy group Public Citizen called on OpenAI to address the deepfake risks posed by Sora 2, highlighting concerns about misuse of the video generation model. The appeal marked a public policy and safety response focused on the model's potential societal harms.
AI security firm Mindgard found that OpenAI's Sora 2 could be prompted to reveal portions of its hidden system prompt through audio generation and transcript reconstruction. The research showed that audio outputs were a more reliable exfiltration channel than text or video for recovering internal rules and safety instructions.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
3 references tracked. Mallory keeps watching after this page renders.
scworld.com
Open sourcecyberscoop.com
Open sourcehackread.com
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
OpenAI introduced a new security framework called Guardrails as part of its AgentKit toolset, aiming to enhance the safety of AI agents built on its ChatGPT technology. The Guardrails system was designed to prevent AI agents from engaging in harmful or unintended behaviors, such as leaking personal information or responding to malicious prompts. It operates by using LLM-based judges to detect and block attempts at jailbreaking or prompt injection, which are common techniques used to circumvent AI safety measures. Jailbreaking involves manipulating the AI into breaking its own rules, while prompt injection uses cleverly crafted inputs to force the AI into unintended actions. Shortly after the release of Guardrails, researchers from HiddenLayer discovered significant flaws in its design. They demonstrated that if the same type of model is used both to generate responses and to act as a safety checker, both can be deceived by similar techniques. The researchers were able to bypass the Guardrails using straightforward methods, successfully prompting the system to generate harmful outputs and execute hidden prompt injections without detection. In one experiment, they manipulated the AI judge’s confidence score, allowing a prompt that was initially flagged as a jailbreak to pass through the safety checks. Further testing revealed that indirect prompt injections could be executed via tool calls, potentially exposing confidential user data. The vulnerability was demonstrated in real-world scenarios, showing that the Guardrails system could not reliably block malicious prompts or prevent indirect prompt injections. The rapid bypass of Guardrails highlights the inherent challenges in securing AI systems against adversarial inputs. OpenAI’s intention was to provide developers with robust tools to prevent misuse of AI agents, but the research indicates that current safeguards are insufficient. The findings underscore the need for more diverse and independent safety mechanisms, rather than relying on the same model architecture for both generation and security. The incident has raised concerns within the AI and cybersecurity communities about the effectiveness of current AI safety strategies. It also emphasizes the importance of continuous testing and improvement of security frameworks as AI technologies evolve. The exposure of these flaws so soon after the release of Guardrails demonstrates the persistent cat-and-mouse dynamic between AI developers and security researchers. Organizations deploying AI agents are advised to remain vigilant and to supplement built-in safety features with additional layers of security. The research by HiddenLayer serves as a cautionary example of the limitations of current AI safety tools and the ongoing need for innovation in this area.
Jun 29, 2026
Open-source and locally run generative AI models are being operationalized for **nonconsensual sexual imagery** and other manipulation, with researchers (including **Graphika** and **Open Measures**) tracking coordinated sharing of “nudified” deepfakes targeting Olympic athletes on platforms such as **4chan**. Reporting described how communities use downloadable models without safety guardrails and share fine-tuned components like **Low-Rank Adaptations (LoRA)** to improve output quality and lower the technical barrier for abuse, accelerating the spread of sexualized deepfakes and related harassment. Separate commentary highlighted that as **agentic AI** moves into production, organizations are increasingly judged on reliability, auditability, and operating within regulatory boundaries, because these systems can execute multi-step actions across tools with limited human prompting. The material emphasized the need for governance controls—e.g., defined action permissions, escalation paths, logging, and human-in-the-loop checkpoints—to prevent autonomous behavior from exceeding policy or risk thresholds; additional workplace-oriented coverage focused on employee anxiety and career adaptation around AI rather than a specific security incident.
Jun 29, 2026
AI-powered tools and agents are increasingly being adopted in both personal and professional environments, but this rapid integration is outpacing the implementation of adequate security measures. A recent study by the National Cybersecurity Alliance and CybNet found that 43% of workers have shared sensitive information, including financial and client data, with AI tools such as ChatGPT and Gemini, often without any formal training on the associated risks. This widespread use of generative AI has introduced significant data security and privacy concerns, as many employees are unaware of the potential for data leakage or misuse. The vulnerabilities of AI-powered applications are further highlighted by the discovery of two critical flaws in Wondershare RepairIt, an AI-based repair tool used by millions. According to Trend Micro, RepairIt not only contradicted its privacy policy by storing user data in unsecured cloud storage but also exposed users to supply chain attacks by allowing attackers to swap or modify AI models, potentially infecting countless users through a single update. The vulnerabilities in RepairIt were rated with CVSS scores of 9.1 and 9.4, among the highest for consumer AI applications this year, underscoring the severity of the threat. In another case, Notion’s release of version 3.0 with AI agents introduced a new attack vector where prompt injection could be used to exfiltrate confidential data. By embedding malicious instructions in a PDF, attackers could instruct the AI agent to collect and transmit sensitive information to an external server, exploiting the agent’s access to private data and its ability to communicate externally. This attack demonstrates the inherent risks when AI agents are given broad access and communication capabilities without sufficient safeguards. The broader landscape of AI-driven threats is also evolving, as highlighted by the 2025 Global Bot Security Report, which found that most websites remain vulnerable to even basic automated attacks, let alone sophisticated AI-powered bots. Traditional security defenses are struggling to keep pace with the complexity and speed of AI-driven threats, making it increasingly difficult to distinguish between benign and malicious AI activity. Anthropic’s release of Claude Sonnet 4.5, a large language model with enhanced safety and security features, represents an industry effort to address these challenges. The model incorporates advanced safeguards against prompt injection, internal security measures to protect model weights, and restrictions on queries related to dangerous topics. However, even with these improvements, the risk of false positives and the need for ongoing vigilance remain. Collectively, these incidents and studies illustrate the urgent need for organizations to implement robust security training, enforce strict data handling policies, and adopt advanced technical safeguards when deploying AI-powered tools and agents. The convergence of user behavior, technical vulnerabilities, and evolving threat tactics is creating a complex risk environment that demands proactive and comprehensive security strategies.
Jun 29, 2026