Skip to main content
Mallory
← Back to News
ai-platform-securityidentity-impersonation-fraudstandards-framework-updateai-enabled-threat-activity

Security and Verification Challenges of AI-Generated Visual Content

Emerging Technology Risk · 2 sources · Updated 5mo ago · First seen Oct 13, 2025

Security and Verification Challenges of AI-Generated Visual Content

The rapid advancement of generative AI technologies has introduced significant challenges in verifying the authenticity and origin of digital images and videos. Researchers at Queen’s University in Canada have investigated watermarking as a method to tag AI-generated images, enabling the verification of their origin and integrity. Watermarking systems function as comprehensive security processes, involving embedding coded messages within images either during or after their creation. These embedded signals can later be extracted and matched to cryptographic keys, confirming the provenance of the image. Some watermarking techniques are integrated directly into the generative model, increasing their resilience to common image manipulations such as compression or cropping. The effectiveness of watermarking relies on the invisibility of the mark to viewers, its durability through standard image handling, and its resistance to duplication by unauthorized parties. Early watermarking methods used signal-processing techniques, but the complexity of modern generative models has led to the adoption of deep learning-based approaches, such as encoder–decoder networks, which can automatically hide and retrieve marks. The emergence of advanced AI video generation models, such as OpenAI’s Sora2, has further complicated the landscape. Sora2, accessible through platforms like Lovart, allows users to generate photorealistic video sequences from text prompts, democratizing access to powerful content creation tools. While this technology empowers creators, it also provides malicious actors with sophisticated means to produce deceptive visual content. The widespread availability of Sora2 raises urgent concerns about the reliability of visual evidence and the potential for misuse in fraud, misinformation, and other security threats. Cybersecurity professionals are now tasked with developing and implementing robust verification frameworks to maintain digital integrity in an environment where AI-generated content is increasingly indistinguishable from reality. The need for new digital verification standards is underscored by the growing difficulty in distinguishing authentic media from AI-generated fabrications. Both watermarking and other verification technologies are being evaluated and improved to address these evolving threats. The convergence of advanced generative AI and the challenges of content verification highlights a critical area of focus for the security community. As generative models continue to evolve, so too must the methods for ensuring the authenticity and accountability of digital visual content. Ongoing research and collaboration between academia, industry, and security professionals are essential to develop effective solutions that can keep pace with the rapid innovation in AI-generated media.

New surface, new adversariesTrack how attackers are adapting to this technology.
Track this risk
Event timeline

How this story unfolded

Events from the most recent confirmed update back to the earliest known activity.

7 events · Jan 10, 2023 – Oct 13, 2025
  1. Oct 13, 20251y ago

    Research highlights weaknesses in AI watermarking defenses

    The analysis identified major weaknesses in current watermarking approaches, including cropping, compression, regeneration, detector-aware attacks, and watermark forgery that can undermine authenticity protections.

  2. Oct 13, 20251y ago

    Queen’s University researchers analyze AI image watermarking security

    Researchers at Queen’s University in Canada published an analysis of watermarking for AI-generated images, examining embedding, verification, attack channels, and detection across traditional, deep learning, and diffusion-based methods.

  3. Oct 12, 20251y ago

    Commentary calls for stronger verification standards for AI video generation

    A published analysis argued that access to advanced AI video generation such as ChatGPT's Sora2 demands new digital verification standards, framing synthetic video as a growing security and trust problem.

  4. May 21, 20242y ago

    OpenAI backs the C2PA provenance framework

    OpenAI expressed support for the C2PA framework for content provenance, signaling alignment with broader industry efforts to verify and label synthetic media.

  5. May 21, 20242y ago

    EU AI Act includes transparency rules for AI-generated content

    The EU AI Act established transparency provisions requiring disclosure around certain AI-generated or manipulated content, adding regulatory pressure for verification standards.

  6. Aug 29, 20233y ago

    Google introduces SynthID for AI-generated media watermarking

    Google launched SynthID as an industry mechanism for watermarking AI-generated content, illustrating growing commercial adoption of provenance and authenticity controls.

  7. Jan 10, 20234y ago

    China mandates labeling of AI-generated content

    China introduced requirements for marking AI-generated content, reflecting one of the earliest policy moves cited in the references toward authenticity and accountability for synthetic media.

Linked entities

Related entities

Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.

Open in app →

Organizations

1 linked
Openai
Do you run this?Follow how adversaries are adapting to this technology, and where it touches your stack today.
Track This Risk
Mallory platform

See the full picture, correlated to your attack surface.

This page covers what’s public. Mallory adds the parts that aren’t: which of your assets are affected, which threat actors are using it right now, and what to deploy next.

Exposure mappingMap indicators from this story to your assets and identify affected systems in minutes.
Threat actor evidenceEvery observed campaign, victim, and pivot linked to actors named in this story.
Associated malwareMalware, exploits, and IOCs connected to the activity described here.
Detection signaturesYARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Scheduled alertsGet matching new stories delivered to your team as they break, not the next morning.
AI threadsAsk questions about this story and take action on the answers.