Skip to main content
Mallory
← Back to News
ai-platform-securitystandards-framework-update

AWS Deception Benchmark Finds AI Security Models Miss Production Accuracy Thresholds

Emerging Technology Risk · 3 sources · Updated 2mo ago · First seen Sep 9, 2026

AWS Deception Benchmark Finds AI Security Models Miss Production Accuracy Thresholds

AWS introduced Deception Benchmark, a public evaluation suite designed to test whether AI models can distinguish genuinely exploitable vulnerabilities from code that only appears vulnerable. The benchmark contains 14,822 purpose-built samples spanning 16 programming languages and more than 70 CWE categories, including scenarios in which environmental or infrastructure controls prevent exploitation.

Testing of 12 general-purpose models found that no evaluated configuration kept both false-positive and false-negative rates below AWS's 10% threshold for production use. Direct vulnerability-detection prompts generally found more flaws but produced excessive false alarms, while prompts requiring proof of exploitation lowered false positives at the expense of missing real vulnerabilities. The benchmark releases samples and an evaluation workflow while withholding labels to preserve verified scoring and limit memorization, complementing broader language-model cybersecurity evaluation efforts such as Cybench.

New surface, new adversariesTrack how attackers are adapting to this technology.
Track this risk
Event timeline

How this story unfolded

Events from the most recent confirmed update back to the earliest known activity.

1 event
  1. Reported Sep 9, 20262mo ago

    AWS releases Deception Benchmark for AI vulnerability assessment

    AWS publicly released the Deception Benchmark, comprising purpose-built vulnerable and safe code/deployment scenarios to evaluate whether AI models can distinguish exploitable vulnerabilities from deceptive false alarms. Its evaluation of 12 general-purpose models found no tested model-and-prompting configuration kept both false-positive and false-negative rates below 10%.

    The state of AI for security: Measuring what matters most for building trust | AWS Security Blog
Linked entities

Related entities

Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.

Open in app →

Organizations

1 linked
Amazon Web Services
Do you run this?Follow how adversaries are adapting to this technology, and where it touches your stack today.
Track This Risk
Mallory platform

See the full picture, correlated to your attack surface.

This page covers what’s public. Mallory adds the parts that aren’t: which of your assets are affected, which threat actors are using it right now, and what to deploy next.

Exposure mappingMap indicators from this story to your assets and identify affected systems in minutes.
Threat actor evidenceEvery observed campaign, victim, and pivot linked to actors named in this story.
Associated malwareMalware, exploits, and IOCs connected to the activity described here.
Detection signaturesYARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Scheduled alertsGet matching new stories delivered to your team as they break, not the next morning.
AI threadsAsk questions about this story and take action on the answers.