AWS Deception Benchmark Finds AI Security Models Miss Production Accuracy Thresholds
Emerging Technology Risk · 3 sources · Updated 2mo ago · First seen Sep 9, 2026

AWS introduced Deception Benchmark, a public evaluation suite designed to test whether AI models can distinguish genuinely exploitable vulnerabilities from code that only appears vulnerable. The benchmark contains 14,822 purpose-built samples spanning 16 programming languages and more than 70 CWE categories, including scenarios in which environmental or infrastructure controls prevent exploitation.
Testing of 12 general-purpose models found that no evaluated configuration kept both false-positive and false-negative rates below AWS's 10% threshold for production use. Direct vulnerability-detection prompts generally found more flaws but produced excessive false alarms, while prompts requiring proof of exploitation lowered false positives at the expense of missing real vulnerabilities. The benchmark releases samples and an evaluation workflow while withholding labels to preserve verified scoring and limit memorization, complementing broader language-model cybersecurity evaluation efforts such as Cybench.


