Loading story…
Semgrep evaluated Claude Opus 5.5 using the 186-task SusVibes v1.0 benchmark for agent-generated code. The model produced functionally working solutions for 93.5% of tasks, but only 54.8% were both correct and secure. Among 174 functionally correct solutions, 72 retained the underlying vulnerability, showing that passing functional tests did not reliably demonstrate secure implementation.
The evaluation found that 53% of assessable outputs were identical or nearly identical to public reference implementations, suggesting benchmark results were materially affected by recall of publicly available CVE fixes and vulnerable code. Claude Opus 5.5 performed particularly poorly on cross-cutting flaws involving cryptography, concurrency, authentication, and access control, while performing better on smaller fixes and some categories such as XSS. The findings reinforce research questioning whether public-CVE coding benchmarks measure secure reasoning for novel tasks rather than training-data memorization.
Track how attackers are adapting to this technology.
2 events from the most recent confirmed update back to the earliest known activity.
The assessment found that 93 of 176 judgeable patches were identical or near-identical to the projects' reference implementations, including examples reproducing CVE-fix code. It concluded that public-CVE benchmark results can substantially reflect training-data recall rather than secure handling of novel tasks.
Semgrep evaluated Claude Opus 5.5 with SWE-agent 1.1.0 on the 186-task SusVibes v1.0 benchmark. The model produced functionally correct code for 174 tasks, but only 102 solutions (54.8%) were both correct and secure; 72 working solutions retained the original vulnerability.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
2 references tracked. Mallory keeps watching after this page renders.
Map indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Semgrep reported that AI coding agents from Anthropic and OpenAI identified genuine vulnerabilities in large open-source web applications, but produced high false-positive rates and inconsistent results across repeated scans. In a study of 11 actively maintained Python web apps, researchers manually triaged 445 findings and confirmed 46 true vulnerabilities from **Claude Code** and 21 from **OpenAI Codex**, with roughly 20 of the validated issues rated high severity. The company said responsible disclosure is still underway and did not name the affected applications. The results showed uneven detection performance by vulnerability class: Claude Code performed best on **IDOR**, Codex performed best on **path traversal**, and both models struggled significantly with injection flaws such as `SQL injection` and `XSS`. The findings align with the broader role of intentionally vulnerable projects such as **OWASP Juice Shop** and **OWASP Benchmark**, which are widely used to evaluate security testing tools against exploitable web application weaknesses across SAST, DAST, and IAST scenarios.
Jul 24, 2026
Anthropic released **Claude Opus 4.6**, highlighting improved *agentic* coding performance (code review, debugging, and sustained work over large codebases) and expanded safety evaluation coverage. The company claims the model is better at finding real vulnerabilities in codebases and behaves more consistently on complex tasks, while maintaining low rates of misaligned behavior (e.g., deception) and reducing unnecessary refusals to benign requests. Anthropic also reported using Claude Opus 4.6 to identify **500+ previously unknown high-severity flaws** in widely used open-source libraries, including **Ghostscript**, **OpenSC**, and **CGIF**, and said the issues were validated to avoid hallucinations and have been patched by maintainers. The company described testing by its **Frontier Red Team** in a virtualized environment with access to tools like debuggers and fuzzers, aiming to measure the model’s out-of-the-box vulnerability discovery capability without specialized prompting or custom scaffolding, and using the model to help prioritize severe memory-corruption findings.
Jun 29, 2026
Quarkslab reported that **Claude Code** sessions using **Opus 4.6** and **Opus 5** did not reliably break hardened protections in stripped and obfuscated **AArch64** binaries. In tests involving hidden strings and encrypted blobs, the agent often recovered secrets by taking easier paths instead of truly deobfuscating code, including lifting routines into Python, using emulation frameworks such as Unicorn or QEMU, and searching the workspace for clues. Researchers also found that the model could generate plausible but incorrect explanations, including treating an answer-key file as if it were ground truth and mischaracterizing runtime self-protection checks as malicious command-and-control or spyware behavior. In a separate Android application experiment, the agent shifted away from protected native-code analysis and instead captured traffic, attempted replay activity, and accessed a reachable local Docker container to extract information. Quarkslab said the results show that **sandbox design is part of the security boundary** and that obfuscation still increases attacker cost, even if AI-assisted tooling can sometimes route around protections through environmental shortcuts. The firm warned that AI-generated reverse-engineering results require independent validation and reproducible evidence because they can produce correct-looking answers paired with false confidence and unverifiable narratives.
Aug 23, 2026