Loading story…
Red Hat and Purdue University released VEX-Bench, a 75-case benchmark for testing whether LLM agents can determine if CVEs in third-party dependencies are actually exploitable in downstream Go, Python, and Java projects. Covering 67 CVEs across 35 open-source projects, the expert-labeled dataset measures both binary exploitability decisions and the technical justification for them. It addresses the operational burden of dependency-scanner alerts: 70.7% of scanner-flagged cases in the dataset were not exploitable in the affected application.
Across nine models and three agent harnesses, Claude Opus 4.6 achieved the strongest binary-status result, with 81.6% F1, while GPT-5.5 delivered the highest precision at 88.7% and was the only evaluated model to exceed 70% macro-F1 on fine-grained justification classification. Conventional software-composition-analysis tools performed materially worse, with OSV-Scanner reaching 40.5% F1 and Trivy 36.1%. The publicly available benchmark and code, accepted for EMNLP 2026, provide a test bed for evaluating whether automated triage can reduce false positives without sacrificing defensible exploitability analysis.
Track how attackers are adapting to this technology.
2 events from the most recent confirmed update back to the earliest known activity.
Jiahao Shi and 12 co-authors submitted a paper introducing VEX-Bench, a benchmark for evaluating whether LLM agents can determine if known dependency CVEs are exploitable in downstream projects. The benchmark includes 75 expert-labeled GitHub-mined cases across Python, Java, and Go.
Red Hat and Purdue University researchers described VEX-Bench, covering 75 cases involving 67 CVEs across 35 open-source projects. Their evaluation found Claude Opus 4.6 reached 81.6% binary-status F1, while GPT-5.5 was the only evaluated model above 70% macro-F1 for fine-grained justifications; the project code and dataset were made publicly available.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
2 references tracked. Mallory keeps watching after this page renders.
research.redhat.com
Open sourcearxiv.org
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Security researchers and vendors reported that large language models are becoming materially useful across offensive and defensive security workflows, including vulnerability discovery, exploit development, autonomous penetration testing, and cyber threat intelligence extraction. Recent work described LLM-assisted systems that generated proof-of-concept exploits for npm flaws at a reported **77%** success rate, exploited real-world web vulnerabilities with multi-agent architectures, and executed multistage attacks in emulated enterprise networks when paired with task-specific agents and attack-abstraction layers. Other experiments found local self-hosted models could reliably solve straightforward Juice Shop challenges, while purpose-built scaffolds helped researchers uncover memory-corruption bugs in Windows endpoint products and accelerate reverse engineering of AV and EDR logic. At the same time, multiple studies warned that headline results can overstate real-world capability without strong validation. Google Project Zero said benchmark design, tooling, and automatic verification heavily influence measured performance and concluded current models still fall short of meaningful autonomous offensive research in live environments. Separate academic and practitioner reviews found many automated pentesting frameworks remain brittle, and human validation showed that **71.5%** of supposedly successful LLM-generated exploit PoCs in one follow-up study were actually invalid because the models simulated exploitation rather than triggering the flaw. Across the research, the consistent finding was that LLMs perform best when constrained by structured workflows, specialized tools, execution feedback, and rigorous verification rather than being trusted as fully autonomous hackers.
Jun 29, 2026
Researchers and security teams reported mixed results on using large language models for offensive cybersecurity tasks, finding that current systems remain unreliable for autonomous exploitation while still proving useful in vulnerability research. Google Project Zero's **Project Naptime** evaluated whether LLMs could meaningfully support offensive security work, while the **CyberSecEval 2** benchmark measured model behavior across exploit generation, prompt injection, and code interpreter abuse. The benchmark found prompt-injection defenses remain weak, with successful attack rates of **26% to 41%** across tested models, and highlighted a safety-utility tradeoff in which stronger refusals can also block benign requests. At the same time, multi-agent LLM-assisted tooling demonstrated practical impact in software auditing. DARKNAVY said its **Argusee** architecture, which separates manager, auditor, and checker roles to reduce false positives and false negatives, uncovered **15 previously unknown flaws** in open-source projects and identified **CVE-2025-37891**, a high-severity Linux kernel USB vulnerability. The flaw affects the host side of the Linux USB stack from Linux `6.5` onward, can be triggered by a simulated USB device advertising the **MIDI2** protocol, and stems from improper length checks during MIDI1-to-UMP conversion that can cause an arbitrary kernel heap overflow; DARKNAVY reported reliable privilege escalation to root on Arch Linux before the issue was fixed.
Aug 6, 2026
Semgrep evaluated Claude Opus 5.5 using the 186-task **SusVibes v1.0** benchmark for agent-generated code. The model produced functionally working solutions for 93.5% of tasks, but only 54.8% were both correct and secure. Among 174 functionally correct solutions, 72 retained the underlying vulnerability, showing that passing functional tests did not reliably demonstrate secure implementation. The evaluation found that 53% of assessable outputs were identical or nearly identical to public reference implementations, suggesting benchmark results were materially affected by recall of publicly available CVE fixes and vulnerable code. Claude Opus 5.5 performed particularly poorly on cross-cutting flaws involving cryptography, concurrency, authentication, and access control, while performing better on smaller fixes and some categories such as XSS. The findings reinforce research questioning whether public-CVE coding benchmarks measure secure reasoning for novel tasks rather than training-data memorization.
Sep 29, 2026