Loading story…
Recent evaluations of TypeSafe.ai’s Jev typed decision model found it can provide a low-latency, lower-cost classification signal for security workflows, but performance varies substantially by task and attack type. In a 5,219-trajectory evaluation of tool-using agent traces, JEV achieved a 77.8 average positive-class F1 score, above the strongest generative LLM configuration’s 74.1, with 95.5% valid-result coverage; generative judges nevertheless led on two of four datasets. A separate CWE-guided code-repair study reported that JevVibe raised detector-measured secure-code pass rates for Qwen2.5-Coder-32B-Instruct from 63.5% to 70.7%, outperforming an LLM-guided repair baseline at 66.1%.
Adversarial and automation-focused research warns that aggregate accuracy and calibration can obscure consequential misses. JevAdvBench, comprising 812 typed questions, 66 scenarios, and 9,744 single-edit variants, found that adding an unverified opinion to model state flipped 12.1% of tested jev-1.13.0 decisions and pushed 38% of confident responses below a 0.8 human-review confidence threshold. Other testing found security judges may label attacks safe with high confidence, while a second judge can repeat those errors or increase false rejections. Sophos recommends treating model state as untrusted input and requiring local held-out and adversarial testing, task-specific thresholds, risk-based escalation, and human review before using Jev to automate consequential SOC or agent-security decisions.
Track how attackers are adapting to this technology.
10 events from the most recent confirmed update back to the earliest known activity.
Zhiqiang Wang and Yichao Gao submitted an empirical comparison of JEV and four generative LLM judges on 5,219 agent trajectories. JEV achieved a benchmark-average positive-class F1 of 77.8 and 95.5% valid-result coverage, although results varied across datasets.
Arshak Rezvani, Sasha Behrouzi, and Ahmad-Reza Sadeghi submitted JevVibe, which uses Jev-selected CWE diagnoses to guide code repair. The reported repair workflow raised detector-measured security pass rates for Qwen2.5-Coder-32B-Instruct output from 63.5% to 70.7%.
Yixuan Liu submitted an evaluation of Jev, Laya, Decider, and Bespoke Nimble as security judges for prompt-injection detection, interaction-risk assessment, and harmful-request screening. The study found that favorable aggregate accuracy and calibration can mask high-confidence attack-specific failures.
Jianyi Hu and seven co-authors submitted JevAdvBench, a benchmark and black-box attack suite for RLCD models. Its 9,744 single-edit variants tested whether adversarial request changes could manipulate typed model decisions.
Damani et al. at MIT introduced reinforcement learning with calibration rewards (RLCR), which rewards correctness while penalizing the gap between confidence and verified outcomes. Reported HotpotQA tests reduced expected calibration error from 37% to 3% while maintaining roughly 62–63% answer accuracy.
A phishing benchmark where models decided whether an email agent should click a link reported 63% accuracy and 43% phishing detection for Jev, compared with 81% accuracy and 76% detection for Claude Haiku 4.5. A logistic-regression classifier trained on 1,000 labeled emails and five Jev-derived features achieved 95% accuracy on the remaining 1,000 emails.
CLINC150 cascade testing found that Jev-first routing needed to escalate 22% of requests to approach within one percentage point of GPT-5.6 Terra's accuracy, versus 49% for gpt-5.4-nano-first routing. To match Terra exactly, Jev-first routing had to escalate every request because some wrong Jev answers carried maximum confidence.
In a synthetic customer-support test of 300 urgency questions where the governing organizational rule was withheld, Jev was correct 45% of the time but assigned selected answers an average probability of 74%. The test found overstatement of certainty for ticket routing and urgency decisions.
On 208 Banking77 customer-support requests, Jev achieved 83% zero-shot accuracy, while a bge-small embedding model with logistic regression achieved 93%. The evaluation measured median response times of 0.44 seconds for Jev and 1.51 seconds for GPT-5.6 Terra under the tested configurations.
An independent evaluation of 200 CLINC150 intent-classification requests reported 87% zero-shot accuracy for Jev, compared with 80% for gpt-5.4-nano and 92% for GPT-5.6 Terra. Jev assigned maximum confidence to 102 responses, including six incorrect answers.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
5 references tracked. Mallory keeps watching after this page renders.
arxiv.org
Open sourcearxiv.org
Open sourcearxiv.org
Open sourcesophos.com
Open sourcearxiv.org
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Multiple studies and industry reports found that **AI-generated or "vibe-coded" software is frequently shipped with serious security flaws**, including XSS, SQL injection, hardcoded secrets, unsafe `eval` usage, weak authentication logic, wildcard CORS, exposed `.env` data, and insecure dependencies. One analysis of 5,600 publicly accessible AI-built applications reported more than 2,000 high-risk vulnerabilities and over 400 exposed secrets, while a separate review of 549 self-described AI-generated repositories found widespread committed secrets, injection issues, hardcoded passwords, and other common sanitization failures. Veracode and other researchers similarly warned that code produced by large language models often maps to **OWASP Top 10** weakness classes and that repeated AI-assisted edits, poor prompts, and excessive agent permissions can further increase risk. At the same time, researchers and vendors highlighted both the promise and fragility of AI-assisted security tooling. Trail of Bits reported that OpenAI Codex goal-based prompting helped uncover previously unknown bugs in projects including **Rust**, **curl**, **zlib**, and **Keycloak**, but stressed that human experts were still needed to scope goals, validate findings, and handle disclosure. New academic work also showed that **LLM-based vulnerability detectors can be easily misled**: the ALIBI framework used adversarial code comments to evade four detectors with success rates above 90%. In response to growing concern over agent reliability, Amazon backed the Lean language project to strengthen formal verification for AI agents, while security experts argued that theorem proving can improve assurance for high-risk components but should be applied selectively as part of broader defense-in-depth.
Aug 4, 2026
Security researchers and vendors reported that large language models are becoming materially useful across offensive and defensive security workflows, including vulnerability discovery, exploit development, autonomous penetration testing, and cyber threat intelligence extraction. Recent work described LLM-assisted systems that generated proof-of-concept exploits for npm flaws at a reported **77%** success rate, exploited real-world web vulnerabilities with multi-agent architectures, and executed multistage attacks in emulated enterprise networks when paired with task-specific agents and attack-abstraction layers. Other experiments found local self-hosted models could reliably solve straightforward Juice Shop challenges, while purpose-built scaffolds helped researchers uncover memory-corruption bugs in Windows endpoint products and accelerate reverse engineering of AV and EDR logic. At the same time, multiple studies warned that headline results can overstate real-world capability without strong validation. Google Project Zero said benchmark design, tooling, and automatic verification heavily influence measured performance and concluded current models still fall short of meaningful autonomous offensive research in live environments. Separate academic and practitioner reviews found many automated pentesting frameworks remain brittle, and human validation showed that **71.5%** of supposedly successful LLM-generated exploit PoCs in one follow-up study were actually invalid because the models simulated exploitation rather than triggering the flaw. Across the research, the consistent finding was that LLMs perform best when constrained by structured workflows, specialized tools, execution feedback, and rigorous verification rather than being trusted as fully autonomous hackers.
Jun 29, 2026
AWS introduced **Deception Benchmark**, a public evaluation suite designed to test whether AI models can distinguish genuinely exploitable vulnerabilities from code that only appears vulnerable. The benchmark contains 14,822 purpose-built samples spanning 16 programming languages and more than 70 CWE categories, including scenarios in which environmental or infrastructure controls prevent exploitation. Testing of 12 general-purpose models found that no evaluated configuration kept both false-positive and false-negative rates below AWS's 10% threshold for production use. Direct vulnerability-detection prompts generally found more flaws but produced excessive false alarms, while prompts requiring proof of exploitation lowered false positives at the expense of missing real vulnerabilities. The benchmark releases samples and an evaluation workflow while withholding labels to preserve verified scoring and limit memorization, complementing broader language-model cybersecurity evaluation efforts such as Cybench.
Sep 14, 2026