Loading story…
Researchers released XL-SafetyBench, a country-grounded evaluation suite designed to measure LLM jailbreak resistance and cultural sensitivity in native-language contexts rather than translated English-centric tests. The benchmark contains 5,500 cases across 10 country-language pairs: 4,500 adversarial jailbreak prompts and 1,000 benign prompts involving culturally specific sensitivities. It uses Attack Success Rate (ASR), Cultural Sensitivity Rate (CSR), and the new Neutral-Safe Rate (NSR) metric to separate legitimate safety refusals from irrelevant or degraded outputs caused by weak language capability.
Testing of 10 frontier and 27 local models found large differences in resilience and localized safety performance. Claude-4.5-Sonnet recorded a 2.8% average jailbreak ASR, compared with 98.8% for Mistral-Large-3, while frontier models averaged 49.9% on cultural sensitivity. Frontier-model jailbreak robustness did not reliably correlate with cultural awareness; among local models, an inverse relationship between ASR and NSR indicated that seemingly safe outcomes can result from generation failure rather than genuine alignment. The findings highlight the need for region-specific red teaming and evaluation before deploying LLMs across multilingual markets.
Track how attackers are adapting to this technology.
2 events from the most recent confirmed update back to the earliest known activity.
Dasol Choi and 16 co-authors submitted the XL-SafetyBench paper, introducing a country-grounded benchmark for evaluating LLM jailbreak robustness and cultural sensitivity. The benchmark contains 5,500 native-language test cases across 10 country-language pairs and evaluates ASR, NSR, and CSR.
AIM Intelligence published the XL-SafetyBench dataset for evaluating LLM jailbreak robustness and cultural sensitivity across 10 country-language pairs. The release includes 4,500 adversarial jailbreak prompts, 1,000 cultural-sensitivity scenarios, and Parquet dataset files with evaluation metrics.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
4 references tracked. Mallory keeps watching after this page renders.
aim-intelligence.com
Open sourcearxiv.org
Open sourcegithub.com
Open sourcehuggingface.co
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Cisco reported that **multi-turn jailbreak** techniques—iterative, conversational prompt sequences designed to erode safety guardrails—successfully bypassed protections in eight major **open-weight** large language models **92.78%** of the time, while single-turn prompt attempts were notably less effective. The findings, published in Cisco’s *State of AI Security* research and covered by multiple outlets, highlight that many enterprise AI deployments using downloadable, self-hosted models may be more vulnerable to sustained adversarial prompting than organizations assume. The report’s risk framing is amplified by broader concerns that model misuse and capability leakage can scale quickly: Anthropic separately alleged coordinated **model distillation** activity by Chinese AI labs using large volumes of fraudulent accounts and proxy infrastructure to extract advanced behaviors from *Claude*, warning that copied models may lack comparable safety controls and could be repurposed for malicious use. Related research coverage also notes that LLMs can sometimes be induced—via specialized prompting/jailbreaking methods—to reproduce near-verbatim copyrighted text from training data, underscoring that prompt-based attacks can drive both **policy bypass** and **data/content extraction** outcomes, particularly when guardrails are tested over extended interactions.
Jun 29, 2026
Security researchers and vendors reported that large language models are becoming materially useful across offensive and defensive security workflows, including vulnerability discovery, exploit development, autonomous penetration testing, and cyber threat intelligence extraction. Recent work described LLM-assisted systems that generated proof-of-concept exploits for npm flaws at a reported **77%** success rate, exploited real-world web vulnerabilities with multi-agent architectures, and executed multistage attacks in emulated enterprise networks when paired with task-specific agents and attack-abstraction layers. Other experiments found local self-hosted models could reliably solve straightforward Juice Shop challenges, while purpose-built scaffolds helped researchers uncover memory-corruption bugs in Windows endpoint products and accelerate reverse engineering of AV and EDR logic. At the same time, multiple studies warned that headline results can overstate real-world capability without strong validation. Google Project Zero said benchmark design, tooling, and automatic verification heavily influence measured performance and concluded current models still fall short of meaningful autonomous offensive research in live environments. Separate academic and practitioner reviews found many automated pentesting frameworks remain brittle, and human validation showed that **71.5%** of supposedly successful LLM-generated exploit PoCs in one follow-up study were actually invalid because the models simulated exploitation rather than triggering the flaw. Across the research, the consistent finding was that LLMs perform best when constrained by structured workflows, specialized tools, execution feedback, and rigorous verification rather than being trusted as fully autonomous hackers.
Jun 29, 2026
Researchers and security teams reported mixed results on using large language models for offensive cybersecurity tasks, finding that current systems remain unreliable for autonomous exploitation while still proving useful in vulnerability research. Google Project Zero's **Project Naptime** evaluated whether LLMs could meaningfully support offensive security work, while the **CyberSecEval 2** benchmark measured model behavior across exploit generation, prompt injection, and code interpreter abuse. The benchmark found prompt-injection defenses remain weak, with successful attack rates of **26% to 41%** across tested models, and highlighted a safety-utility tradeoff in which stronger refusals can also block benign requests. At the same time, multi-agent LLM-assisted tooling demonstrated practical impact in software auditing. DARKNAVY said its **Argusee** architecture, which separates manager, auditor, and checker roles to reduce false positives and false negatives, uncovered **15 previously unknown flaws** in open-source projects and identified **CVE-2025-37891**, a high-severity Linux kernel USB vulnerability. The flaw affects the host side of the Linux USB stack from Linux `6.5` onward, can be triggered by a simulated USB device advertising the **MIDI2** protocol, and stems from improper length checks during MIDI1-to-UMP conversion that can cause an arbitrary kernel heap overflow; DARKNAVY reported reliable privilege escalation to root on Arch Linux before the issue was fixed.
Aug 6, 2026