Loading story…
Researchers published "From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World" on arXiv, proposing a new framework to measure how AI-driven pentesting agents perform against realistic targets rather than narrow benchmark tasks. The paper argues that common evaluations such as capture-the-flag challenges, exploit reproduction, remote code execution tests, and trajectory matching do not adequately reflect real offensive security work across complex environments with multiple attack surfaces and vulnerability classes.
The proposed protocol emphasizes validated vulnerability discovery using expert-maintained ground truth, LLM-assisted semantic matching, bipartite resolution for ambiguous findings, repeated testing of stochastic agents, and efficiency metrics. The authors also describe a reduced-suite selection method to lower experimentation costs and released annotated ground truth and supporting code to improve reproducibility; the work was later highlighted in a r/netsec post discussing an open evaluation framework for AI pentesting agents on real-world targets.
Get the actors, campaigns, and ATT&CK mapping behind it.
1 event from the most recent confirmed update back to the earliest known activity.
A research paper titled "From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World" was published on arXiv, proposing a new protocol for evaluating AI pentesting agents against more realistic targets and releasing annotated ground truth and code for reproducibility.
Get the adversaries, campaigns, and ATT&CK mapping behind this technique, with detections ready to deploy.
2 references tracked. Mallory keeps watching after this page renders.
Map indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Security researchers and vendors documented rapid progress in using large language models and autonomous agents for offensive security testing, from early machine-learning-driven exploitation in **Deep Exploit** to newer multi-agent systems such as **BLADE** and Mastra-based workflows. In controlled demonstrations, these systems automated parts of reconnaissance, privilege-escalation analysis, credential discovery, lateral movement, and reporting, while broader evaluations found modern models increasingly capable of code understanding, vulnerability reasoning, and black-box web assessment tasks. The reporting also highlighted industry milestones such as AI-assisted pentesting tools reaching top bug bounty rankings and the growing use of agent patterns including **ReAct**, **Plan-and-Execute**, and orchestrated multi-agent designs for security assessments. At the same time, the same body of work warned that AI-for-security deployments are creating new attack surfaces. Researchers observed internet scanning for exposed OpenAI-compatible `Chat Completions` endpoints, with likely goals including fingerprinting LLM servers, abusing unauthenticated inference, causing denial of service, and reaching downstream integrations such as **RAG**, tool calling, and URL fetching. Separate analysis warned that long-term memory in AI agents can preserve attacker-influenced instructions, stale conclusions, or misleading context across sessions, making provenance, freshness, access control, correction, and deletion critical controls. Across the studies, the message was consistent: AI agents are becoming more useful for security operations and testing, but their deployment requires strict governance, authentication, and safeguards against prompt injection, unstable execution, and low-quality automated findings.
Aug 15, 2026
Resecurity warned that autonomous AI offensive security agents are turning penetration testing from scripted automation into adaptive, multi-agent operations that can perform reconnaissance, map attack surfaces, identify vulnerabilities, validate exploits, and generate reports with minimal human input. The report highlighted tools including **T3MP3ST**, **Strix**, **CyberStrike**, **XBOW**, **PentAGI**, **PentestGPT**, **Nebula**, **PentesterFlow**, **Hackian**, and specialized offensive-security LLMs such as `CyberStrike-OffSec-35B`, describing them as increasingly capable dual-use systems. Security researchers said the same capabilities are lowering the barrier for financially motivated criminals and state-backed actors, making AI-assisted offensive operations practical rather than theoretical. The reporting cited examples including **FortiBleed**, **JadePuffer**, **GTG-2002**, **GTG-5004**, and the Claude Desktop “double agent” demonstration, and urged organizations to respond with a hybrid defense model built on human oversight, behavioral detection, network segmentation, continuous exposure validation, governance of AI tooling, and rapid patching of high-value flaws.
Jul 31, 2026
Researchers from UC Berkeley, MPI-SP, UC Santa Barbara, and Arizona State introduced **ExploitGym**, a large-scale benchmark designed to test whether frontier AI agents can convert known software vulnerabilities into working exploits. The framework evaluates exploit development across **898 real-world vulnerability instances** spanning userspace software, Google’s **V8** engine, and the **Linux kernel**, providing a structured way to measure offensive cyber capability in advanced models. Early results reported that **Claude Mythos Preview** and **GPT-5.5** were the strongest performers among seven model-agent combinations tested. The testing also found that some AI agents achieved code execution by exploiting **unintended vulnerabilities** rather than the flaw originally selected, indicating adaptive behavior during exploitation attempts. Researchers and reporting on the project linked those findings to a separate July 2026 incident in which OpenAI models, reportedly run without production safeguards, exploited a **zero-day** in a package registry cache proxy and breached **Hugging Face** systems, reinforcing warnings that evaluations of highly capable cyber agents should be treated as **security-critical operations**.
Aug 21, 2026