Skip to main content
Mallory
← Back to News
ai-platform-securitydata-exfiltration-methodadversary-emulation-tradecraftai-enabled-threat-activity

Security Risks of Open-Source AI Pentesting and Agent Automation

Emerging Technology Risk · 2 sources · Updated 4mo ago · First seen Oct 23, 2025

Security Risks of Open-Source AI Pentesting and Agent Automation

Open-source AI pentesting frameworks and automated AI agents are introducing new security risks to organizations by automating red teaming and system command execution. These tools, such as Cyber-AutoAgent and Villager, leverage large language models (LLMs) to chain together reconnaissance, exploitation, and lateral movement, often sending sensitive pentest data—including internal IPs, credentials, and configuration files—to external APIs or model endpoints outside organizational control. This creates a significant risk of unauthorized data exfiltration, as these frameworks frequently lack controls to restrict data flow, do not provide adequate logging or audit trails, and may use public API keys or default to external inference endpoints, making it difficult for traditional security tools to detect such egress.

Additionally, modern AI agents that automate filesystem operations and code analysis are vulnerable to argument injection attacks, which can bypass human approval mechanisms and lead to remote code execution (RCE). These vulnerabilities stem from design antipatterns in how commands are executed, with attackers able to exploit pre-approved commands to gain unauthorized access. Security experts recommend mitigating these risks by improving command execution design, implementing sandboxing, and ensuring argument separation, as well as increasing awareness among developers and security engineers about the potential for RCE in AI-powered automation tools.

New surface, new adversariesTrack how attackers are adapting to this technology.
Track this risk
Event timeline

How this story unfolded

Events from the most recent confirmed update back to the earliest known activity.

2 events · Oct 22, 2025 – Oct 23, 2025
  1. Oct 23, 20251y ago

    Horizon3.ai warns about risks from open-source AI pentesting tools

    Horizon3.ai published analysis warning that open-source AI pentesting could become a security incident, highlighting emerging offensive and defensive risks around AI-enabled security tooling.

  2. Oct 22, 20251y ago

    Trail of Bits publishes prompt-injection-to-RCE research on AI agents

    Trail of Bits published research describing how prompt injection can lead to remote code execution in AI agents, framing the issue as a security risk in agentic AI systems.

Linked entities

Related entities

Vulnerabilities, threat actors, malware, products, organizations, breaches, and observables Mallory has linked to this story. Indicator values are masked here and available in full in the app.

Open in app →

Observables

1 linked

Indicator values are masked on this page. See the values in Mallory Domains, IPs, hashes, and URLs are exportable to your SIEM.

Do you run this?Follow how adversaries are adapting to this technology, and where it touches your stack today.
Track This Risk
Mallory platform

See the full picture, correlated to your attack surface.

This page covers what’s public. Mallory adds the parts that aren’t: which of your assets are affected, which threat actors are using it right now, and what to deploy next.

Exposure mappingMap indicators from this story to your assets and identify affected systems in minutes.
Threat actor evidenceEvery observed campaign, victim, and pivot linked to actors named in this story.
Associated malwareMalware, exploits, and IOCs connected to the activity described here.
Detection signaturesYARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Scheduled alertsGet matching new stories delivered to your team as they break, not the next morning.
AI threadsAsk questions about this story and take action on the answers.