Loading story…
Security researchers and industry experts are increasingly focused on the challenges and opportunities of applying red teaming and blue teaming methodologies to artificial intelligence (AI) systems, particularly those used for automated code generation. Red teaming in this context involves simulating adversarial attacks against AI models to uncover vulnerabilities, such as the ability to bypass restrictions (AI jailbreaks) or generate insecure code. Blue teaming, on the other hand, is concerned with developing defensive mechanisms to detect and prevent these failures, ensuring that code generation models do not inadvertently produce vulnerable or malicious code. The evolving landscape of AI security requires coordinated efforts to probe, evaluate, and harden these systems against both known and emerging threats.
Recent research highlights the limitations of current blue teaming approaches, including poor alignment with security concepts, over-conservatism leading to false positives, and incomplete risk coverage. To address these issues, new solutions like BlueCodeAgent are being developed, leveraging automated red teaming to improve the detection and mitigation of security flaws in code generation AI. The integration of threat intelligence, adversarial testing, and robust evaluation frameworks is critical to maintaining the trustworthiness and safety of AI-driven software development tools. As AI technologies continue to advance, the interplay between offensive (red team) and defensive (blue team) strategies will be essential for securing the next generation of software systems.
Track how attackers are adapting to this technology.
2 events from the most recent confirmed update back to the earliest known activity.
The OSINT Team blog published an article exploring challenges and opportunities in AI red teaming. The reference appears to be general analysis and does not describe a separate concrete security event beyond the publication itself.
Microsoft Research published a blog post introducing BlueCodeAgent, a blue-teaming agent for CodeGen AI enabled by automated red teaming. The reference indicates a research disclosure rather than a real-world incident or campaign.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
2 references tracked. Mallory keeps watching after this page renders.
Map indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
The widespread adoption of generative AI (GenAI) tools in software development has significantly increased productivity, enabling developers to document, write, and optimize code at unprecedented speeds. According to a 2023 McKinsey study, organizations have rapidly integrated AI into their development workflows, with 83% using AI for code creation and 57% relying on AI-powered coding tools as a standard practice. However, this surge in AI-assisted coding has introduced new security risks, as traditional security models focused on perimeter or infrastructure controls do not adequately protect the data and code generated by these tools. Studies have revealed that nearly half of code snippets produced by popular AI models contain vulnerabilities, underscoring the prevalence of insecure code generation. High-profile incidents, such as Samsung's 2023 ban on ChatGPT following a sensitive code leak, highlight the real-world consequences of insufficient safeguards when using GenAI in development environments. The responsibility for securing data and code remains with developers, even as cloud providers secure the underlying infrastructure. The rapid pace of AI-generated code has outstripped the ability of traditional secure coding training to keep up, shifting the focus from training human programmers to ensuring that AI systems themselves are capable of secure coding. Industry experts note that AI is currently less effective at producing secure code than human programmers, with multiple studies and reports from sources like Schneier on Security, Veracode, and SC Media confirming this trend. The volume of vulnerabilities continues to rise, with over 47,000 publicly known vulnerabilities expected in a single year and at least 130 new vulnerabilities reported daily. This ongoing wave of vulnerabilities leads to constant exploitation and patching, further emphasizing the need for secure coding practices at the AI level. While AI has delivered substantial productivity gains—developers report 30% to 40% increases—these benefits are undermined by the security shortcomings of AI-generated code. The industry is now at a crossroads, where the imperative is to teach AI systems to code securely, rather than relying solely on human oversight or post-development security reviews. Integrating security into the AI coding process and providing developers with tools that embed data protection are seen as essential steps to address these emerging challenges. The shift towards AI-driven development necessitates a reevaluation of security strategies, focusing on proactive measures that align with the realities of modern software engineering. Without such changes, organizations risk exposing themselves to significant security threats stemming from the very tools designed to enhance their productivity.
Jun 29, 2026
Enterprises are accelerating adoption of AI coding assistants and agents, with vendors and engineering leaders reorganizing around tools such as GitHub Copilot, Codex, Claude Code, and formal-verification agents to increase software output and support smaller teams. OpenAI has reportedly shifted strategy toward coding and enterprise customers, while companies such as Cursor are deploying always-on security agents that scan pull requests, patch dependencies, and block risky changes before release. Research and industry reporting also show AI is changing how developers work: Copilot users spend more time coding and less time on collaboration and project-management tasks, and engineering teams increasingly treat AI as a force multiplier rather than a replacement for human developers. At the same time, multiple studies and incident reports say the gains are being offset by a growing **review and supervision tax**. Reports from Harness, DORA, Sonar, and other researchers found that heavier AI use often correlates with more code review effort, higher defect and deployment risk, longer remediation and recovery times, and rising burnout among senior engineers who must validate AI-generated changes. Security researchers at RSAC 2026 said AI assistants reproduce common flaws such as `SSRF`, `XSS`, path traversal, command injection, and open redirects at scale, while reporting tied AI-assisted changes to outages and operational failures, including an AWS disruption. Across the coverage, experts urged organizations to keep humans in the loop, apply stricter governance and testing for AI-generated code, and measure validation workload, defect escape, and technical debt rather than code volume alone.
Jul 1, 2026
Security researchers and vendors documented rapid progress in using large language models and autonomous agents for offensive security testing, from early machine-learning-driven exploitation in **Deep Exploit** to newer multi-agent systems such as **BLADE** and Mastra-based workflows. In controlled demonstrations, these systems automated parts of reconnaissance, privilege-escalation analysis, credential discovery, lateral movement, and reporting, while broader evaluations found modern models increasingly capable of code understanding, vulnerability reasoning, and black-box web assessment tasks. The reporting also highlighted industry milestones such as AI-assisted pentesting tools reaching top bug bounty rankings and the growing use of agent patterns including **ReAct**, **Plan-and-Execute**, and orchestrated multi-agent designs for security assessments. At the same time, the same body of work warned that AI-for-security deployments are creating new attack surfaces. Researchers observed internet scanning for exposed OpenAI-compatible `Chat Completions` endpoints, with likely goals including fingerprinting LLM servers, abusing unauthenticated inference, causing denial of service, and reaching downstream integrations such as **RAG**, tool calling, and URL fetching. Separate analysis warned that long-term memory in AI agents can preserve attacker-influenced instructions, stale conclusions, or misleading context across sessions, making provenance, freshness, access control, correction, and deletion critical controls. Across the studies, the message was consistent: AI agents are becoming more useful for security operations and testing, but their deployment requires strict governance, authentication, and safeguards against prompt injection, unstable execution, and low-quality automated findings.
Aug 15, 2026