Loading story…
Security commentary and research are converging on a concern that today’s LLM evaluation methods do not reflect real-world defensive needs, particularly in security operations. SentinelLABS reports that widely cited benchmarks (including those associated with major vendors) tend to measure narrow, clean, and reproducible tasks that map poorly to SOC workflows, which are continuous, collaborative, and frequently disrupted by changing conditions; it also notes a trust issue where many benchmarks rely on LLMs grading other LLMs, sometimes using the same vendor’s models for both, creating a closed loop that can be gamed.
In parallel, industry discussion is shifting from “general-purpose chatbot” hype toward a future of specialized models (e.g., small language models and task-specific “digital workers”), arguing that AI’s durable value will come from embedding into decision-making and operational processes rather than novelty use cases. Taken together, the sources suggest CISOs should treat vendor benchmark claims cautiously and prioritize evaluations tied to actual SOC outcomes (reliability under change, workflow fit, and measurable defensive impact) rather than generic model scores.
Track how attackers are adapting to this technology.
2 events from the most recent confirmed update back to the earliest known activity.
Securely Built published a piece arguing that organizations should use smaller language models and traditional automation for routine cybersecurity tasks, reserving frontier LLMs for complex reasoning via router-based architectures.
SentinelLABS published an analysis arguing that prominent LLM-in-cybersecurity benchmarks do not measure operationally meaningful SOC and CTI performance, and called for workflow-level evaluations with stronger metrics and judging practices.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
2 references tracked. Mallory keeps watching after this page renders.
Map indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Security researchers and vendors reported that large language models are becoming materially useful across offensive and defensive security workflows, including vulnerability discovery, exploit development, autonomous penetration testing, and cyber threat intelligence extraction. Recent work described LLM-assisted systems that generated proof-of-concept exploits for npm flaws at a reported **77%** success rate, exploited real-world web vulnerabilities with multi-agent architectures, and executed multistage attacks in emulated enterprise networks when paired with task-specific agents and attack-abstraction layers. Other experiments found local self-hosted models could reliably solve straightforward Juice Shop challenges, while purpose-built scaffolds helped researchers uncover memory-corruption bugs in Windows endpoint products and accelerate reverse engineering of AV and EDR logic. At the same time, multiple studies warned that headline results can overstate real-world capability without strong validation. Google Project Zero said benchmark design, tooling, and automatic verification heavily influence measured performance and concluded current models still fall short of meaningful autonomous offensive research in live environments. Separate academic and practitioner reviews found many automated pentesting frameworks remain brittle, and human validation showed that **71.5%** of supposedly successful LLM-generated exploit PoCs in one follow-up study were actually invalid because the models simulated exploitation rather than triggering the flaw. Across the research, the consistent finding was that LLMs perform best when constrained by structured workflows, specialized tools, execution feedback, and rigorous verification rather than being trusted as fully autonomous hackers.
Jun 29, 2026
Large Language Models (LLMs) are increasingly being integrated into Security Operations Centers (SOCs) to enhance productivity and automate routine tasks such as transcribing meetings, extracting action items, and prioritizing emails. However, significant challenges remain before LLMs can reliably automate the majority of SOC functions, particularly those requiring high precision and consistent execution across vast, real-time data streams. Current LLMs struggle with real-time ingestion at scale, as they must process rapidly accumulating data from diverse sources like logs, EDR, cloud resources, and code without lag or data loss. Another major limitation is the need for large, durable context retention, enabling the system to interpret actions and sequences over extended periods using historical knowledge such as asset inventories and case histories. Low-latency, low-cost execution is also essential, as LLMs must filter, correlate, enrich, and reason over incoming data at enterprise scale without incurring prohibitive costs. Deterministic logic is required for repeatable, reliable results, but LLMs often lack this capability, making them less suitable for critical SOC triage tasks that demand context assembly, hypothesis generation, and nuanced business-risk judgment. Meanwhile, as attackers increasingly weaponize AI-driven automation, defenders must achieve a balance where machines can thwart most attacks at machine speed, or risk falling behind. In addition to operational challenges, LLMs introduce new security risks, such as system prompt leakage, which is now recognized as a top concern by the OWASP Top Ten Risks for LLMs. System prompt leakage occurs when sensitive information embedded in the prompts used to instruct AI models is inadvertently exposed, potentially revealing credentials, permissions, or internal logic to attackers. This exposure can enable adversaries to understand system guardrails, internal rules, or filtering criteria, making it easier to bypass defenses or launch targeted attacks. To mitigate this risk, experts recommend never embedding sensitive data such as passwords or internal logic within system prompts, ensuring that even if a prompt is leaked, it does not provide attackers with actionable insider knowledge. Examples of prompt leakage include revealing database storage details, internal decision-making processes, or request filtering limitations, all of which can be exploited by attackers. The convergence of these operational and security challenges underscores the need for robust, secure LLM deployment strategies in SOC environments. Organizations must address both the technical limitations of LLMs and the emerging risks associated with their use, particularly as threat actors become more sophisticated in exploiting AI-driven systems. Effective SOC automation will require not only technological advancements in LLM reliability and scalability but also rigorous security practices to prevent information leakage and maintain operational integrity. As the adoption of LLMs in cybersecurity accelerates, continuous evaluation of both their capabilities and vulnerabilities will be critical to maintaining a resilient defense posture.
Jun 29, 2026A new study found that large language models (LLMs) enhance accuracy in routine cybersecurity tasks but can lead to over-reliance and reduced independent thinking, especially in complex scenarios. High-resilience individuals benefit most from LLM support, while low-resilience users may see no improvement or even a decline in performance. Experts recommend human-in-the-loop processes, targeted training, and adaptive AI systems to mitigate risks and ensure safe, effective adoption.
Jun 29, 2026