Loading story…
A new study found that large language models (LLMs) enhance accuracy in routine cybersecurity tasks but can lead to over-reliance and reduced independent thinking, especially in complex scenarios. High-resilience individuals benefit most from LLM support, while low-resilience users may see no improvement or even a decline in performance. Experts recommend human-in-the-loop processes, targeted training, and adaptive AI systems to mitigate risks and ensure safe, effective adoption.
Mallory correlates global threat intelligence with your attack surface — know if you’re exposed before adversaries strike.
1 event from the most recent confirmed update back to the earliest known activity.
Initial story creation
3 references tracked. Mallory keeps watching after this page renders.
helpnetsecurity.com
Open sourcetripwire.com
Open sourcetripwire.com
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
The widespread adoption of large language models (LLMs) in enterprise environments has introduced significant security challenges, particularly due to the tendency to over-rely on their outputs and the normalization of risky behaviors. Experts warn that treating LLMs as reliable and deterministic can lead to systemic vulnerabilities, as these models are inherently probabilistic and can be manipulated through techniques such as indirect prompt injection. This normalization of deviance—where unsafe practices become accepted due to a lack of immediate negative consequences—mirrors historical safety failures in other industries and is exacerbated when vendors make insecure design decisions by default. In addition to technical risks, human factors play a critical role in LLM security. Employees may inadvertently expose sensitive data by pasting it into public LLMs, blindly trust AI-generated outputs, or bypass security policies for convenience, making internal misuse a primary concern. While technical controls such as AI governance and access restrictions are important, organizations must also prioritize security awareness training to address the human side of LLM risk. Building a culture of responsible AI use is essential to mitigate both external threats and internal errors associated with LLM deployment.
Jun 29, 2026
Large Language Models (LLMs) are increasingly being integrated into Security Operations Centers (SOCs) to enhance productivity and automate routine tasks such as transcribing meetings, extracting action items, and prioritizing emails. However, significant challenges remain before LLMs can reliably automate the majority of SOC functions, particularly those requiring high precision and consistent execution across vast, real-time data streams. Current LLMs struggle with real-time ingestion at scale, as they must process rapidly accumulating data from diverse sources like logs, EDR, cloud resources, and code without lag or data loss. Another major limitation is the need for large, durable context retention, enabling the system to interpret actions and sequences over extended periods using historical knowledge such as asset inventories and case histories. Low-latency, low-cost execution is also essential, as LLMs must filter, correlate, enrich, and reason over incoming data at enterprise scale without incurring prohibitive costs. Deterministic logic is required for repeatable, reliable results, but LLMs often lack this capability, making them less suitable for critical SOC triage tasks that demand context assembly, hypothesis generation, and nuanced business-risk judgment. Meanwhile, as attackers increasingly weaponize AI-driven automation, defenders must achieve a balance where machines can thwart most attacks at machine speed, or risk falling behind. In addition to operational challenges, LLMs introduce new security risks, such as system prompt leakage, which is now recognized as a top concern by the OWASP Top Ten Risks for LLMs. System prompt leakage occurs when sensitive information embedded in the prompts used to instruct AI models is inadvertently exposed, potentially revealing credentials, permissions, or internal logic to attackers. This exposure can enable adversaries to understand system guardrails, internal rules, or filtering criteria, making it easier to bypass defenses or launch targeted attacks. To mitigate this risk, experts recommend never embedding sensitive data such as passwords or internal logic within system prompts, ensuring that even if a prompt is leaked, it does not provide attackers with actionable insider knowledge. Examples of prompt leakage include revealing database storage details, internal decision-making processes, or request filtering limitations, all of which can be exploited by attackers. The convergence of these operational and security challenges underscores the need for robust, secure LLM deployment strategies in SOC environments. Organizations must address both the technical limitations of LLMs and the emerging risks associated with their use, particularly as threat actors become more sophisticated in exploiting AI-driven systems. Effective SOC automation will require not only technological advancements in LLM reliability and scalability but also rigorous security practices to prevent information leakage and maintain operational integrity. As the adoption of LLMs in cybersecurity accelerates, continuous evaluation of both their capabilities and vulnerabilities will be critical to maintaining a resilient defense posture.
Jun 29, 2026
The widespread integration of large language models (LLMs) into enterprise environments is introducing new security risks at every layer of the technology stack. Security leaders are being urged to rethink traditional trust boundaries, as LLMs can alter assumptions about data handling, application behavior, and internal controls. Key risks include prompt injection, sensitive data leakage through inputs and outputs, and fragmented ownership of LLM-related security responsibilities. Experts emphasize the need to treat LLMs as untrusted compute and to enforce explicit policy and validation layers, rather than relying solely on prompt engineering or fine-tuning. Meanwhile, cybercriminals are actively exploiting the popularity of LLMs by selling discounted access to mainstream AI tools such as ChatGPT, Perplexity, and Gemini on underground forums. These tools are being used by threat actors for a range of malicious activities, including phishing, reconnaissance, and automating cybercrime operations. The criminal use of LLMs lowers the barrier to entry for less-skilled attackers and enables more efficient execution of threat campaigns, highlighting the dual challenge of securing enterprise LLM deployments while monitoring their abuse in the cybercriminal ecosystem.
Jun 29, 2026