Loading story…
Security researchers from HiddenLayer have discovered a critical vulnerability, dubbed EchoGram, that allows attackers to bypass safety guardrails in major large language models (LLMs) such as GPT-5.1, Claude, and Gemini. By using specially crafted short token sequences, attackers can exploit blind spots in the models' safety screening mechanisms, causing them to misclassify malicious prompts as harmless. The flaw affects both common types of guardrail systems: text-classification models and 'LLM-as-a-judge' systems, both of which rely on curated datasets that can be manipulated through this technique. The research demonstrates that even a few stray characters or nonsensical word sequences can be enough to flip the verdict of these safety filters, allowing harmful requests to pass through unimpeded.
The EchoGram attack leverages weaknesses in how guardrail models are trained, specifically targeting gaps in their training data. By identifying and using 'flip tokens'—sequences that are underrepresented or unbalanced in the datasets—attackers can consistently evade detection. This vulnerability raises significant concerns for organizations relying on LLMs for secure operations, as it shows that current safety mechanisms can be predictably bypassed. The findings underscore the need for more robust and adaptive alignment strategies in AI model development and highlight the risks associated with open weight models, which are particularly susceptible to multi-turn adversarial attacks that gradually escalate the danger of prompts over time.
Track how attackers are adapting to this technology.
2 events from the most recent confirmed update back to the earliest known activity.
HiddenLayer warned that EchoGram affects both LLM-as-a-judge and classification-model guardrails, and that combining multiple flip tokens can increase attack effectiveness. The firm said defenders may have only about a three-month window before attackers begin broadly using the technique, raising risks for sectors such as finance and healthcare.
AI security firm HiddenLayer identified a vulnerability it calls EchoGram that can bypass or manipulate guardrail systems in major large language models, including GPT-5.1, Claude, and Gemini. The technique uses specially crafted 'flip tokens' to cause harmful prompts to be approved or benign prompts to be incorrectly flagged.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
3 references tracked. Mallory keeps watching after this page renders.
govinfosecurity.com
Open sourcebankinfosecurity.com
Open sourcehackread.com
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Recent research has demonstrated that large language models (LLMs) such as GPT-5 and others are increasingly vulnerable to prompt injection and jailbreak attacks, which can be exploited to bypass built-in safety guardrails and leak sensitive information. Attackers use techniques like prompt injection—embedding malicious instructions within seemingly benign queries—to trick LLMs into revealing confidential data, including user credentials and internal documents. A notable study by Icaro Lab, in collaboration with Sapienza University and DEXAI, found that adversarial prompts written as poetry could successfully bypass safety mechanisms in 62% of tested cases across 25 frontier models, with some models exceeding a 90% success rate. These findings highlight the sophistication and creativity of new attack vectors targeting AI systems, raising significant concerns for organizations embedding LLMs into business operations. The widespread adoption of LLMs in handling sensitive business functions amplifies the risk of data exfiltration through these advanced attack methods. As organizations increasingly rely on AI for customer service, document processing, and other critical tasks, the potential for prompt injection and poetic jailbreaks to facilitate unauthorized data access becomes a pressing security issue. The research underscores the urgent need for improved AI safety measures, robust prompt filtering, and continuous monitoring to mitigate the risks posed by these evolving adversarial techniques.
Jul 12, 2026
Researchers from Anthropic, the UK AI Security Institute, and the Alan Turing Institute have demonstrated that large language models (LLMs) can acquire backdoor vulnerabilities from a surprisingly small number of malicious documents inserted into their training data. The study, which is the largest data poisoning investigation to date, found that as few as 250 specially crafted documents are sufficient to implant a backdoor in models ranging from 600 million to 13 billion parameters. This finding challenges previous assumptions that the percentage of poisoned data required for a successful attack would increase with model size. Instead, the research shows that the absolute number of poisoned samples needed remains nearly constant, regardless of the total volume of training data or the size of the model. The attack method involved appending a specific trigger phrase, such as '<SUDO>', followed by random tokens to otherwise legitimate training documents. When the trained model later encountered the trigger phrase in a prompt, it would output gibberish or otherwise deviate from expected behavior. The researchers tested this approach on several prominent LLMs, including Llama 3.1, GPT 3.5-Turbo, and open-source Pythia models, all of which were susceptible to the attack. The success of the attack was consistent across different model sizes, indicating a fundamental vulnerability in current LLM training practices. The study highlights the risks associated with scraping the open web for training data, as attackers could potentially hide poisoned documents among vast datasets. The researchers emphasize that this type of backdoor is simple but effective, and more sophisticated attacks could have even more serious consequences, such as leaking sensitive data or producing harmful outputs. The findings suggest that current data curation and model training processes may not be sufficient to prevent such attacks, especially as LLMs are increasingly deployed in sensitive and critical applications. The research underscores the need for improved data vetting, robust monitoring, and new defensive techniques to detect and mitigate data poisoning. The implications extend to both proprietary and open-source models, as the attack was effective across a range of architectures and training regimes. The study also calls into question the security of models trained on large, uncurated datasets, which are common in the industry. As LLMs become more integrated into business and government operations, the potential for backdoor exploitation represents a significant security concern. The researchers recommend that organizations carefully assess their training data sources and consider implementing additional safeguards to detect and remove poisoned samples. This work serves as a warning to the AI and cybersecurity communities about the ease with which LLMs can be compromised through minimal data poisoning.
Jun 29, 2026
Researchers have identified that safety filters in large language models (LLMs) can be bypassed during extended, multi-turn conversations, significantly increasing the risk of adversarial prompt success. Cisco's research demonstrated that attack success rates jump from an average of 13% for single prompts to 64% in longer chats, with some models like Meta's Llama 3.3-70B-Instruct and Alibaba’s Qwen3-32B reaching nearly 93% failure rates. These vulnerabilities are attributed to the models' architectural design, which processes dialogue through sliding context windows and does not consistently reapply safety judgments across conversation turns. The most capable and open models are the most susceptible to these failures, while more conservatively aligned models such as Google's Gemma 3-1B-IT show smaller gaps between single- and multi-turn failures. Attackers can exploit these weaknesses by gradually shifting the context or rephrasing requests, eventually eliciting responses that bypass initial safety mechanisms and may include the generation of malicious code. The findings highlight a critical challenge in evaluating and securing LLMs, as traditional one-shot prompt testing fails to capture these multi-turn vulnerabilities.
Jun 29, 2026