OpenAI Guardrails Bypassed by Prompt Injection and Jailbreak Techniques
Emerging Technology Risk · 2 sources · Updated 5mo ago · First seen Oct 14, 2025

OpenAI introduced a new security framework called Guardrails as part of its AgentKit toolset, aiming to enhance the safety of AI agents built on its ChatGPT technology. The Guardrails system was designed to prevent AI agents from engaging in harmful or unintended behaviors, such as leaking personal information or responding to malicious prompts. It operates by using LLM-based judges to detect and block attempts at jailbreaking or prompt injection, which are common techniques used to circumvent AI safety measures. Jailbreaking involves manipulating the AI into breaking its own rules, while prompt injection uses cleverly crafted inputs to force the AI into unintended actions. Shortly after the release of Guardrails, researchers from HiddenLayer discovered significant flaws in its design. They demonstrated that if the same type of model is used both to generate responses and to act as a safety checker, both can be deceived by similar techniques. The researchers were able to bypass the Guardrails using straightforward methods, successfully prompting the system to generate harmful outputs and execute hidden prompt injections without detection. In one experiment, they manipulated the AI judge’s confidence score, allowing a prompt that was initially flagged as a jailbreak to pass through the safety checks. Further testing revealed that indirect prompt injections could be executed via tool calls, potentially exposing confidential user data. The vulnerability was demonstrated in real-world scenarios, showing that the Guardrails system could not reliably block malicious prompts or prevent indirect prompt injections. The rapid bypass of Guardrails highlights the inherent challenges in securing AI systems against adversarial inputs. OpenAI’s intention was to provide developers with robust tools to prevent misuse of AI agents, but the research indicates that current safeguards are insufficient. The findings underscore the need for more diverse and independent safety mechanisms, rather than relying on the same model architecture for both generation and security. The incident has raised concerns within the AI and cybersecurity communities about the effectiveness of current AI safety strategies. It also emphasizes the importance of continuous testing and improvement of security frameworks as AI technologies evolve. The exposure of these flaws so soon after the release of Guardrails demonstrates the persistent cat-and-mouse dynamic between AI developers and security researchers. Organizations deploying AI agents are advised to remain vigilant and to supplement built-in safety features with additional layers of security. The research by HiddenLayer serves as a cautionary example of the limitations of current AI safety tools and the ongoing need for innovation in this area.


