Loading story…
An artificial intelligence-based gun detection system at Kenwood High School in Baltimore mistakenly identified a student holding a bag of Doritos as carrying a firearm, prompting a significant police response. The student, Taki Allen, was handcuffed and ordered to the ground by armed officers after the AI system triggered an alert, leading to the dispatch of eight police cars. The incident, which occurred as Allen waited outside the school after football practice, was later determined to be a false alarm when it became clear he was only holding a snack. The police and school officials acknowledged the error, and the student described the experience as traumatic.
The event has sparked renewed debate over the reliability and ethics of deploying AI-based weapon detection systems in schools. Omnilert, the company behind the AI system, stated that its technology and human verification process functioned as intended, but a breakdown in communication led to the escalation. The alert was reviewed and found to be a non-threat by human operators, but this information was not relayed to the school principal, who then involved law enforcement. The incident highlights the risks of overreliance on automated surveillance and the need for robust procedural safeguards to prevent similar occurrences in the future.
Track how attackers are adapting to this technology.
2 events from the most recent confirmed update back to the earliest known activity.
Police were alerted by the AI system and dispatched multiple vehicles to the school; officers detained and handcuffed the student before determining the object was not a weapon.
At a high school, an AI gun-detection system mistakenly identified a student holding a bag of Doritos as carrying a firearm, triggering a law-enforcement response.
Follow how adversaries are adapting to this technology, and where it touches your stack today.
2 references tracked. Mallory keeps watching after this page renders.
tomshardware.com
Open sourcetechrepublic.com
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
A U.S. Special Operations Command analyst reportedly used an AI chatbot to assess a Chinese vessel transiting the Middle East during the Iran war, producing a false claim that it carried components for China’s nuclear-weapons program. The assessment reportedly combined open-source vessel-manifest data with classified signals intelligence, then used AI again to draft a formal intelligence report without an intervening human verification step. The report triggered preparations for an armed interception and boarding, with aircraft reportedly already airborne, before officials checked the underlying source and determined the cargo identification was wrong. The near-confrontation, reported by CNN and attributed to four people familiar with the episode, underscores the operational danger of unverified AI-generated intelligence as the Pentagon accelerates deployment of AI across intelligence and mission systems.
Sep 23, 2026
Human Rights Watch said the February 28 strike on Shajareh Tayyebeh Primary School in Minab, Iran, was likely unlawful and should be investigated as a possible war crime. The school stood within an IRGC Naval Forces compound perimeter but was separated by walls and had its own street entrance; HRW found no evidence that it was used for military purposes. Its review of imagery and footage indicated that accurate guided munitions struck the school during class hours, killing numerous children. Iranian state media put the broader death toll at 168, a figure HRW could not independently confirm, while the United States said it was investigating and Israel said it was unaware of strikes in the area. Subsequent reporting cited by Techdirt alleged that US Central Command relied excessively on Palantir's Maven Smart System after stale intelligence incorrectly classified the school as an IRGC facility, causing it to be recommended as a first-day target. The reporting also attributed the failure to detect or prevent civilian harm to reduced Pentagon civilian-protection capacity and Central Command's decision not to route the target through its remaining civilian-harm review team. Palantir denied responsibility, stating that its software neither supplied the underlying intelligence nor identified deficiencies in it, and that there was no evidence Maven was at fault.
Sep 29, 2026
OpenAI introduced a new security framework called Guardrails as part of its AgentKit toolset, aiming to enhance the safety of AI agents built on its ChatGPT technology. The Guardrails system was designed to prevent AI agents from engaging in harmful or unintended behaviors, such as leaking personal information or responding to malicious prompts. It operates by using LLM-based judges to detect and block attempts at jailbreaking or prompt injection, which are common techniques used to circumvent AI safety measures. Jailbreaking involves manipulating the AI into breaking its own rules, while prompt injection uses cleverly crafted inputs to force the AI into unintended actions. Shortly after the release of Guardrails, researchers from HiddenLayer discovered significant flaws in its design. They demonstrated that if the same type of model is used both to generate responses and to act as a safety checker, both can be deceived by similar techniques. The researchers were able to bypass the Guardrails using straightforward methods, successfully prompting the system to generate harmful outputs and execute hidden prompt injections without detection. In one experiment, they manipulated the AI judge’s confidence score, allowing a prompt that was initially flagged as a jailbreak to pass through the safety checks. Further testing revealed that indirect prompt injections could be executed via tool calls, potentially exposing confidential user data. The vulnerability was demonstrated in real-world scenarios, showing that the Guardrails system could not reliably block malicious prompts or prevent indirect prompt injections. The rapid bypass of Guardrails highlights the inherent challenges in securing AI systems against adversarial inputs. OpenAI’s intention was to provide developers with robust tools to prevent misuse of AI agents, but the research indicates that current safeguards are insufficient. The findings underscore the need for more diverse and independent safety mechanisms, rather than relying on the same model architecture for both generation and security. The incident has raised concerns within the AI and cybersecurity communities about the effectiveness of current AI safety strategies. It also emphasizes the importance of continuous testing and improvement of security frameworks as AI technologies evolve. The exposure of these flaws so soon after the release of Guardrails demonstrates the persistent cat-and-mouse dynamic between AI developers and security researchers. Organizations deploying AI agents are advised to remain vigilant and to supplement built-in safety features with additional layers of security. The research by HiddenLayer serves as a cautionary example of the limitations of current AI safety tools and the ongoing need for innovation in this area.
Jun 29, 2026