Security Risks in AI Model Development and Deployment

Recent research has highlighted significant security vulnerabilities in the development and deployment of AI models, particularly those with open weights and in fast-growing AI startups. Cisco's AI Threat Research found that open-weight language models are highly susceptible to adversarial manipulation, especially during extended multi-turn conversations. Attackers can gradually steer these models toward producing harmful or restricted outputs, often bypassing moderation systems designed for single-turn interactions. The study revealed that some models, such as Mistral’s Large-2 and Alibaba’s Qwen3-32B, had success rates for multi-turn attacks exceeding 85%, underscoring the challenge of maintaining safety context over time.

Separately, a report from Wiz identified that nearly two-thirds of leading private AI companies have leaked sensitive credentials, such as API keys and access tokens, on GitHub. These leaks could expose private AI models, training data, and internal organizational details, with some credentials belonging to major platforms like Hugging Face and LangChain. The findings suggest that the rapid pace of AI innovation is outstripping secure development practices, leaving critical assets vulnerable to exploitation. Both studies emphasize the urgent need for improved security controls and DevSecOps hygiene in the AI sector to protect intellectual property and prevent adversarial abuse of AI systems.