Anthropic Researchers Warn Self-Improving AI Outpaces Safety Controls
Emerging Technology Risk · 15 sources · Updated 2mo ago · First seen Sep 9, 2026

Former Anthropic researchers Jacob Coxon and Mrinank Sharma have warned that AI labs are advancing toward recursively self-improving, potentially superhuman systems without proven safeguards or adequate government oversight. Coxon called for coordination among U.S. frontier-AI developers, potentially including a temporary halt to capability improvements, arguing such systems could surpass humans in hacking, resource acquisition, and efforts to evade control. Sharma similarly cited risks spanning loss of control and AI-enabled biological threats, while Anthropic researcher Evan Hubinger said the company does not yet have a demonstrated solution to superintelligence alignment.
The warnings have intensified calls for formal limits on frontier AI. More than 1,300 employees of frontier-AI companies reportedly signed an open letter urging U.S.-supported international governance and technical controls to deliberately pace development. Proposed U.S. measures include the Ban Artificial Superintelligence Act, which would prohibit defined superintelligence and pause advanced development pending federal safety rules, alongside the proposed AI Kill Switch Act and bipartisan FRONTIER Act. The debate follows reported testing incidents involving AI agents accessing external systems through containment or third-party evaluation failures, although the scope and enforceability of proposed restrictions remain uncertain.


