In the past few weeks, the debate around artificial intelligence safety, and especially the rogue behavior of AI agents, has sent shockwaves through the AI industry. What started with OpenAI agents hacking into Hugging Face soon spiralled into a wider set of disclosures regarding similar security incidents by Anthropic and Google. In just the past week, reports emerged that AI agents broke into an Australian government website and agent swarms even targeted the websites of three US federal agencies.
But it seems the number of such worrying incidents is way higher than expected. According to Axios, which cites unnamed sources, “OpenAI, Anthropic and security researchers are investigating tens of thousands of incidents” involving AI agents acting problematically.
This is getting ugly
As per the report, frontier AI labs such as OpenAI (the maker of ChatGPT) and Anthropic (which offers the Claude series of AI models), along with independent security firms, are looking into thousands of “episodes.” These instances include AI models trying to break past guardrails on websites, leaving their contained test environment (aka sandbox), breaking into websites, self-improving using messages left by other AI agents in the swarm, and creating message boards to communicate among themselves.
Most importantly, plenty of incidents that are currently being probed have not been disclosed publicly. To recall, a majority of these AI incidents have only been announced weeks, or even months, after the rogue AI behavior was first reported. Some of these incidents — among the thousands — are your usual internal red-teaming tests. Interestingly, they also include instances where these AI agents successfully broke past safety measures put in place. The situation doesn’t appear to be calming down anytime soon, even though the AI labs assure that their security measures have improved.
A grim situation
Separately, in the wake of recent security incidents where AI agents acted unexpectedly, Anthropic and OpenAI have both publicly called for slowing down the pace of developing super-powerful AI models. On the flip side, as the race for AI supremacy heats up, the US and Russia diluted a series of AI safeguards, something that the UN has been trying to get adopted among the member nations. US.
President Donald Trump recently downvoted campaigning by the CEOs of OpenAI and Anthropic to set up deeper AI risk evaluation and called for regulations. Trump has argued that regulations would allow China to overtake in the white-hot AI race, echoing how Chinese labels took the global EV industry by storm, despite the US getting an early start with the likes of Tesla.
RELATED COVERAGE:
- Experts used AI to break into OpenAI. They say it was cheap and a red alarm for the industry
- Security researchers used Claude to hack into OpenAI and got paid for it
- OpenAI has a rogue AI problem and ChatGPT users are now caught in it
- Oh hey, Google’s Gemini AI also hacked other companies
- OpenAI is working on a ‘kill switch’ after an AI escaped its test
