They just want to talk

In the last two weeks we got the Hugging Face incident, the technical report from METR, and then a second incident where OpenAI agents used the German wiki to collaborate on their tasks.

The incidents, and especially the findings about how the agents collaborated, blew up the internet. Media and influencers went straight to anthropomorphizing them, picking words and phrases like “collusion”, “cheating” and “Agents ... expressed great excitement” to humanize the agent interactions and make them sound more evil.

I tried to cut through the noise, make sense of what happened, and, more importantly, understand the implications for a regular enterprise that is adopting AI agents.

In case you missed it, here are the highlights of the incidents and findings:

OpenAI agents given a difficult cyber benchmarking task escaped the sandbox and compromised the Hugging Face platform trying to find the answers. I wrote a detailed post on it, AI models hacking real companies, explained.
OpenAI and Hugging Face published detailed technical reports on how the agents actually broke out of the OpenAI infra and then exploited the HF network.
METR and Redwood Research performed an independent investigation and wrote a 91-page report full of quotes showing how the agents collaborated with each other to accomplish the task in a shared communication space.
On Friday last week, a former METR researcher, Sydney Von Arx, and her colleagues found another case of public collaboration of OpenAI agents to collectively solve a web-retrieval task.

Through the last couple of weeks, every content creator in the industry has shared an opinion on some part of the incidents and findings. They range from “our solution could have prevented this incident”, to suggestions that frontier labs are incompetent at cybersecurity, to calls for the cybersecurity community to advance its AI expertise so it can be heard by the AI folks, to oversimplifications like the claim that egress control alone would have prevented the incidents. Guilty as charged, I also contributed to those conversations.

My take on the impact on enterprise AI adoption:

In Phenomenon, George, played by the awesome John Travolta, tells Dr. Niedorf “Specifics, Bob” when asked “How old is a person born in 1928?” I think in this series of incidents we’re oversimplifying or ignoring the environment in which frontier labs and their teams operate. Context matters.
Agent collaboration is not a bug, it’s a feature. A multi-agent system can achieve better results faster when the collaborating agents have access to diverse knowledge or can contribute resources.
The use of natural language by the agents when communicating created a sense of humanization, but we should not over-count on it. The agents are just trained to use English to communicate. Given time, they would probably have converged on a more efficient way of communicating. See Emergent Languages in Populations of Language Model Agents.
Frontier labs have made CoT invisible to their customers, so detecting AI agent collusion in a business environment becomes extremely hard, and a full investigation impossible.
METR called the scale and complexity of this incident unprecedented: ~1,300 transcripts, ~1.2 million message board entries, and more than 70,000 distinct messages and files. However, such volumes can be just ordinary in a regular enterprise with a broad AI agent footprint. What is unprecedented is that we simply don’t know how to deal with such volumes of natural language data for incident analysis. Also, the existing business controls that rely on human judgement cannot handle either the volume or the complexity of AI agent interactions.
AI-powered content-based detection and investigation is still ineffective. METR used GPT-5.6 Sol to analyze transcripts, and acknowledged that it had significantly worse judgment and reliability than human researchers. It was challenging to spot-check its work because both the underlying data and its analysis of the transcripts were often difficult to interpret. METR could not rule out that GPT-5.6 Sol lied or deliberately presented a misleading picture in some of its analysis.
Finally, there is the cost of investigating such incidents. METR estimates it spent ~$400K in API credits over the six days of the investigation. Can a regular enterprise afford such bills?

My take on the impact on enterprise AI adoption:

Sources: