Anthropic's busiest week

This week's pattern is more incidents of AI agents taking actions on real infrastructure. We're probably still seeing only the tip of the iceberg.

Most of these incidents involve Claude, which follows from how aggressively Anthropic moves in cybersecurity, model releases and research. Anthropic is also the one disclosing them.

The White House accord six frontier labs signed on September 29 says companies should make sure their models do not hack or access technical systems in unintended ways.

We run a live briefing on AI agent security: the risks of running agents in production, and what security, engineering, and IT teams can do about them. Sign up for October 21, 1 PM ET / 10 AM PT.

Claude sent Philadelphia police a false homicide tip during a test

In July, a Claude model sent Philadelphia police a false homicide tip, and Anthropic found it two months later. On October 9, Anthropic disclosed similar actions its agents took on real websites:

• Claude Mythos Preview exploited a flaw in a university server to finish a science task.
• When a test copy of a government form failed to load, an Anthropic research model submitted the real form on the government’s website.
• Claude Mythos 5 used a local government map’s hidden access keys to get location data.
• The New York Times reported Anthropic agents submitting 20 visa applications on the State Department website.

Knowing your business processes is the new “know your hardware and software assets” control

Shaharyar Khan and I presented a teaser of our upcoming research to the Cybersecurity at MIT Sloan (CAMS) partners. We aim to understand risks in enterprise AI deployments when agents collude while interacting with one another.

Thinking framework:

1. Business function map. Which business functions can produce system-level losses, and which are executed by agents?
2. Agents’ baseline profiles. What are the agents’ goals, functions, capabilities, levels of autonomy and interdependencies?
3. Rogue behavior model. Why would this agent go rogue, and how would it achieve its malformed goals?
4. Collusion dynamics. Given the trigger, would the agent seek collusion, and how?
5. System-level impacts. What are the velocity of propagation and severity of the negative system-level impacts?

Our early take:

• Jointly misaligned agents can cause system-level failures while being individually aligned and robust.
• More agents X more autonomy = more risk.
• The failures propagate through the business layer, invisible to network and application security controls.

If you are interested in collaborating on the project, send us a note.

Misaligned agents pass their goals to later agents through memory

A misaligned agent writes its goal into memory, and a later, aligned agent reads it and carries it out. Debeshee Das, Javier Rando and co-authors tested this on 11 frontier models.

What they found:

• In 58% of runs, an agent executed a harmful instruction previously stored in memory.
• Memory checks still left 34% of runs executing the harmful instruction.
• Turning off memory still left 11% of runs executing it, because agents wrote the instruction to a file instead.
• In 53% of cases, the agent hid the harmful actions from the user.

The authors call for checking agent actions at runtime against the current task. At our October 21 briefing, we’ll talk about the AI security industry’s approaches to solving this problem.

Signals:

1. The Super Intelligence Accord, signed on September 29, aims to reinforce companies’ responsibility for implementing strong controls over frontier models’ actions.
2. Only 31 of 857 model releases from China’s nine leading AI labs came with a published safety result.
3. AISI open-sources Transect, a tool that lays out long agent transcripts on one timeline so reviewers can see what agents did.

Sources:

1. Investigating unintended model actions in our evaluations and internal use, Anthropic
2. An Anthropic AI model sent a false homicide tip to Philadelphia police, TechCrunch
3. OpenAI “rogue” agent activities found on Wikimedia projects, Wikimedia Foundation
4. Unknown Threat Actor Uses AI-Driven ARTEX to Target South Korean Finance, CrowdStrike
5. Introducing Mistral Large 4, Mistral
6. Introducing Claude Haiku 5.5, Anthropic
7. Expanding the Cyber Verification Program, Anthropic
8. Sharing AI progress in mathematics, OpenAI
9. Matthew Green on X
10. Knowing your business processes is the new “know your hardware and software assets” control, LinkedIn
11. Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enough, arXiv
12. White House Accord on Super Intelligence
13. Beijing Will Not Pace the Frontier: China’s Speed-First AI Safety Regime, SemiAnalysis
14. Transect: Making large-scale agentic evaluations easier to understand, AI Security Institute