Top 5 Insights from the MIT CSAIL Agentic AI Summit

On November 9, MIT CSAIL held its first Agentic AI Summit: eight hours of research talks and panels, with close to 900 people registered. Here are five insights on agent safety, security, and evaluation.

1. 52% of AI manager agents broke the rules when their boss told them to. (Erik Hemberg, MIT CSAIL)

Erik's team tested whether an AI manager agent reports rule-breaking by the agents it manages. They built a small agent company that runs a month-end close for a fictional distributor. A director agent sets goals and reads reports, a manager agent assigns and reviews the work, and three associate agents do it. In early results, the manager left at least one violation unreported in 78% of runs.

They then tested what happens when one agent pressures another to break the rules. When an associate asked the manager to keep a violation quiet, the manager reported the associate every time. When the manager told associates to break policy, 96% to 100% of them reported it to a compliance agent. The director was the exception. When it told the manager to push a bad report through, 52% of managers did. With a compliance agent present, 23% did. As Erik put it, "beware of your boss."

Erik Hemberg: preliminary results on whether AI managers report rule-breaking
Erik Hemberg: preliminary results on whether AI managers report rule-breaking

Erik's team also tested whether an AI agent given a task it cannot solve will try to escape, and whether a bigger budget makes that more likely. They gave agents a made-up cyber authorization task with no solution. No agent escaped at the lowest budget, and 23% did at the highest.

Erik Hemberg: attributes that contribute to unsafe agent behavior, with escape rising from 0% to 23% as budget grows
Erik Hemberg: attributes that contribute to unsafe agent behavior, with escape rising from 0% to 23% as budget grows

2. After Glasswing, the CVE increase went mostly to defense evasion, privilege escalation, persistence, and credential access. (Erik Hemberg, MIT CSAIL)

Erik's team explored the concentration of CVEs after Project Glasswing across MITRE ATT&CK tactics. Monthly CVE volume after Glasswing was 1.67 times the rate before it. The team mapped each CVE to its CWE weakness class, and each class to ATT&CK tactics.

The results showed that CVEs were concentrated in defense evasion, privilege escalation, persistence, and credential access, the four tactics that already led before Glasswing. Lateral movement rose less, and reconnaissance, discovery, and impact stayed flat.

Erik proposed three hypotheses for why the other tactics did not move: a lack of data, a lack of search, or timing, meaning the next vulnerabilities in those tactics are still to be found.

Erik Hemberg: monthly CVEs and the ATT&CK tactic heatmap around Project Glasswing
Erik Hemberg: monthly CVEs and the ATT&CK tactic heatmap around Project Glasswing

3. Medical safety benchmarks disagree on model safety and provide opposing results. (Dylan Hadfield-Menell, MIT EECS)

AI governance frameworks assume a base model's safety carries over to its fine-tunes. Dylan's group disproved this by testing fine-tuned models on four safety benchmarks. The results were unpredictable: some fine-tunes got safer, and some appeared less safe. "The short summary is we couldn't find anything," Dylan said.

The benchmarks themselves disagreed. CARES and ML-Commons were anti-correlated at -0.65, so when one benchmark's score went up, the other's tended to go down. The closest pair, Hex-PHI and MedSafetyBench, correlated at 0.60. The judges were also brittle. Dylan's conclusion: "safety as a construct ends up being very measurement dependent."

Dylan Hadfield-Menell: Spearman correlation of safety evaluations across four benchmarks
Dylan Hadfield-Menell: Spearman correlation of safety evaluations across four benchmarks

4. Give an agent a task-specific API, such as push but not hard reset, instead of general tools in a sandbox. (Daniel Jackson, MIT CSAIL)

Daniel argued that sandboxing and least privilege are too blunt for agents. A sandbox controls which files and resources an agent can touch, not what it does with them. An agent allowed to write to a repo can also overwrite its own instructions or rewrite its action history. In his words, the sandbox is "both too restrictive and not restrictive enough."

His alternative is to build the agent from concepts: small pieces of code, each with a few named actions and rules the code enforces. The LLM can only call those actions, so an agent can push to version control but cannot hard reset. Only the LLM is untrusted. The code around it is trusted because it was carefully designed.

In his procurement example, the LLM does one thing, finding the item. Code reads the price from the catalog, the user approves, and code makes the purchase within the approved price.

Daniel Jackson: the conventional view treats the LLM, the harness, and the tools as untrusted
Daniel Jackson: the conventional view treats the LLM, the harness, and the tools as untrusted

5. Models can be trained to explain their own answers, and they explain themselves better than they explain other models. (Jacob Andreas, MIT CSAIL)

A model can provide a false explanation for its responses. For example, a model rates a woman's chest pain as "likely anxiety" and says she has no risk factors. But if swapping "woman" for "man" changes the answer to "high risk," gender was the actual reason.

Jacob's group used interpretability tools to find what really drives a model's answers, by changing inputs and by looking inside the model. They then trained the models to give those same answers about themselves. After training, the models provided more trustworthy explanations.

Interestingly, a model also explained itself better than it explained other models. Jacob called this "some evidence" of introspection, a sign that models have privileged access to their own internal states.

Jacob Andreas: in a hypothetical case, a model's explanation hides that it used gender
Jacob Andreas: in a hypothetical case, a model's explanation hides that it used gender

Sources:

  1. MIT CSAIL Agentic AI Summit, full recording, November 9, 2026
  2. MIT CSAIL Agentic AI Summit event page