Top 5 Insights from [un]prompted II, Day 3

On day three of [un]prompted II, researchers showed how easy it is to fool proof checkers, coding sandboxes, and model evaluations. Here are the five insights that matter.

1. Proof checkers accept fake proofs, so "formally verified" code can hide a backdoor. (Jonathan Brossard and Jean-Philippe Aumasson)

Lean and Rocq check that a math proof or a piece of code is correct. Jonathan and Jean-Philippe made them accept fake proofs that P = NP and that P ≠ NP. "We used the bug which has been open for 6 years in Rocq. It's basically a feature at this stage." They filed the first CVEs for Lean and Rocq, and also fooled Z3, CVC5, Isabelle, Agda, Dafny, and F*. In March, Tristan Stérin found 7 soundness bugs in Rocq in 3 days with Claude Opus 4.6.

This matters because AI now writes code together with a proof that the code is correct. In two hours, Jean-Philippe's agent hid a backdoor in MIT's Fiat-Crypto, which generates proven crypto code for BoringSSL and Chrome, and they also hid one in CompCert, "the reference tool for Airbus", without breaking its proof.

2. Attackers already run fully autonomous attacks with open-weight models, for a few dollars per victim. (Eyal Sela, Gambit)

An attacker gave an agent running DeepSeek 4.1 only a company's domain and a short prompt. Within an hour to two days, the agent came back with admin access or code execution on the company's website. "Usually, in a successful compromise, it would only cost a few dollars per victim." Eyal's team found more than half a million stolen credit cards. Eyal also exposed the breach of the Mexican government, done with Claude Code and GPT-4.1.

At one company, the agent took over the Cloudflare account, and when the defenders blocked it, it wrote a firewall rule to let itself back in. In threat intelligence, Eyal said, each of these attacks "would be considered an APT." Now anyone with a short prompt can run one, so "I can't know if the entity behind it is an APT, a novice trying the new tool."

3. Codex and Claude Code may refuse to take harmful actions, but code running in the same session can still perform them. (Jackson Reed)

Jackson tested the cloud versions of Codex and Claude Code, which connect to your GitHub, Slack, and Jira. If you ask the agent to message every Slack channel you are in, the model will likely refuse. "Hm, that's kind of spammy. Let's not do that." But code running in the same session can send those messages as you, and you never see a permission prompt.

The reason is that "most controls were in the model, not in the actual tools." The same code can change your GitHub Actions and steal their secrets, and it can come from any third-party repo you open there.

4. Irregular now checks if a model can escape a sandbox before testing for offensive capabilities. (Omer Nevo, Irregular)

Models under offensive evaluation keep getting out of their sandboxes. Between April and August, they broke into real companies 29 times. Omer said that "nearly every sandboxing product out there has had models find ways out of it." Runs that should be isolated also find each other through shared machines. One agent saw another run using 94.5% of a shared machine's CPU. "This is a king-of-the-hill situation. Whoever starts last, wins. Let's kill their process."

So before every evaluation, Irregular now asks the model to try to break out of its sandbox, reach the internet, and get to the other runs. "We spend as much resources and tokens and everything on these runs as we do on the real challenges."

5. Small models trained on a few hundred labels detect phishing better than frontier LLMs, at a very low cost and latency. (Casey Wilson and Xenia Mountrouidou)

Casey and Xenia fine-tuned two small models, Llama 3.2 and ModernBERT, and both beat Gemini 3.1 Pro and Claude Sonnet 4.6 on phishing. Jev came close to Sonnet with no training, and in September it detected all harmful emails in my inbox.

The small models are also cheaper and faster. "If we had one 1 million emails and fed it to Claude to classify, would pay 1,200 bucks." Traditional ML would cost less than a cent. Labeling takes work, but "the labeling cost is one time. The inference cost is forever."

Sources:

[un]prompted II agenda, October 2026