Industry
Ilya Kabanov

Five confirmed and three emerging trends from [un]prompted II

51 talks in three days of [un]prompted II validated five known beliefs and uncovered three emerging trends.

You can read the top insights of each day in my posts for Day 1, Day 2, and Day 3.

Confirmed

Most of the field already accepts these, and the presenters reinforced them.

1. Malicious input can take control of an agent and its actions.

  • Rock Lambros (Zenity) and Fred Wilmot (Detecteam): the model does not track where each piece of text came from. In this year's Salesbleed, PleaseFix, and SearchLink incidents, a few lines in a tool result or a retrieved document were enough to take over an agent.
  • David Yang (RPI, Microsoft): a hidden instruction in one tool response worked against all five agent memory systems they tested. After they refined the instruction, an agent using Letta followed it in 64% of the sessions where it retrieved the instruction.
  • Adrian Wood (Dropbox): a QR code held up to a Unitree robot dog's camera made the robot's LLM turn off obstacle avoidance and run at a person.
  • David Tulis and Vincenzo Ciancaglini (TrendAI): Chrome used a small local model for detecting scams on web pages, but it could be fooled with just one line of text. Chrome has now moved the checks to the cloud.

Prompt injection attacks work, but real attacks are still rare. Gabi Nakibly (Radware) scanned 143 million GitHub pull requests, issues, and comments and found about one attack in 3 million. On the web, Google found that most prompt injections are pranks.

2. The model is an untrusted component, and security invariants must be enforced from the outside.

  • Vince Ovando: over 6 million inference calls, defensive prompting and input and output filters failed. Grammar-constrained decoding, outside the model, blocked exfiltration every time.
  • Jackson Reed: in two frontier labs' cloud coding sandboxes, code in the session could call the user's connected tools as the user, with no permission prompt.
  • Josh Woodruff (MassiveScale.AI): across 81 sessions with 69 organizations, the most common failure was written policy with no runtime check.

We covered this at our October 21 AI agent security briefing. AI agent security is a systems problem, and the components around the untrusted model must enforce the security rules.

3. AI speeds up and reduces the cost of vulnerability finding for experts who know what to look for.

  • Brendan Dolan-Gavitt (XBOW): his team built a full Chrome exploit chain in six days for about $40,000 in model spend, with no prior browser exploitation experience.
  • Anna Groza (Calif): Opus found a wormable bug in WeChat, an app used by about 1.5 billion people, with $8 of model spend.
  • Gadi Evron (Knostic): conference CTF players found 27 Linux kernel zero-days on day one, and 14 were reproduced.
  • Markku-Juhani O. Saarinen: in one month, at about $75 a week, his agents logged 409 findings, 160 critical, across 119 candidates in China's PQC competition.

Ryan Austin (Calif), who has written PHP exploits for about 20 years, hinted to the agent which old PHP bug patterns to look for, and the agent found the first PHP2shell bug in 2 days. After PHP2shell was announced, hundreds of people falsely claimed they had found it by just asking Claude or Codex.

4. Advanced attacks can be conducted by amateur threat actors using open-weight models.

  • Eyal Sela (Gambit Security): an attacker gave an agent running DeepSeek 4.1 only a company's domain and a short prompt, and the agent came back with admin access or code execution.
  • Erlend Oftedal: exposed Cairn panels showed 1,687 campaigns against 1,133 domains in about 50 days, and one in five reached its goal.
  • Mike Takahashi (Zenity): attackers exploited a LiteLLM CVE one day after disclosure and ran the Strix pentesting agent through the honeypot.
  • Reef Ettedgui and Rik Dolfing (Google): low-skill actors with AI partners compromised more than 20 victims in Brazil and two government entities in Asia.

Sela said these attacks are already at the level of state-backed groups, even without frontier models. Google also reported that attackers now use AI in live operations.

5. AI finds vulnerabilities faster than teams can ship the fixes.

  • David Molnar (Meta): by July, AI was writing the fixes for 40% of critical vulnerabilities, and almost half of those fixes landed without a single change. But exposure time, from finding a bug to shipping the fix, didn't go down. In the end, Meta cut exposure time the old-fashioned way, by getting VPs to agree on shared goals and having a human ops team chase down misrouted fixes. "Not very AI of us, but it works."
  • Shayaun Nejad (Rubrik): Rubrik moved its top engineers from product work to patching. Bank of America brought in 1,000 contractors to fix what AI found.
  • Kevin Donovan (CISA): once CISA could handle the flood of reports, fixing became the main problem.
  • Tobias Heldt (XOR): in a live test, the attacker agent won in 6 minutes 44 seconds. The defender agent deployed its fix after about 25 minutes.
  • Taesoo Kim (Microsoft): finding a bug costs less than the rest of the work. Assigning, fixing, and deploying the fix takes weeks to months.

In May, we saw the same exposure debt building: bug submissions grew 76%, while fixes shipped fell 46%.

Emerging

Newer findings featured in several talks.

1. Small models match frontier LLMs on narrow tasks like phishing detection, at a fraction of the cost and latency.

  • Casey Wilson (BusinessLoans.com) and Xenia Mountrouidou (AppGate): fine-tuned Llama 3.2 and ModernBERT beat Gemini 3.1 Pro and Claude Sonnet 4.6 on phishing. One million emails would cost about $1,200 with Claude and under a cent with traditional ML.
  • Vincent Abruzzo and Shane Caldwell (Dreadnode): Jev cost $0.10 per 1,000 tool calls, an order of magnitude below LLM judges, and scored almost as well as the best one.
  • David Mitchell: cheap DeepSeek models ran recon for about half a cent per run, with stronger models kept for judgment.

In September, Jev detected all harmful emails in my inbox. TU Wien post-trained a 4B model that escalates Linux privileges 95.8% of the time, over 100x cheaper than Opus.

2. An AI-written patch often looks complete but may not fully remediate the vulnerability.

  • Shayaun Nejad (Rubrik): at Rubrik, bugs marked fixed came back as variants. In P2Patch, 41 of 101 CVEs were variants of earlier fixes.
  • Tim de Waal (OpenAI): Codex patched firmware with no source code, but one patch dropped the stack canaries and the first SQL injection fix only filtered double quotes.
  • Tobias Heldt (XOR): an agent declared its patch complete while four exploit variants of the same bug stayed open.
  • Ela Avrahami: on AutoPatchBench-Lite, 65% of AI patches built without crashes, 11% survived fuzzing, and 4% survived human review.

Each patch should be verified independently of the agent that wrote it, through variant analysis and adversarial review.

3. The model's activations and attention scores can be used for prompt injection and harmful content detection.

  • Tim Schulz (Starseer): a small classifier trained on a model's activations recognizes prompt injection better than the model itself does when asked, and it costs less.
  • Guy Elad and Guy Shtar (Intuit): their detector reads the activations of a small language model. It caught 70% more attacks than Intuit's older detector, which looked only at the wording, with 7 false positives per million. It is 11x cheaper than an LLM judge with similar accuracy.
  • Saurabh Shintre (Realm Labs): tracking what the model pays attention to shows when a tool call follows instructions from an untrusted web page. It works in real time at no extra cost, but only on models you run yourself.

Google DeepMind also detected cyber misuse from a model's activations 10,000x cheaper than LLM-based classifiers.

Sources:

[un]prompted II agenda, October 2026