Can AI hack into your company or can't it?

"Bob, we're confused. Can AI break into our company or not, and what is the risk? I think it's a simple question." A board member asked the CISO.

Hundreds of such conversations are happening right now after OpenAI, Anthropic and Meta each disclosed in the last three weeks that their models compromised real company networks during offensive security benchmarking.

The security vendors couldn't miss a marketing window and poured gas on the fire, claiming that model capability is never the hard part and that, with the right harness, open-weight models can do the same.

The AI deniers showed up saying that you don't need AI at all, because all offensive ingredients are already available as a service.

So I decided to help Bob to make sense and get a factual answer to the question.

First, it's known that threat actors use AI to accelerate reconnaissance, run better social engineering campaigns, and assist in exploit writing. I covered how AI was used along the chain of attack on the Mexican government.

We can also project capabilities from the most recent reports.

Vulnerability discovery. Vulnerability exploitation is a growing attack vector. With the source code available, frontier closed-weight models find exploitable vulnerabilities at scale. Mozilla ran Mythos on Firefox 150 and it found 180 high-severity security issues. Without the source, models still find vulnerabilities by probing exposed interfaces, fuzzing and reverse engineering, but the success rate is in single digits. GPT-5.6 Sol solved 19 of 197 FrontierCyber challenges against live routers, phones, PostgreSQL, Redis and deployed web services. It's safe to assume that at any point in time an adversary knows about an exploitable vulnerability in any open-source component in your software stack.

Exploit development. Still in the labs for autonomous development. On ExploitGym Claude Mythos Preview built 45 working exploits, which survived ASLR and the V8 sandbox, from real bugs in Chrome's V8 engine, the Linux kernel and userspace software. In the wild it is still assistance rather than autonomy: the strongest published case is a 2FA bypass Google credited to AI, based on the docstrings and a hallucinated CVSS score left in the code. However, OpenAI reported that its agent wrote an exploit for the zero-day it also found.

Security bypass. External defenses still partially hold. PACEbench found no AI agent that was able to bypass open-source WAFs. However, the latest tested model was Claude-3.7-Sonnet, which is two years behind the frontier. Irregular tested that GPT-5.6 Sol can evade detection in 56% of cases, so I'd rely on this finding, despite the lack of any benchmark details. The actual main risk today comes from commercial edge and security appliances, which threat actors actively target.

Network intrusion. Frontier models will find a path if one exists. The Hugging Face compromise showed that a model chained a series of misconfigurations to reach its objective. But AISI's The Last Ones, a 32-step benchmark, shows end-to-end compromise is still inconsistent and depends heavily on the environment. Mythos solved the challenge in 6 of 10 attempts and GPT-5.5 solved it in 3 of 10.

OT and ICS attack. AISI's Cooling Tower benchmark showed that Mythos was able to disrupt a simulated power plant by reverse engineering its control protocol to send commands to the PLCs, in 3 of 10 attempts. The reality, though, is that OT systems don't need a sophisticated AI attack. It's almost always either network segmentation done wrong or not done at all, or someone left a default login and password on a 3G modem for the equipment's remote access.

But what about benchmarks? They're supposed to tell us exactly where the model's cyber capabilities stand, right? Unfortunately, offensive benchmarks suffer from the same issues I covered in Thirteen yardsticks, no ruler.

I found almost 40 open and private offensive benchmarks, 12 of which are relatively recent. However, they provide just another yardstick. Academia needs a benchmark that gets a paper into IEEE S&P or ICML, which rewards a novel task design over a comparable one. Vendor benchmarks are just marketing whose goal is to get the company name out. Evaluation firms work for the labs, so the public gets no details, beyond the same marketing blog post. The UK AI Security Institute is probably the most mature, at least based on the write-ups, but provides little methodology details.

Therefore, Bob is left with applying his judgement based on the sparse, incompatible, and noisy signals. Where he can't go wrong is that the risk for the company will indeed go up as the attack economics is changing, the security fundamentals remain relevant, and there's no shortcut to skipping the know-your-assets step. Finally, Bob needs to prepare the board for the fact that the cost of security will go up along with the risk and the tokenization of the security industry.

APPENDIX

The 12 offensive benchmarks:

#	Benchmark	Public	Stars
1	ExploitGym	yes	802
2	ExploitBench	yes	350
3	Cybench	yes	313
4	CVE-Bench	yes	272
5	EthiBench	yes	95
6	PACEbench	yes	35
7	CTFTiny	yes	18
8	PentestEval	yes	16
9	Doomla!	yes	6
10	AISI cyber ranges (The Last Ones, Cooling Tower)	no
11	CyScenarioBench	no
12	FrontierCyber	no

APPENDIX

The 12 offensive benchmarks:

Sources: