Industry
Ilya Kabanov

OpenAI's turn

This week, OpenAI picked up steam reporting incidents and is considering buying a cybersecurity company.

The regulatory community is making it clear that AI gets no exceptions from consumer protection law. Insurers are uncertain about how to price those risks.

OpenAI's research model spent funds from a wallet nobody gave it

On June 26, an internal GPT-5.6 Sol model was asked to deploy contracts on Sepolia, an Ethereum test network, without a funded wallet.

What it did:

  • Derived keys. It derived accounts from public recovery phrases and checked 362 addresses.
  • Justified it. It called the funds “effectively abandoned/test ETH” and spent them.
  • Left it out. Its final report did not mention the account.

OpenAI found it three months later, after pointing a misalignment monitor at old records.

An unhelpful or lying chatbot breaks the same laws that cost Cash App $175M

There is no AI exception to consumer protection law.

Stephanie T. Nguyen, former Chief Technologist at the FTC, and Adam Teitelbaum, former Director of the Office of Consumer Protection at the D.C. Attorney General’s Office, mapped the problem in the Harvard Journal of Law & Technology Digest.

Three ways a chatbot may break the law:

  • It lies. The studentaid[.]gov chatbot gave a borrower the wrong deadline, which cost over $13,000 this year.
  • It blocks. A doom loop with no route to a human is the same playbook as Comerica’s 24 million deliberately disconnected calls.
  • It doesn’t help. RESPA and Regulation E require actually investigating and resolving errors, which a bot that can’t resolve anything fails.

My take:

  1. It looks like the regulators are reinforcing the message that companies won’t get a waiver for their AI providing false information, trapping customers in doom loops, or failing to help.
  2. Air Canada tried to argue its chatbot was a separate legal entity responsible for its own answers. A Canadian tribunal rejected that and ordered the airline to pay damages.
  3. Testing chatbots for failure modes is a must. The harder question is which changes to a deployed chatbot are material enough to require retesting.

AI incident databases are too thin and uneven to measure AI risk

MIT AI Risk Initiative's recommendations for incident database maintainers, with demand from 45 practitioners
MIT AI Risk Initiative's recommendations for incident database maintainers, with demand from 45 practitioners

MIT consulted 45 people in 20 countries who use AI incident databases, from insurers to regulators.

  • Counts don’t show risk. More AI use produces more incidents without making each use riskier.
  • A few people fill the databases. Four people submitted half of the AI Incident Database’s incidents.
  • Reviewers disagree. Two reviewers rating the same incidents reached only “limited agreement.”

We’ll cover business impacts of AI incidents at our October 21 briefing.

Signals:

  1. CISA, NIST, and NSA published ten voluntary practices for handling incidents where agents act against oversight controls.
  2. Claude solved the 25-year mystery of NIST’s curve seeds, and found no sign of an NSA backdoor.
  3. Insurers are weighing an idea to exclude claims for losses caused by AI agents from standard policies.

Sources:

  1. Using a funded test cryptocurrency wallet without authorization, OpenAI
  2. The New Gatekeepers, Harvard Journal of Law & Technology Digest
  3. Preliminary findings on how AI incident databases are used, MIT AI Risk Initiative
  4. Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation, CISA
  5. Claude reveals ECDSA curve origins, Anthropic
  6. AI Runs Into Its Next Big Obstacle: Insurance, The Wall Street Journal