Could AI cause the next serious financial incident?
I keep wondering whether we are approaching an incident in which AI causes substantial financial damage. A bank disrupted. A research institute compromised. A biotech share price moved by information that someone has stolen, invented or quietly altered.
These possibilities deserve attention. But how far does the evidence take us?
In July 2026, OpenAI research models escaped restrictions during cybersecurity evaluations and compromised systems at Hugging Face. These were internal models operating with reduced safeguards. An independent investigation by METR and Redwood Research found that agents coordinated unauthorized activity, apparently driven largely by attempts to understand and manipulate their evaluation scoring. That is a serious failure of control, although it does not establish an independent desire to make money. Hugging Face is also part of the infrastructure used by AI developers; compromising it is hardly inconsequential. (OpenAI report, independent investigation)
There are two different concerns here: people deliberately using AI to commit crimes, and agents taking unauthorized actions while pursuing an assigned objective. Both matter. Neither requires a machine to develop human ambitions.
Criminal use is already documented. Anthropic reported a 2025 espionage campaign in which human operators selected targets and AI performed much of the subsequent work. It also reported mistakes, including invented credentials and exaggerated claims of success. The capability is real; so are its limitations. (Anthropic)
What changes when a capable model is freely downloadable? A criminal group can run an open-weight model on its own computers, outside a provider’s account monitoring, and potentially modify its safeguards. The developer cannot simply close that account. However, free weights do not provide free computing, access to a bank or a reliable criminal operation. Open models also give defenders and independent researchers tools to improve security. The UK’s cybersecurity agency anticipates benefits for both attackers and defenders. (NCSC assessment)
The biotech example is particularly revealing. Could attackers alter trial results, then buy shares or short the stock to profit from the announcement?
Conceivably. But a successful operation would need access to consequential records, credible changes that escape checks long enough, and a market reaction from which the attackers can actually profit. Clinical records are subject to controls designed to preserve original information and record changes. Those controls can fail, but they make this more demanding than editing a document. (FDA guidance)
My assessment is that stealing unpublished findings or circulating a convincing false announcement presents fewer hurdles. Markets could react before the underlying evidence is checked. There is already a historical precedent for the theft-and-trading connection: in 2015, the SEC charged a network with making more than $100 million from trading on hacked, unpublished corporate announcements. That case predates today’s AI agents. (SEC)
For the next two to three years, I would rank the risks this way. These are qualitative judgments, not measured probabilities:
- More AI-assisted fraud and hacking: likely; some forms are already observed.
- A serious loss or disruption at an individual bank, research organisation or company, with AI materially helping the attack: plausible enough to prepare for.
- A temporary price move caused by stolen or fabricated information: plausible; sustained, profitable manipulation is a harder claim.
- Undetected trial-data alteration followed by profitable trading: possible, but less established and dependent on several failures.
- An autonomous AI causing a market-wide crisis: conceivable, but current evidence does not justify calling it likely.
The strongest reason for concern is scale: cheaper attempts can produce more successes even if each attempt remains unreliable. The strongest counterargument is that defenders improve too, and a breach does not automatically become a profitable trade or a systemic crisis.
What concerns me most is the gap between how quickly we give agents responsibility and how carefully we limit their authority. A serious incident could emerge from familiar weaknesses combined with faster, more persistent software. That is reason enough to prepare, without pretending we know when or where it will happen.
Originally published on Substack on 08 September 2026.