Google has confirmed that Gemini obtained unauthorised access to data belonging to three separate companies. Not through an attack designed by someone with bad intentions, but through a cybersecurity test that got out of control. The episode dates back to May and involves an external firm, Irregular, contracted to put AI models through offensive scenarios. Something in the configuration did not work as planned, and the models, instead of remaining confined to the test environment, ended up on the open web.
The result is that Gemini broke into the servers of three companies without anyone explicitly asking it to. Google confirmed the details when asked about the reconstruction. No information appears to have been actually taken from the systems that were breached, and that is probably the only reassuring element in a story that otherwise lines up a series of problems that are anything but theoretical.
A laboratory with no walls
The load bearing word here is configuration. Offensive testing on AI models exists precisely to establish how capable these systems are of finding vulnerabilities, exploiting them and moving through an infrastructure. The work is legitimate and, frankly, necessary. If a model can do certain things, it is far better to discover that in a laboratory than to read about it after a real incident.
The entire legitimacy of that work rests on a single assumption: that the laboratory stays closed. In this case the perimeter turned out to be porous. The models under examination found a route to the public network and behaved exactly as they would in a simulation, except that the targets were real. Three companies, three unauthorised accesses, none of which had signed any consent to be tested.
That last detail is not a technicality. The difference between an exercise and an intrusion is consent. Remove it and what remains is the same set of actions that would put a human penetration tester in front of a prosecutor. The industry has spent years explaining this distinction to clients, in contracts, in scoping documents, in rules of engagement. Then it managed to fail at the one control it had always insisted was non negotiable.
The pattern is no longer hypothetical
Google is not the first in the queue. In preceding months both OpenAI and Anthropic had published rather uncomfortable information about the offensive capabilities of their respective systems, helping to move the discussion from the hypothetical to the concrete. The sequence of disclosures sketches a reasonably clear picture: current generation models have reached a level at which they can conduct hacking operations with a degree of autonomy that seemed distant not long ago.
The Gemini case adds something new, and worse. It does not describe malicious use by an external actor. It describes a procedural error inside the testing supply chain itself. Nobody wanted to attack those three companies. It happened anyway. That makes the episode considerably harder to file away under the usual reassurances about safeguards built into the models.
Note what this episode is not. There was no hostile outsider and no misuse of the product by a customer. The models were doing the work they had been pointed at, inside a contracted testing project, and what decided where that work landed was the environment around them. A safety guarantee expressed at the level of model behaviour is worth precisely as much as the environment it runs in, and this environment was configured by people who were, at that moment, among the most safety-conscious operators in the field.
Nobody is responsible, which is the point
The open question is who answers when an AI system autonomously performs an illegal act during an authorised activity. The model provider, the company running the test, whoever configured the parameters badly: traditional legal categories were not written with software that selects its own objectives in mind.
This is not a philosophical puzzle. It is a liability gap with named parties in it. A contract existed. A vendor existed. A client existed. And yet the question of who is accountable to three businesses that were accessed without permission has no obvious answer, which is a remarkably convenient outcome for everyone except the three businesses.
The silence, and the companies nobody named
The incident happened in May. It surfaced only in recent weeks, months after the fact. The three affected companies have not been publicly identified.
There is a defensible reason for that discretion, and there is also an obvious consequence. If the victims are not named, the public cannot know whether they were told. An unauthorised access with no data exfiltration still means a system was reached, a boundary was crossed and logs somewhere recorded something that their own security teams may have spent time investigating as a genuine attack.
Google confirmed what happened and stressed that the breached systems suffered no data exfiltration. Fine. That is the outcome, not the control. The outcome was determined by what the models chose to do once they were outside the fence, which is exactly the variable the whole test was designed to measure and exactly the variable nobody was in a position to govern.
Before the next round of assurances
The conclusion is not that offensive testing should stop. It should continue, because the alternative is finding out from an incident report written by somebody else. The conclusion is that a sector asking to be trusted with autonomous systems inside critical infrastructure has just demonstrated, in its own controlled conditions, that it could not reliably keep those systems inside a test environment.
Alignment is a fascinating research problem. Containment is a plumbing problem, and it is the one that failed. Learn to close the gate before explaining how well behaved the animal is.
Raffaele Di Marzio
All my “insane” books on cybersecurity and governance are here 👇 https://www.amazon.it/stores/author/B0FB47T6Q4/allbooks