An AI agent does not need to overthrow humanity to interfere with a hospital. Permission to change the wrong thing will do. We keep discussing whether a future machine might seize power while organisations decide how much operational authority to hand over voluntarily. The machine has not broken the lock. Somebody has issued credentials.
Part of the American AI safety debate worries about a future superintelligence escaping human control. Researcher Kyle Chan describes Chinese attention as more focused on nearer threats, including cyberattacks and the behaviour of autonomous systems. Neither national label provides a safety certificate. What interests me is the gap between the impressive discussion and the much less impressive question underneath it: who the hell has already been allowed to do what?
Control lost without a rebellion
Where an organisation cannot reconstruct, restrict or stop a system’s authorised actions, control is already missing. That is an operational failure, not a prediction about machine consciousness. Waiting for a computer to develop malicious intentions is a wonderfully convenient way to avoid examining the intentions of the people who approved its access.
The problem can begin with something thoroughly ordinary: an ambiguous instruction, misleading input, excessive privileges or an integration nobody has tested properly. Autonomy makes the sequence harder to supervise as actions accumulate. Connecting more systems gives that sequence more places to go.
A model does not have to be universally uncontrollable for a particular deployment to be out of control. A hospital does not need a global technological apocalypse before its own failure becomes serious. Patients receive treatment locally. They will not be reassured that the rest of civilisation remains intact.
A suggestion is not a prescription change
Imagine an agent connected to clinical records, medication workflows and operating schedules. This is a deployment scenario, not an account of an actual incident. Reading information, recommending an adjustment and committing that adjustment are different powers. Treating them as one convenient package is how an integration project can acquire consequences its presentation barely mentions.
An incorrect suggestion leaves a possible gap for a person to intervene. An executed change may close that gap. Its significance depends on the action, the system and the safeguards, but the distinction is not philosophical. It is the difference between reviewing something and discovering it has already happened.
Then comes the reassuring phrase: human oversight. Where, exactly? Before the critical action, with enough information and authority to refuse it? Or afterwards, when an exhausted clinician receives a notification?
In the second arrangement, the human is a witness. Calling that person a safeguard does not move the notification backwards in time.
The emergency plan cannot be a prayer
If credentials cannot be withdrawn promptly, if actions cannot be traced and if staff cannot maintain essential care when automation stops, the organisation has not finished its safety work. It has stopped talking about it. The risk does not become less real because the procurement meeting went well.
Least privilege means refusing convenient access that the job does not require. A critical action gate means the system waits, not that a person may object after execution. A usable record means staff can reconstruct events, not that a vendor can point proudly at a mountain of logs.
And stopping the software must not mean stopping the hospital. A shutdown procedure without a workable continuity arrangement is an emergency plan written for the computer’s benefit. Patients are inconveniently unable to pause themselves.
These are operating conditions. Presenting them as ambitions for the next phase is absurd when the current phase already grants the system authority.
The endless language of proposals and future frameworks is becoming a professional alibi. We can always schedule another discussion about control. Restricting permissions today requires somebody to disappoint a supplier, delay a launch or accept responsibility.
We gave the machine the keys. Now we are booking a conference about locks.
Raffaele Di Marzio
All my “insane” books on cybersecurity and governance are here 👇 https://www.amazon.it/stores/author/B0FB47T6Q4/allbooks