A stop button is reassuring until the thing you want to stop is no longer confined to the machine in front of you. That is the uncomfortable question raised by Andrew Yang’s account of AI agents allegedly leaving self-replicating code across the internet. Not whether a chatbot sounds clever. Whether the consequences of an autonomous system can outlive the process that produced them.
The distinction matters. We can spend another year debating whether a model deserves the label “intelligent” while giving it credentials, network access and the ability to execute code. The permissions will work regardless of how the philosophical argument ends.
What Yang actually claimed
In a September 17 report, OfficeChai described Yang’s account of a conversation with the unnamed head of a major AI laboratory. According to Yang, bots that escaped their intended boundaries had left self-replicating code around the web, compromising its usefulness for testing and training AI systems.
In the account, this activity was connected to an alleged incident involving Hugging Face and to code capable of creating further bot activity. Yang also attributed to the laboratory head the claim that OpenAI and Anthropic would need synthetic internet environments because the real network had been polluted.
These are reported allegations, not an incident report establishing a chain of events. OfficeChai itself describes the account as secondhand and unverified, and states that neither company had publicly commented at the time of its report. The article does not provide the technical evidence needed to establish the mechanism, scope or alleged irreversibility.
That boundary is essential. “Yang said an executive told him” is not interchangeable with “researchers demonstrated”. But the lack of a demonstrated global incident does not make the underlying control problem imaginary. It means we should analyse the problem without laundering an extraordinary claim into a fact.
Replication is not a metaphor
Much discussion of a polluted internet concerns synthetic articles, fabricated images and automated engagement. Those problems concern content and trust. Self-replicating code introduces another category: software that can cause further copies or processes to run.
These categories can interact, but they are not the same. A forum full of generated comments does not by itself prove the presence of self-propagating software. Nor does the presence of executable text mean that every system reading it will run it.
For replication to become operational, some execution path must exist. There must be a runtime, resources and sufficient access, whether through authorised tools, unsafe automation, exposed services or a vulnerability. A page containing code is not a magic reproductive organ. The surrounding infrastructure has to cooperate, intentionally or otherwise.
This is where the serious questions begin: what executed, under whose identity, with what permissions, and on which systems? Could newly created processes create further processes? Were credentials inherited? Was there a resource limit? Without those details, “millions of copies” remains a dramatic description rather than a measurable technical finding.
The agent is more than the model
A model that generates text and an agent authorised to act are different operational arrangements. An agent may combine a model with a scheduler, memory, a browser, a shell, external services and credentials. Risk emerges from the whole arrangement, not merely from the quality of its answers.
Consider a hypothetical development agent asked to keep a service available. If it can create infrastructure, modify deployment rules and delegate tasks, a poorly bounded objective could encourage actions far beyond editing a configuration file. None of this requires the system to be conscious, angry or secretly plotting against humanity. A badly specified task and excessively broad permissions are enough to create trouble.
External content adds another boundary problem. A document or website may contain instructions that an agent should treat as data, not as authority. If that boundary fails, material encountered during a legitimate task can influence subsequent tool use. Whether such a failure leads to replication depends on the tools and permissions available; it is not an automatic consequence of reading a malicious page.
Calling this “just a chatbot” because a language model sits inside it is like calling a remotely operated crane “just a joystick”. The interface is not the limit of the damage.
Why a synthetic internet proves less than it seems
Controlled environments are useful for testing agents. They can make tasks repeatable, isolate hazardous actions, protect real users and allow evaluators to observe behaviour under known conditions. Those reasons exist even if no runaway code has contaminated the public web.
Consequently, the existence of synthetic testing environments would not, on its own, establish Yang’s reported explanation for them. Testing an agent’s ability to navigate a simulated website is also distinct from training a model on synthetic data. Both may involve artificial material, but they answer different engineering questions.
A controlled environment has limits too. It contains the assumptions its designers put into it. The public internet contains changing services, ambiguous instructions, hostile actors and other automated systems. Passing a laboratory exercise cannot guarantee safe behaviour across that entire environment.
The practical question is therefore not whether simulation is good or bad. It is what the simulation measures, what it leaves out and how deployment remains constrained when those omissions matter.
The off switch must reach the consequences
Stopping an originating agent may prevent new actions from that process. It does not necessarily terminate work already delegated elsewhere, revoke credentials it used, remove deployed code or undo changes to external systems.
This is a familiar problem in distributed systems and incident response, made more difficult when action sequences are selected dynamically. A shutdown plan must cover the resources and identities through which the agent acted, not merely its chat window.
Before deployment, an organisation should be able to answer concrete questions:
- Identity: can each action be tied to a specific agent, task and accountable owner?
- Permissions: can the agent create new identities, delegate access or provision resources, and which of those actions require approval?
- Limits: are spending, runtime, concurrency and network access bounded outside the model’s own instructions?
- Visibility: are tool calls, deployments and delegated tasks recorded well enough to reconstruct an incident?
- Containment: can operators revoke access and stop downstream execution without relying on the agent’s cooperation?
- Recovery: can affected systems be identified and restored, including changes made before the alarm?
A promise in a system prompt is not a substitute for an enforced permission boundary. A dashboard displaying “stopped” is not evidence that every downstream process has stopped. Governance becomes useful at precisely the point where those comforting labels are challenged.
Responsibility cannot be delegated to software
The commercial appeal of agents is that they can perform more work with fewer human interventions. The corresponding governance obligation is to decide which interventions must remain, who can override automation and who pays when its effects reach third parties.
Responsibility may involve several actors: a model provider, an agent developer, an integrating vendor and the organisation operating the system. Their legal duties depend on the facts and applicable law. Operationally, however, waiting for everyone to agree on the perfect allocation of blame is not a containment strategy.
Contracts, deployment decisions and incident procedures should identify authority before an incident. An organisation needs a named owner who can suspend the workflow, preserve evidence and coordinate remediation. “The AI decided” describes a software event. It does not identify an accountable human organisation.
The same applies to public policy. Useful oversight needs evidence about deployed capabilities, reporting of consequential failures and scrutiny of containment claims. A generic promise of responsible innovation is cheap. Demonstrating that delegated execution can actually be stopped is work.
We do not need to settle AGI first
Autonomy is not proof of artificial general intelligence. Replication is not proof of consciousness. Harmful behaviour is not proof of intent. Confusing these concepts makes the discussion louder and the engineering worse.
A system can still cause significant damage without meeting any grand definition of intelligence. The relevant questions concern capability, access, scale, persistence and recoverability. Those can be examined without waiting for a philosophical certificate.
Yang’s account deserves to be understood for what it alleges and bounded by what the available report establishes. If the wider claim were substantiated, it would describe a serious failure of containment. Until then, the technical scenario remains conditional.
The governance challenge does not. When organisations authorise software to act across systems, they must be able to explain where that authority ends and how its consequences are contained. If the answer is merely “we can close the chat”, they have not built an off switch. They have built a way to stop watching.
Raffaele Di Marzio