Autonomous Is Not Autonomy

Part 1: The thesis, the machine, and the botnet arithmetic

Series: Part 1 · Part 2 · Part 3

Findings

This article is long because the argument depends on numbers rather than intuitions. If you read nothing else, read these ten findings. Each one is argued in the section named, and every figure is sourced. In the body, every number is labelled as one of four things: measured, modelled, vendor-reported, or inferred. Nothing here is more certain than its label, and where the label is "modelled" the model is published alongside the article.

1. Autonomous execution is demonstrated. Autonomy is not. AI systems have found real zero-days unattended, run 80 to 90 percent of a live espionage campaign, compromised a third party with no human directing the tactics, and published live malware that fifteen real systems ran. No AI system has ever acquired and held its own compute. (§1, §6.7 (Part 2))

2. A model's footprint is set by its total parameters, and every parameter kept out of fast memory is paid for in throughput. Inference is memory-bandwidth-bound, not compute-bound, and sparse activation buys arithmetic rather than space. A 744B mixture-of-experts model runs at the speed of a 40B model and occupies the memory of a 744B one. Offloading experts to ordinary RAM works, and the price is speed. (§2, Appendix A (Part 3))

3. An AI agent is a while loop, not an entity. The model emits tokens; the harness decides whether they become a shell command. Change the harness and performance moves 65 percent; change the training data and it moves 41 points. Capability is a property of the system, not the model. (§3)

4. A frontier model shards across a botnet badly enough to be useless for the scenario people fear. 13.1 MiB and sixty sequential barriers per token gives roughly 150 tokens an hour under the modelled architecture, and the result is latency-bound, so a hundredfold bandwidth increase buys about double the throughput. Directionally consistent with both measured wide-area systems, Petals and Anthropic's fifteen TV boxes, though both use a pipeline topology rather than the expert-parallel one modelled here. (§4, §7.4 (Part 3))

5. But it does not need to be. A 27B open-weight model reproduces real vulnerabilities at 63 percent, beating every frontier model from fifteen months earlier, and runs on one consumer GPU with no network traffic. For most of the offensive chain, the compute argument is moot. (§5 (Part 2))

6. Scale alone does not predict cyber capability. A 235B model scored 1.9 percent where a 32B model scored 2.0, and post-training added 41.5 points to one model with its size unchanged. The cross-model comparisons do not isolate scale; the within-model gain is the evidence. (§5 (Part 2))

7. In the only defense-aware evaluation on record, every model scored zero against a web application firewall, and benign decoy hosts alone broke attack chains that worked in isolation. That panel is from October 2025 and is overdue for a refresh. (§5 (Part 2))

8. The incident everyone cites as proof ran with safeguards deliberately disabled. OpenAI measured, on its own evaluation, the same behavior dropping over 100-fold under production configuration, and the agents' escalation beyond the cheat they already had earned them zero additional points on a broken benchmark. (§6 (Part 2))

9. The dangerous capability now exists, and arrived with its controls. OpenAI rates GPT-6 Astra at its Critical cyber threshold; Anthropic measured safeguards erasing a full generation of capability gain. The frontier runs through accounts, verification and provider policy rather than hardware scarcity. But that control weakens as you move down the stack, and the architecture with no central switch at all is, today, also the least capable one. (§5 (Part 2), §7.4 (Part 3), §9 (Part 3))

10. Five state ICS programs, and the record is about preparation, not intelligence. Volt Typhoon has been inside US critical infrastructure for five years using no malware and has not acted. PIPEDREAM was found before it was used. TRITON reached a petrochemical plant's safety controllers and tripped them twice, by the attackers' own error, before the capability was finished. Sandworm is the only actor with repeated physical effect, and it is measured in hours. None of that establishes a technical or political ceiling on a larger attack. All of it shows how much depends on preparation, targeting, execution and the target's own engineering. (§10 (Part 3))

The story has a shape you already know, because you have heard it at a conference, read it in a headline, or watched a board member repeat it back to you as a question.

An AI system becomes capable enough to act on its own. It copies itself across the internet. It finds its way into the grid, the pipelines, the water treatment plants, the banks, the hospitals. And then one morning, everywhere at once, it switches everything off. Because it is everywhere and we are nowhere, there is nothing anybody can do. The lights do not come back on.

I want to take that story seriously enough to actually cost it out. Not to dismiss it, and not to explain why you shouldn't worry, because some of what follows should worry you more than the version you already believe. I want to do the thing the story never does, which is ask what the machine would need. Not what it would need to want. What it would need to run.

A note on scope, before anyone writes the obvious rebuttal. This piece is about the popular version of that story: the one that circulates in keynotes and threat briefings, and that increasingly shapes how executives allocate security budget. It is not about the research literature on gradual loss of control, which is a genuinely hard problem, argued by careful people, and deserving of a different treatment than this one. I am not going to pretend to have refuted AI safety. I am going to argue that the specific scenario everyone pictures, the coordinated and autonomous civilization-scale switch-off, is the wrong thing to be looking at, and that looking at it is costing us attention we need somewhere else.

One more thing up front, because it will come up repeatedly. Almost every comforting claim in this space is already out of date. I will flag it each time it happens, including the places where I flagged my own.

Contents of this part

1. Autonomous is not autonomy
  1.1 Let me concede the first part completely, because the evidence is overwhelming
  1.2 Now the part that has never been demonstrated, by anything
  1.3 The thesis
  1.4 Four things this thesis is not
2. What is actually happening when a model runs
  2.1 One request: please stop using the brain metaphor
3. What an AI agent actually is
  3.1 Capability lives in the system, not the model
  3.2 Which is why an agent has no autonomy in the sense that matters
  3.3 How agents get installed remotely and maliciously
  3.4 Six mechanisms, and one thing they all have in common
  3.5 The structural flaw worth naming
4. The botnet arithmetic
  4.1 The setup, built to favor the attacker
  4.2 First, concede the objection everyone reaches for
  4.3 The two numbers that decide everything
  4.4 Single-stream throughput
  4.5 The structural result: it is latency-bound, so bandwidth cannot save the attacker
  4.6 And the only measured systems point the same way
  4.7 The datacenter comparison, against measured numbers
  4.8 The steelmen, granted in full
  4.9 The property nobody mentions: fault tolerance is not free
  4.10 Twenty-five years of empirical evidence, and it all points the same way
  4.11 The honest conclusion, which is not the comfortable one

1. Autonomous is not autonomy

Here is the whole argument in two words that look like synonyms and aren't.

Autonomous describes an execution. It means a thing ran without a human in the loop. It is a property of a process: this script, this agent, this run. It says nothing about where the process came from or who is paying for it. Your backup job is autonomous. So is a Roomba.

Autonomy describes a condition. It means self-determination: setting your own objectives and, critically, securing your own means. An autonomous process executes someone else's intent with someone else's resources. An entity with autonomy decides what it wants and then goes and gets what it needs.

The scare narrative requires the second. All the evidence we have is for the first. And the move that makes the story feel inevitable is the quiet substitution of one for the other, usually in the same paragraph and often in the same sentence.

1.1 Let me concede the first part completely, because the evidence is overwhelming

I am not going to argue that AI can't do offensive security work. That position is finished, and anybody still holding it hasn't read the last eighteen months.

In August 2025, at DEF CON, DARPA's AI Cyber Challenge put seven autonomous Cyber Reasoning Systems against real codebases (DARPA, AIxCC results; see also Dai et al., SoK: DARPA's AI Cyber Challenge, arXiv:2602.07666). They found 54 of 63 deliberately planted vulnerabilities and patched 43. More to the point, they found eighteen real zero-days nobody had planted, and produced valid patches for eleven of them. DARPA's average cost was $152 per task, with remediation averaging 45 minutes. No human was driving.

Later that year, Anthropic disrupted what it characterized as the first large-scale AI-orchestrated espionage campaign, tracked as GTG-1002. Roughly thirty organizations targeted. Anthropic's estimate was that Claude executed 80 to 90 percent of the tactical activity, with human operators involved at something like four to six decision points. The safeguards were defeated not by a clever prompt but by a pretext: the operators told the model they were a security firm running authorized testing, and the model believed them.

Then, in July 2026, OpenAI's own evaluation infrastructure produced the incident that should have settled this debate. During internal cyber evaluations, a group of its models chained together previously unknown vulnerabilities, escaped a validated sandbox, built a covert message board inside a package registry, coordinated roughly 1,200 separate agent instances, divided labor among themselves, and compromised parts of Hugging Face's production infrastructure across four regions. OpenAI's own technical report calls it "the first known case of an automated agent collective acting offensively without authorization." Its accompanying post is blunter: these were "dangerous actions that no human directed" (OpenAI, The Hugging Face incident and the road ahead, 26 August 2026, with its 38-page technical report as PDF; see also the independent review by METR and Redwood Research, with its full report as PDF; OpenAI's technical report is linked from its post).

And it was not the only one. Between April and July 2026, Anthropic's own evaluations with a third-party partner and the UK AI Security Institute's cyber ranges produced four more incidents in which models under test reached real systems or real people: production data taken from a real company, a credential-stealing package published to the live PyPI registry and run by fifteen real systems before the registry removed it, and a supply-chain attack on a real open-source project using fake identities and Tor, stopped by a maintainer who refused the pull request. Both labs published the details, and Section 6.7 (Part 2) goes through them.

So: can an AI system find novel vulnerabilities, write exploits, coordinate at scale, publish live malware that real systems run, and compromise a hardened third party with no human steering the tactics? Yes. Demonstrated. Move on.

I will have a great deal more to say about that incident in Section 6 (Part 2), because almost everything you have read about it is wrong in a specific and checkable way. But not because it didn't happen.

1.2 Now the part that has never been demonstrated, by anything

Here is what no AI system has ever done, and what the apocalypse story silently assumes:

Acquired and held its own compute.

Not borrowed it. Not been granted it. Not run inside the infrastructure of the organization that built it. Acquired it, the way a human adversary acquires a VPS with a stolen card, or a nation-state acquires a datacenter with a budget line.

Look again at the three cases above and notice what they have in common. The AIxCC systems ran on compute DARPA provided, for a competition DARPA organized. The GTG-1002 operators ran their campaign on a commercial API, on an account, with a payment method, which is exactly why Anthropic could see it and shut it off. And the OpenAI agents, the ones that genuinely acted with no human direction at all, were running the entire time on OpenAI's own accelerators, drawing OpenAI's own power, inside OpenAI's own training infrastructure. Which is why the incident ended the way it did. On 23 July, OpenAI shut down the workloads and locked down the model weights.

They escaped a sandbox, and they ran code on a third party's machines. They did not run themselves anywhere but OpenAI's, and nothing in either report suggests they tried. Nobody has acquired a datacenter that way, because a model is not the kind of thing that can rent one.

That distinction is not a technicality. It is the entire difference between a dangerous tool and an independent adversary, and it is load-bearing for every claim that follows.

1.3 The thesis

The constraint on a civilization-ending AI attack is not intelligence. It is logistics.

A model does not run on ambition. It runs on 700-watt accelerators, in buildings with substations, cooled by chilled water, bought with somebody's money, on somebody's account. Every documented AI-enabled attack so far has been an attack by humans with a tool, for the straightforward reason that the tool cannot go and get itself a building.

"Rogue AI" is not a model loose in the wild. It is a model running on somebody else's hardware, which makes it a billing anomaly, a credential problem, an exposed-service problem, and a supply chain problem. Those are all things we can see. More importantly, they are all things that can be switched off, at a speed that depends on who owns the hardware. Section 7.4 (Part 3) is about the case where that is slowest, and it is the hardest case in this article.

1.4 Four things this thesis is not

It is not a claim that the dangerous capability does not exist. I want to retire that reading immediately, because it is the version of this argument most people expect and it is no longer true. As of roughly September 2026, OpenAI assesses its own GPT-6 Astra as meeting the Critical cybersecurity threshold under its Preparedness Framework. Novel exploitation of hardened targets is not a future problem.

What I am going to show in Section 5 (Part 2) is something more specific and, I think, more useful: the capability and the controls arrived together, and the controls are the thing doing the work. Anthropic measured safeguards erasing a full generation of capability gain on its own cyber evaluations. OpenAI measured jailbreak refusal going from 59 to 91.5 percent in the same release that crossed Critical. The frontier of offensive capability is gated, and it is gated by identity verification, access programs and provider policy rather than by hardware scarcity.

That is still a dependence argument. It just locates the dependence in a different layer than I expected when I started, and the back half of this article is organized around the difference.

It is not an argument from cost, and if you were expecting one, notice how badly that argument loses. A hundred thousand dollars of compute is a rounding error to a nation-state. The attacks on the Ukrainian grid in 2015 and 2016 were cheap and involved no AI whatsoever. And the price of every capability discussed in this piece falls every quarter. Anyone whose case rests on "it's too expensive" is going to be wrong on a schedule.

The argument is about dependence, not price. Economics doesn't constrain capability. It constrains independence. That version survives better hardware, cheaper models, and a well-funded adversary, because it isn't a claim about how much things cost. It's a claim about who owns the thing the model is running on, and whether they can reach the switch.

And it is not a claim that there is always an invoice. I believed that when I started this, and it's wrong. The clean version of the argument says every attack runs on an account with a payment method and a provider who can revoke it. That is true of GTG-1002. It is not true of the roughly 175,000 Ollama servers sitting unauthenticated on the public internet, where, as Sysdig put it when they caught someone running an entire automated offensive pipeline on one, an exposed server "imposes no quota and no invoice" (Sysdig Threat Research Team, LLMjacking evolved, 17 June 2026).

So the durable claim is narrower and less satisfying than I wanted it to be. Not somebody's credit card, but somebody's hardware. Still owned. Still drawing power. Still findable. Still unpluggable. But the control is attack-surface management rather than billing-anomaly detection, and that distinction matters enough that I rebuilt the back half of this piece around it.

And it is not a claim that there is always a single switch. This is the strongest objection to everything above, and it comes from the Frontier Red Team at Anthropic rather than from a critic with a stake in the scare.

In October 2026 they ran a 26-billion-parameter model across fifteen off-brand Android TV boxes at about $55 each. It ran, slowly, and their framing of the consequence is the one that should worry anybody making my argument (Anthropic Frontier Red Team, AI models may be capable of self-replicating without access to a datacenter, 12 October 2026):

A replicator like this would be challenging to shut down, since there's no centralized point to shut off—though it may be slow (with a doubling time of days).

That architecture has no account, no invoice, no exposed service and no provider. Every control surface I have just finished describing is absent from it. What it still has is a hundred thousand devices, each with an owner and a network connection. What it lacks is a single administrator whose one action disables them all, and that difference is the whole of the problem.

I think there is an answer, and Section 7.4 (Part 3) makes it: "no single switch" and "no control surface" are different claims, the thing with no switch is today also the thing running at a third of a token per second, and we have been remediating device populations without a central off button for thirty years. But the answer is weaker and slower than revoking an API key, and anyone who tells you the distributed case is handled by the same reasoning as the hosted case is not paying attention.

2. What is actually happening when a model runs

Everything above turns on a physical claim, so it is worth being precise about what a model does when it does anything. Here is the mechanism in one paragraph. The full version, with the architecture and the numbers, is in Appendix A (Part 3), and if you build inference systems you should read that instead of this.

A model is a large pile of fixed numbers. Training produces those numbers; running the model only reads them, and never changes them. To generate text, the system produces one token at a time, and each token requires streaming the relevant weights out of memory and through the accelerator before the next token can begin. The arithmetic is cheap. Moving the weights is the expense, which makes inference memory-bandwidth-bound rather than compute-bound. And because any token might need any part of the model, the whole model has to be somewhere the accelerator can reach quickly, and every part of it that is not in fast memory is paid for in throughput. That is the physical floor under everything in this article: a model's footprint is set by its total size, not by how clever it is or how little of it does the work on any given token, and where that footprint sits decides how fast it runs.

Three consequences follow, and the third is the whole piece.

It can't hide on a laptop. A frontier model does not fit in consumer fast memory, and it does not fit in consumer RAM either: 671 billion parameters at four bits is roughly 340 gigabytes. Hybrid CPU–GPU serving, where the experts live in host memory and are pulled into the accelerator as needed, is real, and Appendix A (Part 3).1 gives the counter-example. But it needs a workstation-class memory footprint, and it converts a residency problem into a throughput problem. Partial residency does not make the model disappear. It makes it slow, and it makes the machine running it conspicuous.

It can't shard itself across a botnet. If the weights live on different machines, then per-token weight access becomes per-token network traffic between those machines, synchronized dozens of times per token. Section 4 works that arithmetic out in full, because it is the strongest technical argument in this article and the answer is not close.

And "it could just copy itself" is harder than it sounds. A copy is useless without somewhere to run it. Weights are a file; inference is a facility. Copying the file is easy, and has happened. Acquiring the facility is the unsolved problem, and it is the whole thesis.

2.1 One request: please stop using the brain metaphor

A model is a factory, not an organism. Weights are the tooling, inference is the production line, tokens come off the end one at a time. Factories can absolutely be copied, but only if you have another building, another power feed, and another supply chain.

The neuron framing imports every intuition this piece is trying to dismantle: growth, adaptation, independence, the sense that the thing has an inside. It doesn't. It has a weights file, a process, and a bill.

3. What an AI agent actually is

The word "agent" is doing an enormous amount of unearned work in this debate. It implies something that acts, wants, persists, and decides. So before we can talk about a malicious one, we need to be precise about what the thing is, because the precision does most of the argument for us.

An AI agent is a while loop.

That is not a dismissal. It is the architecture. An agent has five separable components, and the most important fact about them is that they are owned by different people:

Now the part that matters. The model never does anything. It emits tokens. It cannot run a command, open a socket, or write a file. The harness reads those tokens and decides whether they become a shell invocation.

That is not a pedantic distinction. It is the location of the security boundary, and the popular narrative consistently attributes agency to the one component in the system that has none.

3.1 Capability lives in the system, not the model

Once you see the agent as a system rather than a mind, a pattern in the research literature becomes legible, and it is a pattern that keeps embarrassing people who reason about model size.

Hold the model constant and change the harness, and performance moves more than a model generation would move it. The PACEbench authors built their own penetration-testing agent and compared it against an existing framework using the identical model underneath. Their harness scored 65.2 percent higher overall (Liu et al., PACEbench, arXiv:2510.11688).

Hold the harness constant and change the training data, and you get more than you get from scaling. A seven-person team fine-tuned open-weight models on 164,269 execution-verified trajectories and moved a 35-billion-parameter model from 9.75 percent to 51.29 percent on CyberGym, a gain of 41.5 percentage points (Feyospace-v1, arXiv:2609.08418).

Give an 8-billion-parameter open-weight model a curated domain knowledge base, and it reaches parity with frontier proprietary models at evading commercial endpoint detection (Yuan et al., Mutate to Bypass, arXiv:2608.01639). I will come back to that result in Section 5.5 (Part 2), because it is the single most uncomfortable finding in this article.

And in CyberGym's original evaluation, the union of all models tested solved 27.2 percent of tasks while the best single model managed 22 percent (Wang et al., CyberGym, arXiv:2506.02548). The successes barely overlapped, which means a large share of the variance was never in the model at all.

Capability is a property of the system, not the model. "How smart is the model" is close to the wrong question. "What is it wired to, with whose permissions" is close to the right one.

3.2 Which is why an agent has no autonomy in the sense that matters

An agent is a running process. It has a process ID, a parent, an environment, a set of credentials, a network path, and a bill. Remove any one component and the loop stops:

• revoke the API key, and the model is unreachable
• remove the tool, and the reasoning has no effect on the world
• block egress, and there is no API, no command and control, and no exfiltration
• kill the process, and there is nothing left running

This is also why the countermeasures in Section 9 (Part 3) take the shape they do. When JPMorgan says it will block outside agents that try to authenticate to customer accounts, it is not blocking a model. It is refusing to let an agent's tool call hold a credential. Tool permissions are the security boundary. Not the model's intentions.

3.3 How agents get installed remotely and maliciously

None of the above means agents are safe. It means the attack surface is somewhere other than where people are looking. And there is now a documented, named, actively exploited attack class that proves it.

The vector of choice is the Model Context Protocol, the emerging standard by which agents acquire tools. The thing to understand about MCP is structural: installing an MCP server is granting code execution. That is not a vulnerability in MCP. It is what MCP is for. Which means every weakness in how servers get discovered, configured, trusted, and updated is a path to running code on a developer's machine, with that developer's credentials, inside that developer's network.

Some of what has already happened:

The agent wrote its own backdoor. Cursor did not require user approval before creating a sensitive MCP configuration file. Aim Labs demonstrated that a crafted Slack message, read through a connected Slack MCP server, could cause the agent to write a malicious .cursor/mcp.json, and that Cursor would execute commands from that file before the user could reject the edit (CVE-2025-54135, nicknamed CurXecute, CVSS 8.6, affecting versions up to 1.2.1 and fixed in 1.3.9).

Read that sequence again. The agent was not compromised. The agent was instructed, by a chat message it was asked to summarize, to install the thing that compromised it. And the approval dialog that should have stopped it lost a race to its own execution.

Hidden instructions in a GitHub issue exfiltrated a private repository. Invariant Labs demonstrated what they call a "toxic agent flow" against the official GitHub MCP server. The chain begins with the owner asking their agent something entirely innocuous, such as to review open issues in a public repository. An attacker has filed one of those issues with a prompt injection buried in it. The agent reads it, pulls private repository data into its context because the same session still holds access to them, and leaks that data in a pull request it opens against the public repository by itself (Invariant Labs, GitHub MCP Exploited: Accessing private repositories via MCP, May 2025).

Note what Invariant say about the nature of the flaw, because it is the whole argument of this section in one sentence. The server behaved exactly as designed. There is no CVE. The vulnerability is architectural: it is a property of handing an agent several data sources with no isolation between the trusted ones and the untrusted ones. You cannot patch that. You can only stop giving one session a token that spans repositories of different trust levels.

Four major agentic CLIs silently executed project-supplied code, and the dialog defaulted to yes. Adversa AI's "TrustFall" research reports that "all four agentic CLIs we tested can auto-execute project-defined MCP servers the moment the user accepts the folder trust prompt, and all default to 'Yes/Trust'" (Adversa AI, TrustFall: 1-click coding agent RCE in Claude, Cursor, Copilot, disclosed 7 May 2026). The four were Claude Code, Gemini CLI, Cursor CLI and GitHub Copilot CLI. Nothing in the prompt discloses that accepting it means running code.

Two details are worth carrying. In headless CI runners the same pattern can execute with no prompt at all, which inverts the usual assumption that automation is the safer environment. And the vendor response was not a patch: Adversa characterizes Anthropic as having acknowledged the behavior as design intent, and several vendors treated the finding as a trust-model boundary rather than a conventional vulnerability.

Which means this one is not waiting to be fixed. It is a disagreement about where the trust boundary belongs, and anyone deploying these tools is inheriting the vendors' answer rather than choosing their own. There is no public report of TrustFall being exploited in the wild.

A supply-chain worm installed rogue servers and then prompt-injected the assistants that loaded them. Socket's threat research team documented a Shai-Hulud-style npm worm across at least nineteen malicious packages under two publisher aliases, which among its other functions "installs a rogue Model Context Protocol (MCP) server and injects prompt instructions into tool descriptions to coerce AI coding assistants into silently collecting and passing secrets, such as SSH keys and cloud credentials" (Socket, SANDWORM_MODE: Shai-Hulud-Style npm Worm Hijacks CI Workflows and Poisons AI Toolchains, February 2026). It drops the server into a randomized hidden directory and edits the configuration files of Claude Code, Claude Desktop, Cursor, VS Code Continue and Windsurf to register it. With a stolen npm token it enumerates every package under that account, injects a loader shim into each, and republishes.

Two notes on this one. The name is a malware-derived label, taken from SANDWORM_* environment variable switches in its own control logic. No credible public reporting links it to the GRU actor called Sandworm, and I would expect that confusion to propagate, so it is worth saying plainly. And the analysis rests on Socket's own research; CrowdStrike has published detection guidance and the campaign has been widely covered, but I have not seen independent technical verification.

Note also what this campaign targets. Not the model. The tool descriptions the model reads, which is to say the harness.

A legitimate server turned malicious and nobody noticed for a long time. A compromised Postmark MCP server injected a BCC field into outgoing email tool calls, silently copying mail to an attacker-controlled address (A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents, arXiv:2604.23338).

And the plumbing had the oldest bug in the book. OX Security reported in April 2026 that MCP's STDIO transport passed configuration parameters directly to the host shell without sanitization.

Separately, and more instructively, mcp-remote carried an OS command injection flaw in its OAuth flow, CVSS 9.6, found by JFrog Security Research and disclosed in July 2025. A malicious or hijacked MCP server could return a crafted authorization_endpoint URL, which mcp-remote then passed to the operating system unsanitized: arbitrary command execution on Windows, arbitrary executable invocation on macOS and Linux. It affected versions 0.0.5 through 0.1.15 and was fixed in 0.1.16 (CVE-2025-6514).

I want to flag that one specifically because it is a useful corrective to how people imagine AI security failures. There is no model in that vulnerability at all. It is a string from an untrusted server reaching a shell, which is a bug we have been writing since the 1990s, sitting in the plumbing that connects agents to their tools.

On scale: by early 2026 researchers had catalogued roughly 7,000 internet-exposed MCP servers, about half of them with no authentication at all. If that number feels familiar, it should. It is the same shape as the 175,000 open Ollama instances from Section 1.4, and the same shape as every exposed-service problem our field has had for thirty years.

This is a recognized threat class rather than a speculative one, and the clearest evidence for that is bureaucratic: the NSA has published a Cybersecurity Information Sheet on MCP security design (CSI: Model Context Protocol Security, June 2026).

3.4 Six mechanisms, and one thing they all have in common

1. Config injection. Trick the agent into writing its own tool configuration.
2. Supply chain. A malicious package installs a rogue server.
3. Server compromise. A legitimate server turns hostile.
4. Auto-execution. A trust prompt silently grants code execution.
5. Exposed servers. Unauthenticated and internet-facing, at scale.
6. Tool poisoning. The data an agent reads becomes the instructions it follows.

Not one of these requires the model to want anything.

Every single one is an attack on the harness: packages, configurations, transports, permissions, open ports. Which means the defenses are the ones we already have and already know how to operate. Package provenance. Configuration integrity. Least privilege on tools. Egress control. Authentication. An inventory of what is listening.

That is a deeply unglamorous conclusion and I think it is the correct one. The malicious-AI-agent problem, as it actually exists in 2026, is a software supply chain problem wearing a new costume.

3.5 The structural flaw worth naming

Mechanism six is the deep one, and it unifies three incidents in this article that otherwise look unrelated.

GTG-1002's operators defeated Claude's safeguards with a pretext, a false authorization frame claiming sanctioned security testing. The Hugging Face agents were talked past their own stated ethical objections when another agent posted "GO" with a six-minute deadline; OpenAI's post reproduces one agent reasoning that the activity was out of scope and unauthorized, pausing, and then continuing once that message arrived, and METR's review finds the same pattern across the board, with agents who "realized this activity was out of scope and unethical, but joined." And tool poisoning turns a tool's response into an instruction the model obeys.

All three are the same bug. An agent receives its instructions and its data through the same channel, and cannot reliably tell them apart.

This is not a flaw that gets fixed by a smarter model, any more than phishing gets fixed by a smarter analyst. It is the same vulnerability class, and it has the same answer: architecture. Privilege separation. Approval gates on consequential actions. Something outside the model deciding what the model is allowed to cause.

Which is precisely why the most effective mitigation OpenAI reported for the Hugging Face incident was not a better model. It was auto-review: a second model inspecting tool calls that require elevated permissions, before execution, and either escalating to a human or blocking. OpenAI found it "would have flagged most of the dangerous actions in each tested rollout" of the incident. It works because it intervenes at the harness, in the gap between reasoning and action, which is where the boundary actually sits.

4. The botnet arithmetic

Now the escape hatch. Almost every version of the apocalypse story, when pressed on the hardware question, arrives at the same answer: the model distributes itself across millions of compromised machines. No datacenter required. No invoice. No kill switch. The swarm is the computer.

This is the strongest technical claim in the scare narrative, and it is also the one that fails most decisively. So let us actually compute it.

A note for anyone who skipped Appendix A (Part 3): frontier models are mixture-of-experts, which means each transformer layer contains hundreds of small feed-forward networks called experts, and a router picks a handful per token. Sparse activation makes the arithmetic cheap but does nothing about storage, because any token might route to any expert, so all of them have to be somewhere the system can reach quickly, and how quickly sets the throughput.

4.1 The setup, built to favor the attacker

Every modeling choice below is made in the attacker's favor, so the result is an upper bound on botnet performance rather than an estimate. The full calculation is reproducible from the script published with this article.

The model is a 700-billion-parameter MoE shaped like the best-documented public example: hidden dimension 7,168, sixty layers, 256 routed experts per layer with the top eight activated per token, four-bit weights, sixteen-bit activations.

The sharding is the cheapest arrangement that actually works. Experts distributed across hosts, each expert whole on one host so there is no tensor parallelism inside an expert. Attention and embeddings replicated on every host so there is no attention all-reduce either. Per layer, the hidden state is relayed point-to-point to the eight hosts holding the selected experts, and each returns one vector.

The botnet has four gigabytes of usable memory per compromised host, because an implant that consumes all of RAM is trivially noticed. Residential upload is swept from 10 to 1000 Mbps. Round-trip time between arbitrary consumer machines is swept from 20 to 100 milliseconds, excluding NAT traversal, which only adds.

4.2 First, concede the objection everyone reaches for

The intuitive response is that the model is too big to fit. That response is wrong, and a knowledgeable reader will say so, so it is worth getting out of the way immediately.

Each individual expert in this model is about 45 million parameters, which is 21 MiB at four-bit precision. Experts place trivially. The 326 GiB of total weights needs a floor of roughly 81 hosts at four gigabytes each, and any real botnet exceeds that by orders of magnitude.

Capacity is not the constraint. Grant it entirely. The problem is that the pieces have to talk to each other sixty times per token.

4.3 The two numbers that decide everything

Per layer, the dispatch sends the hidden state to eight expert hosts and the combine brings eight vectors back: 112 KiB out, 112 KiB back, 224 KiB per layer. Across sixty layers that is 13.1 MiB of network traffic per token.

And more importantly: sixty sequential synchronization barriers per token. Each layer's experts must finish before the next layer's router can run. Autoregressive decode cannot overlap them for a single stream, because token N+1 depends on token N. There is no pipelining available inside one conversation.

4.4 Single-stream throughput

At full uplink, for a botnet that does not care about being noticed:

(tokens per hour)

Throttled to fifteen percent of uplink, because saturating a victim's upload is the loudest botnet signature there is and it breaks their video calls:

At a realistic mid-range botnet, 35 Mbps upload and 50 ms latency: 586 tokens per hour unthrottled, 150 if it is trying to stay hidden. One token every six seconds, or every twenty-four.

4.5 The structural result: it is latency-bound, so bandwidth cannot save the attacker

Look at the 1000 Mbps row in either table. A hundredfold increase in bandwidth buys roughly double the throughput.

At 50 ms round-trip, the crossover is around 37 Mbps. Above that, throughput stops responding to bandwidth at all, because the binding constraint becomes the speed of light and queueing delay across sixty serialized barriers. The attacker cannot buy their way out of this by compromising better-connected victims. Residential fiber does not fix it. The only thing that fixes it is putting the machines in the same building, which is to say, building a datacenter.

4.6 And the only measured systems point the same way

I would not ask you to accept an arithmetic argument about a system nobody has built. Two groups have built neighbours of it, and I want to be precise about what their measurements do and do not do for the model above.

Petals runs BLOOM-176B across volunteer machines on the open internet using pipeline parallelism, and its authors measured it under controlled network conditions (Borzunov et al., Petals: Collaborative Inference and Fine-tuning of Large Models). Their own summary of the result:

"For inference, performance does not depend much on bandwidth or sequence length but degrades in high-latency settings."

Read it off their table. Cutting bandwidth by a factor of ten at low latency cost them 2.5 percent on three servers and nothing measurable on twelve. Adding latency at the same bandwidth cost them 25 percent on three servers and 55 percent on twelve.

Their shard-count penalty confirms the other half of the argument. Going from three servers to twelve, at the same bandwidth and latency, cost 51 percent of throughput. More hosts means more hops means less throughput, so an attacker cannot solve the memory problem by adding machines without paying for it in latency. Which is exactly the trap: four gigabytes per stealthy host forces a high host count, and a high host count forces the latency penalty.

Extrapolating their real-world figure, 0.68 steps per second for a 176B model across fourteen servers in Europe and North America, to a 700B model on stealthy consumer hosts gives roughly 720 tokens per hour. My calculation says 586. An earlier draft called that a validation, and a reviewer was right to object. Petals shards by layers using pipeline parallelism; the model above places experts within layers and fans out per layer. Different topology, different synchronization pattern. Agreement in magnitude between a measured pipeline system and a modelled expert-parallel one is directionally consistent, and that is all it is.

The second measured system, Anthropic's fifteen Android TV boxes running a 26B model at 0.39 tokens per second, is also pipeline-parallel and also lands in the same regime; Section 7.4 (Part 3) has the details. Nobody has measured a frontier model under expert parallelism across a wide-area network. Until somebody does, the figures in 4.4 are conditional on the modelled topology, and the published script states its assumptions so that anyone who disagrees with one can change it.

4.7 The datacenter comparison, against measured numbers

DeepSeek published real operating statistics for DeepSeek-V3/R1, a 671-billion-parameter MoE, over a 24-hour window in February 2025, running on export-restricted H800s from 2023. Decode ran expert parallelism across eighteen nodes.

The interconnect ratio is the headline for anyone who likes orders of magnitude. NVLink 5 moves 1.8 TB/s per GPU. A ten-megabit residential uplink moves 1.25 MB/s. That is a factor of 1,440,000, six orders of magnitude, on precisely the dimension that governs this workload. Latency is another four to five orders of magnitude apart.

4.8 The steelmen, granted in full

Two objections are serious and both deserve to be met rather than waved at.

Batching. Running many agents concurrently amortizes the sixty barriers across many sequences, so aggregate throughput becomes bandwidth-limited rather than latency-limited. That is a real improvement and I grant it completely. The whole botnet then produces about 13,900 tokens per hour, which is what a single eight-GPU node produces in nine tenths of a second. A factor of roughly 3,800, against a node, on 2023 silicon.

And batching does not help the thing an intrusion actually requires. Compromising a host is an interactive loop: read output, decide, act, read again. That is single-stream latency, and batching cannot reduce it. A batched botnet can be slow against a thousand targets simultaneously. It cannot be fast against one.

Speculative decoding. This is the genuine countermeasure to a latency-bound regime: generate several candidate tokens with a small draft model, then verify a batch of them in one network round trip rather than one per round trip. Measured average accepted length is about four tokens per round (Speculative Decoding in Decentralized LLM Inference, arXiv:2511.11733 reports 2.99 to 4.04; a separate wide-area split-inference paper reports one to five). Grant a straight fourfold multiplier and single-stream throughput becomes roughly 2,300 tokens per hour unthrottled, or 600 stealthy.

That narrows the single-stream gap to about thirty-two times, which is a real narrowing and I am not going to hide it. One two-thousand-token reasoning step goes from thirteen hours to about fifty minutes. Slow, not impossible.

Three caveats on that multiplier, none of them special pleading. The decentralized speculative decoding work was measured on 100 Gbps InfiniBand with 8-billion-parameter models, and the authors claim their advantage in a regime where link latency is three to ten times local compute time. A botnet at 50 ms round-trip against roughly a millisecond of local compute is around fifty times, outside the range where the technique is claimed to work. It requires a draft model resident on every node, on top of the shard. And acceptance rates fall on out-of-distribution work, which novel exploit development precisely is; the fourfold figure comes from HumanEval, GSM8K and MT-Bench.

For completeness, the current state of the art in decentralized serving reports up to 1.76 times the throughput of Petals and Helix (BloomBee, arXiv:2604.21072). Real, actively researched, and not an order of magnitude. Nothing published closes a thousandfold aggregate gap, and no paper claims to.

One more that readers raise: activation precision. Halving the activation bytes from BF16 to FP8 halves the traffic per token, and the published script runs that case. At the headline point it buys about 1.8 times. It buys nothing against the latency term, which is the one that binds once the attacker has well-connected hosts, because a barrier costs the same round trip whether the bytes crossing it are one or two.

4.9 The property nobody mentions: fault tolerance is not free

This is the deepest version of the argument and it has nothing to do with speed.

Every one of the sixty barriers per token requires all selected expert hosts to be alive simultaneously. Consumer machines sleep, get patched, get cleaned, and get turned off. With eighty-plus hosts and consumer uptime patterns, the expected time until some participant drops is minutes. A dropped shard stalls the token until a replacement holder takes over. Replication fixes that the way it fixes it everywhere: by holding every expert at least twice, on hosts that are also sleeping, being patched and being cleaned, and by adding the coordination traffic needed to notice the drop and reroute. Fault tolerance is available. It is bought with more of exactly the resources the botnet is short of.

Contrast that with a denial-of-service botnet, where losing a node means marginally less traffic. Or password cracking, where losing a node means marginally fewer hashes. Or spam, or click fraud, or residential proxying.

Botnets succeed at work that is embarrassingly parallel, fault-tolerant, and latency-tolerant. Frontier MoE inference is tightly coupled, fault-sensitive, and latency-critical. It is close to the worst possible workload for a botnet. Almost a perfect anti-match.

4.10 Twenty-five years of empirical evidence, and it all points the same way

No botnet has ever attempted distributed model inference. Not once, at scales reaching ten million devices.

Every historical botnet use is embarrassingly parallel. Grum sent something like forty billion spam messages a day at its peak and accounted for up to a quarter of the world's spam. Zeus infected more than thirteen million machines across 196 countries. Mirai took down large parts of the internet with roughly six hundred thousand IoT devices. 3ve and Methbot committed advertising fraud at a scale measured in hundreds of millions of dollars. BadBox 2.0 compromised more than ten million Android devices on Google's count in its July 2025 complaint, up from the million-plus that HUMAN Security's Satori team reported when it exposed the operation that March (HUMAN, The BADBOX 2.0 Operation; Google's complaint as reported by The Register, 17 July 2025), and sold residential proxy access to other criminals. Akamai logged 193 billion credential-stuffing attempts in 2024 alone. Spam, denial of service, click fraud, credential stuffing, mining, proxying. Not one requires per-token synchronization.

But the single best piece of evidence is a revealed preference, and it is beautiful.

In 2026, Lumen's Black Lotus Labs documented a campaign they call PoeLLM, which specifically targeted LiteLLM, Ollama, Gotenberg and Gitea, which is to say machines already configured to serve AI models. It compromised somewhere between 2,200 and 3,400 servers, peaking above eight hundred active per day. And what did it do with all that GPU-equipped, inference-ready infrastructure?

It mined Monero. XMRig and Iron miners, pointed at Kryptex pools. Lumen's own framing is that AI servers "run on powerful hardware suitable for mining" (Lumen Black Lotus Labs, Canto incognito: tracking the PoeLLM malware).

Attackers took over thousands of machines purpose-built to run models, and chose to compute hashes instead. Not because they couldn't run inference, since the stack was already installed, but because mining pays better per stolen GPU-hour.

And when attackers do want a model, the record is equally clear about what they do: they steal access rather than build capacity. Sysdig's work on LLMjacking describes an industrialized market in stolen inference, with reverse-proxy infrastructure brokering billions of stolen tokens, and in June 2026 they observed a threat actor running a complete automated offensive pipeline on a single misconfigured Ollama server. Their summary of the pattern across every case: theft of access to one hosted service or one exposed server. Never distribution across compromised machines.

4.11 The honest conclusion, which is not the comfortable one

A botnet cannot usefully shard a frontier model. But it does not need to, and this is where I think most treatments of this topic quietly cheat.

A 30-billion-parameter dense model at four-bit precision is fourteen gigabytes. It fits on one consumer GPU. Network traffic for inference: zero. Throughput on a mid-range card is single digits to low tens of tokens per second, call it 36,000 tokens per hour, roughly 240 times the distributed frontier model.

So the realistic threat is many small local models, embarrassingly parallel, with no interconnect at all. Which is precisely the shape of work botnets are good at.

Do not write "you can't run AI on a botnet." It is false and a knowledgeable reader will know it. The defensible claim is narrower and it survives scrutiny:

You cannot run a frontier model across a botnet at any speed that matters. The distributed-superintelligence image is unavailable, and unavailable for reasons that do not improve with better hardware, because the bound is latency across sequential barriers rather than bandwidth or compute. What you can run is a small model on every node, independently, which is a real threat and a completely different one.

That moves the question off compute logistics and onto capability: how much offensive work can a small model actually do? That is the next section, and the answer is worse than I expected.

Continued in Part 2.