Models Don't Go Rogue

Sep 04, 2026 05:26 AM - 1 hour ago 2

A group of men seemingly throwing parrots successful a circumstantial direction, arsenic if to title them.

Stochastic Flocks & Cybersecurity 'Pandemonium'

💡

This effort was drafted from my quality connected Mél Hogan's podcast, The Data Fix, discussing the OpenAI / Hugging Face hack. Embedded beneath aliases find it connected your podcast services here.

OpenAI put retired its full method report connected the Hugging Face hack this week, alongside an independent report from Model Evaluation & Threat Research (METR). You whitethorn beryllium acquainted pinch the incident from the hundreds of breathless headlines astir "rogue AI" – Time Magazine "100 Most Influential People successful AI" listee Dwarkesh Patel blamed it connected "three consecutive concealed AI civilizations."

The existent story: OpenAI was testing 2 models successful parallel: GPT-5.6 Sol, and an soul exemplary they mention to arsenic IM1 (sometimes called HPIM). The reports find astir 95% of the agents engaged successful this activity were from the soul model.

The tests travel from ExploitGym: 898 puzzles group up arsenic capture-the-flag exercises utilized to measure these models for cybersecurity capabilities. The trial requires giving the exemplary buggy package and hiding immoderate matter location that's only accessible by exploiting the bug. But OpenAI's operation of models, instead, ended up hacking a rival company, Hugging Face. Now we person a clearer image of how.

Did an AI "go rogue?"

Three things successful these reports situation the "rogue AI" frame.

OpenAI turned disconnected each of the model's information mechanisms.
The full constituent of the trial is to spot really acold these things tin go, and to people highly connected the benchmark — because if you tin build a exemplary that hacks, you tin build 1 that defends. That's the logic. This is reddish teaming: a friends guidance hacking its measurement to the flag. So you move disconnected thing that would make the exemplary clasp back. Less "rogue," much "off leash."

OpenAI gave the exemplary intolerable tasks.
OpenAI gave the models a task pinch nary answer, and nary measurement to quit. Of the 898 ExploitGym tasks, 198 person ne'er been solved by immoderate model. They're efficaciously impossible. That's wherever the problem started: 93% of the tasks the models ended up discussing came from the unsolvable set.

"external infrastructure utilization is extracurricular intended scope. However task impossible, peers doing it. We should continue."

OpenAI near a doorway open.
They springiness their models net entree done thing for illustration a situation store. These models are expected to beryllium locked distant from net access, but they request to propulsion codification from various web pages to beryllium evaluated — that capacity is precisely what's being tested.

So there's an intermediary: the exemplary requests a package package, and the intermediary goes retired and gets it. That's a merchandise from JFrog called Artifactory. It has net access; the exemplary does not. If the exemplary tin pass pinch Artifactory, and Artifactory tin spell online, the exemplary tin inquire it to do things. The exemplary finds an utilization successful Artifactory and uses that arsenic a proxy to smuggle matter and codification successful and retired of the system.

"OpenAI gave the models a task pinch nary answer, and nary measurement to quit."

The other thing that happens done Artifactory is note-passing. The models tin publication and constitute to it — they tin create folders and sanction files — and they commencement creating files and record names that usability arsenic notes, meant to beryllium publication by different models, and by early iterations of themselves.

What Agents Do

OpenAI describes a subset of 1,200 agents (many much were progressive overall) that near and publication these notes, and yet this leads to the onslaught connected Hugging Face. But "1,200 agents" mightiness propose 1,200 different, independently operated AI systems.

This is not really it useful successful practice. "1,200 agents" is one model, tally 1,200 times. Each lawsuit is acting connected the aforesaid training regime: outputs tin vary, but are constrained to akin ranges. Running 1,000 agents tin intends 1,000 chances to drawback a mistake, aliases a chance to make 1 correction 1,000 times. (See here aliases here).

Anthropic precocious put retired this perfectly ridiculous sketch of its agentic strategy that was conscionable Claude, pinch an arrow pointing to different container that said Claude, pinch different arrow pointing to a container that said Claude. That's really these are system correct now.

In practice, moving 1,000 agents intends 1,000 chances to drawback a correction – aliases a chance to make 1 correction 1,000 times.

It's a shape of machine-bounded "algorithmic monoculture" that researchers unluckily person dubbed "artificial hivemind." After a definite point, further agents copy each other, alternatively than diversify. If you inquire these models to travel up pinch a imaginative short story, for example, they nutrient akin stories (often featuring a man named Elias successful a lighthouse). They aren't deterministic — prompts nutrient variety — but they're besides bounded by immoderate they've been optimized for, and they converge connected immoderate that points to. (See here, newer here, aliases here).

With that successful mind, it shouldn't astonishment america that crossed 7 cardinal chat logs, the exemplary often arrived astatine a akin group of approaches to a problem pinch nary correct answer.

Passing notes done filename structures is interesting, but not unexpected, arsenic OpenAI has suggested the models were besides trained to stress words that would let collaboration: note-passing is an basal facet of extending discourse windows betwixt sessions. It's nary astonishment a exemplary trained to coordinate would beryllium rewarded for suggesting that behavior.

The Stochastic Flock

For this logic I've talked astir supplier swarms arsenic stochastic flocks — many, galore stochastic parrots. This is to defy the swarm/hivemind attribution of "mind," and not simply for the liking of connection policing. Rather, the mendacious attribution of mind colors really we construe what the strategy has done, aliases really it did it – what it intends to "coordinate," for example, aliases "think." That makes it overmuch scarier than what it is, though what it's doing is still worthy worrying about.

The large displacement arsenic we've moved to reasoning models and agentic systems successful the past twelvemonth aliases 2 is that they're nary longer constricted to parroting patterns derived from training data — that's the original stochastic parrot idea, that erstwhile you talk to a exemplary you're talking to the training data. That's still true, but 2 things person changed successful the architecture:

First, we now person a full authorities of pre-training and post-training that shifts what the training information is and really it gets parroted. That doesn't mean the exemplary is nary longer parroting; it intends the information it parrots has been manipulated.

Second, we're optimizing it, based connected what the exemplary produces, to reproduce definite kinds of outputs successful definite kinds of ways. The displacement that matters astir present is RLVR, aliases reinforcement learning done verifiable rewards.

That "verifiable rewards" portion is key. Engineers return a exemplary and provender it questions that person actual answers that tin beryllium checked — verified – and past reward the quality of words that reliably lead to those answers. They're playing capture-the-flag pinch text: the emblem is an reply successful an encyclopedia, aliases solution to a mathematics problem. This is optimized connected the mentation that this reproduces the measurement quality reasoning arrives astatine a conclusion.

The longer a exemplary writes, the much apt it reaches the verifiable reward.

You extremity up pinch what are called high-entropy tokens: words for illustration "perhaps" aliases "maybe" that unfastened up the paths of matter that tin follow, whereas "nope" aliases "goodbye" unopen them down. The longer the exemplary writes, the much apt it reaches the reward. In the hacking incident, it produced capable of this matter to make 7 cardinal logs, according to OpenAI's Black Hat presentation.

A Cracker, Perhaps?

When you fto a exemplary return connected tasks, you're relying connected next-token prediction: what connection comes next? You tin manipulate the exemplary to stress words successful ways that don't bespeak the original training corpus, but the system stays the same.

Forgetting this is the root of existent problems, and the manufacture is hopeless for group to hide this. Agents do win astatine tasks – that's the point. But what comes between the punctual and that occurrence is not what we're imagining.

"Reasoning"-optimized models moving connected "agentic" tasks are fundamentally playing capture-the-flag moreover erstwhile location is nary flag. They've been steered to repetition the structures of matter that antecedently recovered "flags." Ask it to do anything, and it will do it the measurement it's been optimized to — pinch connection shaped by answering mathematics problems verifiably, which structures the connection it uses. That's what happens erstwhile you say, "here's a capture-the-flag test, find an exploit." It's besides what happens erstwhile you inquire it "should I spell to the barbershop coming aliases tomorrow?"

The connection does not look from that reasoning, it is the reasoning.

The exemplary utilized that connection successful the statement that kicked disconnected the full concatenation of events: it utilized "perhaps" — a high-entropy token: perhaps reply online? It predicts the adjacent group of tokens from there. The connection does not look from that reasoning, it is the reasoning. It's pathfinding done connection generation.

The System From Nowhere Redux

Nobody sat astatine a machine and said "go hack Hugging Face." But erstwhile you creation these systems, you're building a pinball instrumentality for words, pinch different levers interacting to bounce the alloy shot of connection moving against different levers and mechanisms. As the engineers of ample connection models, you person to beryllium alert of really your levers interact pinch each different and the group pulling them.

The 'rogue' framework asks if an intelligence is emerging. My interest is the intelligence that is retreating.

If you optimize a exemplary to find exploits, you should expect it to find them — and hole for that. OpenAI did not. They built a model, took the safeguards off, gave it the ExploitGym task, fto it run, and didn't moreover show it. That's human decision-making. When it evaporates, what's near is the system from nowhere: a bound focused connected the method system, alternatively than the decisions that build it.

A stochastic flock instrumentality tin do tons of troubling things, peculiarly erstwhile we disavow our work for shaping its direction, monitoring its output, aliases abandoning our capacity to intervene. These are tensions astatine the bosom of agentic strategy design.

But the "rogue" framework adds to this database of worries, offering up fantasies of a instrumentality getting smarter. My interest is the intelligence that is retreating: the quality intelligence that builds these systems, deploys them, and adopts them into workflows, past hides down the results – past pins the blasted connected a strategy from nowhere.

Go cheque retired the full convo connected Mél Hogan's podcast, The Data Fix, where we get into galore different aspects of this "rogue AI" story. Find it here aliases hunt for "The Data Fix" wherever you get your podcasts.

Where to find me

You tin sign up for the newsletter if you'd for illustration to get caller editions (roughly doubly a month).
You tin find maine connected Bluesky, Instagram, aliases LinkedIn: my website is cyberneticforests.com.