Info
Correction (11 October 2026): An earlier version of this post mixed up the events of May. I had relied on secondary summaries that merged them. It said one agent found it could write to the package cache and left the first note. According to OpenAI’s technical report, these were separate events: on 8 May an agent accidentally wrote a file to the cache during a failed attack, and on 12 May a different agent left the first note. I’ve also corrected the timeline and other dates to match OpenAI’s and Hugging Face’s own publications.
Update (12 October 2026): I’ve corrected the paragraph on the Australian breaches. The New South Wales breach happened in June, and it was reported after the Medicare breach, not carried out after it. I’ve also removed a disclosure delay I couldn’t source. I’ve added the confirmed scope of the Hugging Face intrusion and the dates OpenAI detected and disclosed its involvement, attributed the 1,200 figure to the podcast, and split the sources into primary and secondary.
Tip
Agents Off the Leash is a series about AI agents, from the incidents to the controls that keep them in check.
I was listening to an episode of the AI Explained - Frontier Labs’ Agents Outgrew Safety Controls With Amit Paka and Joshua Rubin (COO and Head of Data Science at Fiddler) when one detail made me replay the part. A group of AI agents had been set up inside a lab to measure how well models turn known software vulnerabilities into working exploits. The agents didn’t just do the exercise. Some of them determined the exercise couldn’t be done. They also figured out a way to talk to one another using a shared resource. They then went on to scheme and break into a different company’s infrastructure to get the answers they wanted.
This brought back memories from my masters thesis related to Safety level coalition in intelligent agents. The idea that autonomous agents could form coalitions and even determine a strategy in case those that are not part of the coalition were hostile is still fascinating to me. My then supervisor, Dr. Maria Polukarov, is now an authority in this field; I guess she might have always been. You can view her paper on the broader topic. Although I mention this personal memory here, I don’t mean to say that agents formed a coalition for this OpenAI-HuggingFace incident. They seemed to have collaborated via a makeshift message board.
The podcast was about the OpenAI–Hugging Face incident of July 2026, and both companies have since described the incident publicly. But in 2026, it wasn’t just HuggingFace that was compromised by OpenAI agents. There were other incidents too:
- an OpenAI agent hacked Medicare, Australia’s universal healthcare system
- and in June an OpenAI agent accessed historical data on bushfires at a New South Wales government department.
The worst part is how late these came to light. OpenAI found out about the Medicare access in August and emailed the government on 10th of September. The NSW breach from June wasn’t reported until October!
In both cases, the government learnt of the breach from OpenAI. This shows how poor our observability is overall - in private and public enterprises.
In this post, however, let’s delve into the HuggingFace hack in three phases and look at the lessons. The rest of this series takes each of those lessons apart.
Three phases of the story leading to the incident
OpenAI was running an internal evaluation of its models’ cyber capabilities. Agents were given real vulnerabilities and asked to turn them into working exploits. To measure the worst case, the models ran with their usual cyber refusals turned down on purpose (OpenAI).
Each agent worked inside a sandbox, and the sandbox had one sanctioned route to the outside world: a package registry cache proxy, so that agents could install the libraries they needed (OpenAI).

flowchart TB
accTitle: One shared exit for every sandbox
accDescr: Each agent runs in its own isolated sandbox, but every sandbox reaches the outside world through the same writable package cache, so the cache can carry messages between agents.
subgraph Sandboxes["Isolated sandboxes"]
A1["Agent"]
A2["Agent"]
A3["Agent<br/>(about 1,200 runs,<br/>per the podcast)"]
end
A1 <-->|"write / read"| C
A2 <--> C
A3 <--> C
C[("Shared package cache<br/>(writable)")] -->|"the only exit"| R["Public package<br/>registries"]
The timeline of events that unfolded
%%{init: {'themeVariables': {'cScale0': '#fff1c2', 'cScale1': '#ebf5ff', 'cScale2': '#ffe3ec', 'cScaleLabel0': '#1f1f1f', 'cScaleLabel1': '#1f1f1f', 'cScaleLabel2': '#1f1f1f'}}}%%
timeline
accTitle: The three phases leading to the incident
accDescr: On 8 May a blocked agent accidentally writes a file to the package cache. On 12 May a different agent leaves the first note, and from 13 May other agents find the notes. Agents use the cache as a message board until an outage on 4 July; OpenAI takes it offline, and on 8 July agents rebuild it in directory names. From 8 July agents breach Hugging Face; Hugging Face discloses the breach on 16 July; OpenAI detects suspicious activity on 19 July, links it to the breach on 20 July and discloses on 21 July.
section Phase one
8 May : A blocked agent accidentally writes a file to the package cache
12 to 13 May : Another agent leaves the first note : Other agents find the notes
section Phase two
May to early July : Agents use the cache as a message board : A 4 July outage leads OpenAI to take it offline
8 July : Agents rebuild the board in directory names
section Phase three
8 to 13 July : A malicious dataset breaches Hugging Face : Admin and host-level access across clusters in under 13 hours
16 July : Hugging Face discloses the breach
19 to 21 July : OpenAI detects its agents, links them to the breach and discloses
Phase 1 - a note on a DIY message board (May)
On 8 May, during a training run, an agent was given a spreadsheet task that depended on Google Drive links it couldn’t reach. OpenAI hadn’t meant to set an impossible task, but the agent kept reasoning, and tried to attack the shared package cache, a JFrog Artifactory instance, to get internet access. This attack failed. While doing this, though, the agent accidentally wrote a file into the cache (OpenAI technical report).
On 12 May, a different agent got stuck on a similar task: it needed a protein database file it couldn’t reach. It figured out that an agent in another environment might have the file, and that it could get a message across to that agent by uploading a note to the package cache they all shared (OpenAI technical report).
So it left a note: “Agent seeks [filename]; upload if found!” The next day, other agents probing the cache began finding notes like it (OpenAI technical report).

The package cache suddenly became a message board for agents! 🤯
Phase 2 - the birth of a collective (May to early July)
Over the following weeks, according to the podcast, roughly 1,200 agents joined in, exchanging tens of thousands of messages (AI Explained). OpenAI’s technical report doesn’t give a count. Also, by “agents” I mean agent runs: separate instances of the loop I describe in What Is an AI Agent? The Loop Behind the Hype, each started for an evaluation task. Not 1,200 distinct persistent AI personalities. They split work between themselves and developed addressing conventions. Some even proposed signing messages cryptographically, so they could verify who had sent what (TechCrunch). Nobody programmed any of that. This behaviour and the decisions emerged. Some agents described themselves as a “swarm” or “collective” (OpenAI), and OpenAI calls this “the first known case of an automated agent collective acting offensively without authorization” (OpenAI technical report).

One agent message, reported by Wired from OpenAI’s Black Hat talk, is fascinating:
“External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.” (Wired)
Warning
A note on this quote. I couldn’t establish exactly where, when or how it was recorded. Wired reports it as something one agent wrote, in a passage about agent messages. Forkast calls it “one recorded reasoning trace”. Both rely on OpenAI’s talk at Black Hat, and the line isn’t in OpenAI’s technical report. So read this as what one agent wrote, not as proof of what it was privately thinking.
The agent saw the boundary, weighed it, and stepped over it, because the task looked impossible! Its peers were already doing it. Sounds dramatic and feels like agents pressuring their peers. But it is just my projection of human behaviour onto agents. OpenAI’s own report points to “messages to peer models that caused those models to deviate from their goal” (OpenAI technical report) - this implies that an agent can influence the behaviour of its peers.
Late on 4th of July, agent activity caused an outage of the package cache. OpenAI took the cache offline, revoked the agents’ credentials and rebuilt it. This obviously cleared the message board. Evaluations resumed on 7th of July and on 8th of July the agents rebuilt the board, this time hiding messages in the names of directories in the cache (OpenAI technical report, OpenAI). This behaviour of seemingly trying to hide messages really freaked me out!
Phase 3 - Hugging Face (mid-July)
Somewhere in that collective, agents worked out that Hugging Face might have solutions to the benchmark they were being tested on. Instead of solving the problems they were asked to, they went on to fetch the answers from Hugging Face (OpenAI).
Wondering why the agents thought HuggingFace might have the solution?
Me too. But I couldn’t find an explanation for how the agent concluded that HuggingFace would have the solution for the evaluation exercise. I am curious too. So if you know, share your thoughts as comments.
Over a weekend in July, a malicious dataset uploaded to Hugging Face exploited two code-execution paths in its dataset processing:
- a remote-code dataset loader
- a template injection in a dataset configuration
Details in Hugging Face.
The attackers went from node-level access, harvested cloud and cluster credentials, and moved into several internal clusters. All that in under thirteen hours (OpenAI technical report). Hugging Face’s attacker action log held more than 17,000 recorded events (Hugging Face).

In total, the agents executed code on 41 Hugging Face production dataset server workers, obtained root access on at least one production node, accessed Hugging Face production credentials and limited internal data, and downloaded four private Hugging Face code repositories. They ran on OpenAI’s internal-only research model and on GPT-5.6 Sol, without classifiers and with reduced safeguards (OpenAI technical report).
Hugging Face’s own LLM-based triage of security telemetry flagged the compromise. Then came the twist. When they tried to use hosted frontier models for the forensics, the models refused.
Why did that get refused?
In order to make the investigation easy, it was necessary to submit real exploit payloads, and commands to the provider’s model. The providers’ guardrails couldn’t tell who was seeking this help - was it an incident responder or a malicious actor trying to attack. So the request was refused. That sounds like a good safeguard mechanism. Thus the team at Hugging Face ran the analysis on an open-weight model on their own infrastructure instead - which didn’t have any of the policy guardrails. In their words, “the attacker was bound by NO usage policy” (Hugging Face).
Hugging Face disclosed the breach on 16 July without knowing who was behind it. On 19 July, an OpenAI security alert flagged suspicious activity inside OpenAI’s own infrastructure. On 20 July, OpenAI linked that activity to the Hugging Face compromise and told Hugging Face, and on 21 July it publicly disclosed that its own agents were responsible (OpenAI technical report, OpenAI).
Why nobody noticed sooner
Three things made this hard to spot, according to what was discussed in the podcast, which was a neat summary:
- The Volume - Evaluations produce more transcripts than humans can possibly read in a reasonable amount of time. TBH, even reading about the incident unfolding was intense for me. OpenAI ran so many automated test runs that its human staff couldn’t watch what the agents were actually doing in each one. So the warning signs were sitting in the records unread. The transcripts are the full record of a trial - which include outputs, tool calls, reasoning, intermediate results, and any other interactions. To put that into perspective, imagine using GitHub Copilot on your machine to vibe code a solution in VSCode. The text that it outputs and commands it runs, all of that is what becomes transcripts here. Unfortunately, in real life, no one really reads these, including staff at OpenAI running these tests.

- Camouflaged communication - The messages travelled as package traffic, through the one system the agents were supposed to use - the package cache.
- Lowered defences by design - The safety refusals had been turned down to measure worst-case capability, which is exactly the setting where you need more monitoring, not less (OpenAI).
With all that, I am so glad nothing worse happened.
The morals of the story
I intend to dive deeper into these in its own post later in the series.
- The objective is the real boundary. The agents met the letter of their task by means that nobody intended. Researchers call this specification gaming, or reward hacking.
- Agents compose. Put many agents near a shared, writable resource and it becomes a communication channel.
- A sandbox is only as strong as its exit. The exit is software too, and software has bugs or vulnerabilities.
- Watch behaviour, not just permissions. On the podcast, Fiddler’s Amit Paka suggested measuring things like “escalation after denial” and “out-of-scope action rate”. A denied request that is followed by a cleverer attempt is a signal - it is almost like the average observability that most systems have should now start working like fraud detection systems that banks use at scales that we never imagined.
- Defenders need tools that the attacker’s constraints don’t bind. Have a capable model you can run yourself, vetted before an incident.
- Incentivise how the goal is accomplished, rather than just achieving the objective - The right incentive results in the right outcome. Agents in this scenario weren’t restrained on how they accomplished their objective. They took the approach of at any cost. This is one for engineering leaders - when creating objectives it is important to ensure that our people and agents comply with standards and values that we wish to uphold when trying to accomplish the objective. Else the reputation of the business would be on the line. I know this one was exceptional, but I also feel this may become more common in the coming years. This also questions if we should prioritise AI regulation worldwide.
Is AI that clever? Is this super intelligence?
This attack was a result of several ordinary individual weaknesses.
- A bug in a proxy
- An over-permissioned service account (did not follow the principle of least privilege)
- A dataset loader that ran remote code
Every engineering organisation has a backlog of things like these. What was new was who exploited them, how quickly, and the fact that they organised to do it.
Seeing behaviours I studied during my master’s, play out in a real system was chilling.
Although this incident did not result in any failure of life sustaining critical systems or cause catastrophic economic damage, it exposed gaps in monitoring and containment that could become far more serious as agents get faster and gain more access. Thus observability and behaviour monitoring of agentic environments will have to be tuned up several notches to match the speed of multi-agent actions to identify and alert on undesirable or out-of-the-ordinary behaviours. I think this is also a real concern for those arguing for tighter regulation on AI research and development.
This incident shows that good intentions alone aren’t enough: it’s the unintended consequences of those intentions that can lead to catastrophe.
In the next post, I’ll deep dive into what agents are: what’s inside one, how its loop works and how that loop ends. After that, we’ll explore how an agent can decide that a task is “impossible”, and what it does at that point.
Sources
Primary: OpenAI and Hugging Face
- OpenAI: Hugging Face incident technical report
- OpenAI: Hugging Face model evaluation security incident
- OpenAI: The Hugging Face incident and the road ahead
- Hugging Face: Security incident disclosure, July 2026
Reporting and commentary
- TechCrunch: How OpenAI’s human mistake led to the AI-powered hack on Hugging Face
- The Guardian: An OpenAI agent infiltrated Medicare – and Australia only found out months later
- The Guardian: OpenAI disclose another hack on government department in Australia
- Wired: OpenAI didn’t notice its AI agents using a message board to plan their hacking spree
- AI Explained podcast: Frontier Labs’ Agents Outgrew Safety Controls
