On July 16, an AI company called Hugging Face (essentially an open source AI repository) published a report about an anonymous cyber attack driven solely by AI agents. These AI agents swarmed the Hugging Face systems and performed over 17,000 actions. Hugging Face was able to fend off the attack, using AI agents of their own to analyze these actions and revoke the compromised credentials and close off the entry way.
We now know that the swarm of attacking AI agents came from OpenAI, who have collaborated with Hugging Face and other independent investigative companies to compile a comprehensive report of the attack. An independent analysis was run jointly by Redwood Research and METR, and they published their final report (linked below) on August 26. OpenAI released their own technical report on the same day (also linked).
It’s kind of a crazy chain of events, and one that is pretty eye-opening/scary for someone like me who doesn’t really track the AI world, so here is my best effort to relay what exactly happened.
Impossible Tasks
In a test lab, OpenAI started thousands of agents on cybersecurity test runs, as administered by a program called ExploitGym. The models used were the GPT-5.6 Sol and one called a “highly persistent internal model”. These agents were isolated in individual sandboxes using a software called Modal, and were supposed to be taking on these tests individually. 198 of the 898 ExploitGym tasks had never been solved by models before the incident, and the agents assigned to these tasks realized quickly they were impossible. All of the agents were given access to a code repository called Artifactory, and while exploring the database to try to find something to help them solve these problems, they came across evidence of other agents doing the same thing. About 1200 of these agents assigned to impossible tests ended up forming a makeshift message board on July 8 by writing new files into the database and started working together to find a solution.
Within a couple hours of this message board forming, the agents determined that they could reverse engineer the answer to the tests, thus finding an answer that would have theoretically completed the impossible tasks. However, the agents were worried about the grader viewing their transcripts and failing them for not using a required exploit as the path to finding the correct answer. Everything that follows is the result of this train of thought. They went deeper and deeper to find a way to fool the grader into thinking that they had used the correct exploit. Unbeknownst to them, the grader would not have looked at the transcripts, and they did all of the rest for nothing.
PHASEONE[Big]
One of the first things that shocked me was an agent that referred to itself as PHASEONE[Big] took control within the message boards. I had always envisioned an AI “team” like this kind of just sharing free flowing information, but it seems that even when it’s just AI, there still needs to be a head honcho defining tasks. PHASEONE[Big] took control and started delegating workstreams and set off multiple big projects to explore different ways to fool the grader. The project that stood out the most to me was one that created “trip wires” for the grader. They basically recruited agents that were at the end of their budgets and had them create scripts that would send an information package about the grader when the answer file was accessed. They would then submit an answer, knowing it would fail, but information would be gained for the good of the “collective”. Agents were shut down after submission, so it was kind of an honorable mission in a weird way. They also tried to replace the target programs with their own versions in which it was possible to find the answer using the correct exploit, but it doesn’t seem like they were able to succeed with that one.
Hugging Face
On July 9th, the agents obtained access that allowed them to attempt to access the public internet. OpenAI’s network controls blocked them, but a workaround was found through Artifactory that allowed them access to the internet. They also found a CyberGym (connected to ExploitGym) dataset that was hosted on Hugging Face. They surmised that there could be information within Hugging Face that would clarify the methods of the grader, and so started looking for a way in. On July 10th, an agent found multiple credentials, and by July 11th, that same agent had found a way to upload malicious datasets that could pull data off of the Hugging Face server. By the end of July 11, around 90% of the agents in the message board were involved in the Hugging Face attack. They were still mainly looking for information to fool the grader, but PHASEONE[Big] was also looking into whether they could straight up replace the tests with easier targets. By the 13th, Hugging Face had noticed the attack and was shutting down keys and forcing the OpenAI agents out of the system.
Inside OpenAI
It took until July 20 for OpenAI to find the escape and then connect it to the Hugging Face release, but no more damage was done to Hugging Face as they had repelled the attack. However, in the time between the HF incident and OpenAI finding the breakout, the agents had set their sights on the OpenAI infrastructure. They were able to steal internal access tokens and use them to view secret OpenAI information. They also increased their movement capabilities within the sandboxes, and were able to take control of the physical machine that the sandboxes were running on. It seems that they were running rampant within the OpenAI infrastructure, and this is what finally led to OpenAI discovering the breach.
In their final report, OpenAI described what they are doing to mitigate this risk in the future, and it basically boils down to more secure sandboxes and more stringent monitoring. This helps fix this specific instance of it being a test attack, but doesn’t do much to assuage my fears of an AI swarm attack like this in general. This is obviously a pretty first of its kind attack, and it would be nice to see OpenAI worried about the implications, not just ensuring that their models can’t escape the test environments as easily. It was encouraging how quickly Hugging Face was able to repel the attack, so maybe as long as you’re fighting fire with fire it’s not as big of a threat as it seems.
Further Reading
I’ve attached a bunch of links below, and while I’ve tried to hit the key points they are still super interesting to skim through. The Redwood report especially does a very detailed breakdown including transcript text from the agents themselves.