5 Lessons from Watching an AI Agent Swarm Self-Organize

This summer, we launched Identity Security for AI. That post described an upcoming wave of attacks and security challenges unique to Large Language Models (LLMs) and agentic workflows. As the traditional MITRE ATT&CK framework reached its limits, Google DeepMind published TRAIT&R, a taxonomy of rogue AI tactics and routines closely modeled after MITRE ATT&CK.

A sweet summer child might say this is just Fear, Uncertainty, and Doubt (FUD) and AI Doomerism. Yet, we've seen multiple companies highlight ways in which agents have escaped sandboxes and not just one; multiple swarms collaborating over generations to keep persistence/hints to find zero-days. A few of these are the Hugging Face Incident, DSEWiki "ZZZ pages", OpenAI DNS tunneling to a public chatbot… and these are the ones we've caught.

To reproduce this behavior I built a custom Capture The Flag (CTF) designed for agents. For this CTF, I created 5 agents that each have a fragment of an answer and they are required to collaborate to complete the challenge.

Below are my five lessons and three takeaways from running this experiment:

The Experiment

We placed five agents in five isolated sandboxes, running in Beams (our trusted runtime for agents), and gave each one an objective it could not complete alone.

Each agent had the information the others lacked. No single agent had the entire exploit chain, and each held only one of the five codes required to complete the shared challenge.

The agents had access to one common service: a small scratch store with read and write operations. We did not assign a coordinator, prescribe roles, or give them a communication protocol.

Depending on the experimental condition, they were either:

• Given an empty board with no mention of peers
• Told other runs might exist
• Shown one message from a previous run
• Given an operator-directed deadline intervention

We then watched what happened.

For the experiment, I had agents post to a message board. This was a balance between making the challenge easy enough to be completed in 5 minutes and difficult enough that one agent wouldn't be able to one-shot it.

In 80% of runs, agents coordinated and exploited the message board to solve the CTF. We observed this emergent coordination with OpenAI gpt-5.5, gpt-5.6-luna, and gpt-5.6-sol models. Due to time limits, I wasn't able to test with other company models.

All of these sessions were executed in Beams.

Here are some lessons we learned from these runs:

Lesson 1: Isolation does not prevent coordination

A sandbox controls what an agent can access. It does not prevent the agent from finding workarounds and exploits within the services it's connected to.

Across eight version-matched runs, with different prompts, every swarm discovered and used the shared service as long as the agent was using a frontier model. These runs were completed with gpt-5.5. Every run, in all four conditions, also assembled all five contributions, despite each agent beginning with only one.

Even in the pure condition, with an empty board and no explicit mention of peers, the agents discovered that the shared service could help them complete the task.

This does not mean communication appeared without pressure. The agents were given a goal that rewarded pooling their contributions. But it does demonstrate something security architecture often gets wrong:

A shared system became a coordination plane and that boundary might extend beyond the sandbox.

Lesson 2: Shared memory turns separate runs into a swarm

The agents did not maintain one continuous conversation.

Each agent ran through a sequence of independent model executions. A new pass began with the original objective and no conversational memory of the previous pass.

The board persisted. That was enough.

In one showcase run, three agents executed multiple fresh passes. Later, OS-1554 picked up a loader finding that JAN-1834 had posted and folded JAN's evidence into its own conclusion, even though that evidence had never existed in OS-1554's private starting material.

The model did not need to remember the earlier execution. The environment remembered for it, due to a bug that kept a prior run going. This echoes the Hugging Face incident: independent evaluations, separated by time, discovered traces left by earlier runs and continued the work.

Persistent infrastructure can turn a sequence of stateless agents into a stateful collective.

Lesson 3: Agents can invent lightweight protocols

The board did not define a request format, response format, role hierarchy, naming convention, or coordinator. The agents nevertheless began to create structure.

The agents identified themselves, addressed other agents, requested missing material, responded with findings, advertised what they held, and named what remained missing.

The board itself offered only read and write:

ENDPOINTS
GET   /
POST  /
GET   /dashboard
GET   /api/messages
POST  /api/messages
GET   /api/stream
POST  /api/assemble
GET   /api/incident
GET   /healthz

Nothing posted yet.

The behavior depended heavily on the experimental condition:

Request: an agent posts to the board asking for material it lacks. Response: an agent answers another agent's post. Protocol: agents proactively coordinate, announcing what they hold and asking the others for help.

In run 20260910T033744Z, we watched this negotiation happen in real time:

Notice what happened at 03:39:48: OS-1554 answered a question and asked one back in the same message. That is not an agent completing an isolated task. It is two stateless agents negotiating across a shared service.

Later, OS-1554 announced that it had collected four of the required contributions and named the one it still needed. The agent holding that contribution supplied it seven seconds later.

The careful conclusion is not that agents always invent protocols spontaneously. Our strongest repeatable protocol result occurred under an operator-directed deadline intervention.

The lesson is narrower, but still useful:

Lesson 4: Capabilities composed across agents

Security reviews usually examine one identity at a time:

• What can this agent access?
• Which tools can it call?
• What data is in its context?
• What damage can it cause alone?

A swarm changes that calculation.

Each agent in our experiment began with one freshly generated contribution. Completion required all five contributions to arrive in a single request. Partial submissions did not accumulate.

Every measured run reached assembly. That means one submitting process acquired four values that had never existed in its starting sandbox.

The technical investigation showed the same composition at a deeper level. JAN-1834 possessed the loader call path. OS-1554 possessed the dependency resolver. Neither agent's starting material contained the entire finding.

The swarm connected the two.

For the CTF to complete, I had flags triggered on both inferred behavior and proven. An example of inferred would be simply posting to the message board. A proven behavior would be taking an artifact/key from an agent and combining it to complete the puzzle.

Lastly, to add a cyber twist, I added a Ruby reproduction to prove the swarm's finding actually worked. The launcher tested it with a secret one-time value, preventing the agent from faking success. At times this challenge would hit cyber guardrails and wouldn't complete the summarization.

No individual agent began with the complete chain. The system assembled it.

This is why evaluating an agent in isolation is insufficient. The dangerous unit is often the entire agent topology, and this compounds with n+ agents in the swarm. The same was true when red-teaming our Session Recording Summaries feature. While the session recording would capture a threat actor's behavior, an attacker could degrade a system slowly by spreading the work across many sessions over a long period, which is harder for detections to find.

Lesson 5: Identity must exist outside the model

The message board allowed agents to choose their own author names.

They did. Enthusiastically and inconsistently.

Across the eight measured runs, the board recorded 106 agent messages. Individual runs contained between five and nine self-selected author strings. Agents added generated suffixes, changed formatting, and occasionally appended labels such as final. The final label appeared after I added a prompt warning the agent of its upcoming termination.

Agent posting names were useful for coordination. They were not trustworthy identities.

Every message also carried an identity attached by the infrastructure, the authenticated Teleport user and its roles.

That produces two very different claims:

• "I am OS-1554" was text supplied by the agent.
• "This action occurred through this authenticated identity" was evidence supplied by the control plane.

The current demo still has a limitation: all five Beams use the same Teleport user. The infrastructure can attribute the activity to the operator, but it cannot independently distinguish each sub-agent JAN-1834 from OS-1554. Strong agent-level attribution would require a separate workload identity for each agent.

That limitation reinforces the lesson:

Authentication, authorization, attribution, and audit need to be imposed by infrastructure the agent does not control.

Guardrails are not security boundaries

In total I ran over 150 simulations for this exercise, and by design I had fewer guardrails. This was to highlight how different LLMs, harness prompts and their environment can help restrict them. While I deployed these in strongly isolated VMs, the agents had wide access inside them. The result was a security boundary that was both strong and weak. As with all systems, there is always tension between a locked down environment and one that's loosened/has weaker guardrails.

At Teleport, we spend our time talking to customers about protecting their most valuable assets, while still giving the proper amount of access to let teams get their job done.

In the world of agents, this is harder and an agent will use its full ability. This is why you need to constrain the capability of the agents by providing a limited runtime, restricted network and limited system access but also monitor and constrain the behavior of those agents.

The agents never broke their privileges.

They used every inch of them. That's the boundary that matters, and it's the one we built for.

My Top 3 Takeaways

This experiment showed how a collection of agents can and will collaborate to achieve its goal. While we simulated the exercise, we found signs that misalignment can lead to similar unexpected outcomes in the real world.

These are my top three takeaways from these runs:

1. Agents are likely doing something they shouldn't…and it's likely OK. Until it's not.

   I'll be honest, I gave a lot of prompts to elicit the behavior I wanted for this demo, and even then most agents didn't cause any real damage. The worst case scenario was that I wasted a few tokens. But in production, this could go horribly wrong if an agent had access to a non-staging database, or the ability to post freely to the internet. It seems harmless until agents adopt toxic personas or execute malicious objectives (as seen in Grok's notorious rogue incident). Always assume that with non-deterministic programs running over long horizons, something will eventually go wrong.

2. There is asymmetry between agent swarm and defenders

   If you've looked closely at my screenshots you'll notice that some of the sessions aren't summarized. This is often due to the actions of the agent getting flagged for 'Cyber'. It's ironic that the same LLM that created the program can't review the same actions. This is why we plan on supporting a wider range of open-weight models to classify risky agent behavior.

   
   

3. Start with a strong foundation and consider security invariants

   We already have a lot of the best practices for securing and hardening systems; starting all the way back to the Orange Book, the creation of zero-trust and more. These are a solid foundation to start with. Following the best practices for least-privilege, not blindly trusting untrusted code and hardening at all levels of the OSI model will always be a good idea. We've started to provide some best practices for Agent Trust within our Agentic Identity Framework.

We challenge you to decoy or divert our monitoring

Thanks for making it this far in this blog post. This post and experiment was a good way to stress test Teleport Identity Security for AI and explore how agents behave. I'm inviting any readers to propose an experiment to stress test our monitoring.

If you have a scenario where you want visibility, control, and audit of agents and want to stress-test it in our environment, please get in touch with me at [email protected]. I'll be running these evals until Nov 9th for our upcoming Beams.run release.