Independent PPO (IPPO) — Cooperative MARL Tutorial

When no one planned it.

Nobody told the agents to form a ring around the landmark.

There were no instructions. No coordination protocol. No central controller deciding who goes where. Each agent had one simple goal: get close to the landmark. And yet, when you watch the swarm at the end of training, they distribute themselves evenly — a perfect ring, every agent equidistant, coverage maximised.

Nobody planned this. It happened.

This is emergence — complex group behaviour arising from simple individual rules — and it's one of the most striking things you'll ever see come out of a training loop.

What the swarm experiment actually does

Six agents share a world. There's a landmark — a fixed point in space. Each agent gets a reward based on how close the group is to covering the landmark from all angles.

No agent is told where to stand. No agent knows where the others are planning to go. Each agent only knows its own position, the landmark's position, and the positions of the other agents within its observation radius.

The agents run IPPO — independent PPO, one policy per agent. They share no parameters, no communication channel, no explicit coordination.

And yet.

Early training: agents pile onto the landmark from one direction.
               Reward is high for the cluster — but low for total coverage.

Mid training:  some agents start moving to uncovered sides.
               They've learned that spreading out increases their individual reward.

Late training: a stable ring forms. Each agent occupies a roughly equidistant position.
               Any agent that moves toward a cluster reduces the global coverage reward.
               The ring is a Nash equilibrium.

The ring is a Nash equilibrium: no individual agent can improve its reward by moving. The cooperation is stable because deviation is individually costly.

Why this feels remarkable

Coordination in human groups is hard. It requires communication, negotiation, trust. Ants coordinate to build colonies. Birds coordinate to flock. Bees coordinate to swarm. But they have millions of years of evolutionary pressure shaping their behaviour.

The swarm agents learn this in a few thousand episodes. From scratch. With nothing but a reward signal and a gradient.

What's actually happening is this: the individual reward structure encodes the coordination problem. Each agent maximising its own reward is equivalent — in this environment — to the group solving the coordination problem. When the reward is designed well, individual optimisation is collective optimisation.

This is a deep idea. And it's why multi-agent RL is relevant far beyond robotics — it's a framework for thinking about how to design incentive systems that produce the collective outcomes you want.

The mathematics of emergence

Emergence in multi-agent systems has a formal characterisation: the joint behaviour cannot be predicted by studying any individual agent in isolation.

For $n$ agents, the joint policy space is:

Π = π_{1} \times π_{2} \times \dots \times π_{n}

The emergent behaviour lives in $Π$ — it's a property of the combination, not any individual $π_{i}$ . You could study each agent's policy forever and never predict the ring formation, because the ring only exists in the interaction between agents.

Three conditions for emergence in MARL:

Local interactions. Agents only observe and affect nearby agents/states. If every agent can see and control everything, there's no emergent structure — just a complicated single-agent problem.
Shared or aligned reward structure. The reward signal must make collective coordination individually rational. If agents can do well by defecting, they will.
Sufficient training time. Emergent structures often appear abruptly, after a period of apparent randomness. This phase transition is characteristic of complex adaptive systems — from ant colonies to financial markets.

The ring formation in our swarm satisfies all three. Each agent sees only nearby agents (local), the reward favours coverage (aligned), and the ring appears suddenly after episode ~800 (phase transition).

The connection to real swarms

Real swarm robotics uses remarkably similar principles. Search-and-rescue drones that distribute across an area without central coordination. Warehouse robots (exactly what we saw in the previous chapter) that develop informal traffic rules. Autonomous vehicles that negotiate intersections without a traffic controller.

The algorithms aren't identical to what we're running. But the philosophy is the same: design local reward signals. Let coordination emerge. Don't try to solve the global coordination problem centrally — it doesn't scale.

What you'll notice in the demo

Open the Swarm Coordinator ↗ — 6 agents covering a 2D landmark.

Three things to watch:

The transition from cluster to ring. There's usually a visible phase transition — a moment when the agents suddenly spread out. Watch the reward jump at that moment.
Stability under perturbation. Click on an agent and drag it away from its position mid-episode. Watch it navigate back to its equilibrium spot — not to where it started, but to the gap in the ring that's now uncovered.
Odd numbers. Run the experiment with 5 agents instead of 6. Watch the ring become slightly asymmetric — and watch agents compete over the "best" positions near the landmark. The equilibrium is less stable with odd numbers.

Try it yourself

Experiment 1 — Multiple landmarks. Add a second landmark at the opposite end of the world. With 6 agents and 2 landmarks, watch whether they split 3-3 or pile onto one. The split depends on initial positions. Run it five times with different random seeds — notice the variance.

Experiment 2 — Reduce observation radius. If agents can only see nearby agents, they can't coordinate globally. The ring breaks into small local clusters. Collective intelligence requires enough shared information to allow coordination.

Experiment 3 — Competitive mode. Give each agent a reward based on its distance to the landmark, ignoring others. Watch the ring collapse instantly. Every agent tries to get as close as possible. Pushing and clustering, not spreading. Same algorithm, same environment, opposite outcome.

All the algorithms so far are model-free: they learn directly from experience, with no internal model of how the world works. What if we gave the agent the ability to imagine what would happen before it acts?

Independent PPO (IPPO) — Cooperative MARL Tutorial ​

What the swarm experiment actually does ​

Why this feels remarkable ​

The mathematics of emergence ​

The connection to real swarms ​

What you'll notice in the demo ​

Try it yourself ​

Next ​