Events as Triggers for Behavioral Diversity in MARL

Decouple behavior from agent identity: agents draw behaviors from a shared manifold, and events decide when they switch.

Abstract

Effective multi-agent cooperation requires agents to adopt diverse behaviors as task conditions evolve—and to do so at the right moment. Yet, current Multi-Agent Reinforcement Learning (MARL) frameworks that facilitate this diversity are still limited by the fact that they bind fixed behaviors to fixed agent identities. Consequently, they are ill-equipped for tasks where agents need to take on different roles at very specific moments in time. We argue that, to define these behavioral transitions, the missing ingredient is events. Events are changes in the state of the system that induce qualitative changes in the task. Based on this view, we introduce a framework that decouples agent identity from behavior, capturing a continuous manifold from which agents instantiate their behaviors in response to events. This framework is based on two elements. First, to build an expressive behavior manifold, we introduce Neural Manifold Diversity (NMD), a formal distance metric that remains well-defined when behaviors are transient and agent-agnostic. Second, we use an event-based hypernetwork that generates Low-Rank Adaptation (LoRA) modules over a shared team policy, enabling on-the-fly agent-policy reconfiguration in response to events. We prove that this construction ensures that diversity does not interfere with reward maximization by design. Empirical results demonstrate that our framework outperforms established baselines across benchmarks while exhibiting zero-shot generalization, and being the only evaluated method that solves tasks requiring sequential behavior reassignment.

Figure 1. (a) Timestep-based, episode-based and event-based behavior shifts on a time axis. (b) The NMD metric as the average Wasserstein distance between behaviors on the manifold. (c) A hypernetwork with LoRA maps behaviors from the manifold to agents, with a behavior shift at an event. (d) t-SNE of the Pressure Plate behavior manifold in three phases, separated by the events 'door opens' and 'no agent left', above rollouts of the three agents. — **Proposed framework.** (a) Behavioral transitions are triggered by task events rather than a fixed timestep or episode schedule. (b) NMD is a diversity metric defined directly on the behavior manifold, independent of which agent executes which behavior. (c) An event-driven hypernetwork generates LoRA modules over a shared policy to allocate behaviors from the manifold. (d) t-SNE projection of the behavior manifold across 128 replications of Pressure Plate, with agent trajectories for one rollout.

Method

The method has three parts: Neural Manifold Diversity (NMD), a metric that measures how diverse the behavior manifold is; an event-driven hypernetwork that draws behaviors from the manifold and assigns them to agents whenever an event fires; and a diversity control rule that keeps the manifold's diversity at a chosen target.

Neural Manifold Diversity

NMD is the expected 2-Wasserstein distance between the action distributions of two behaviors drawn from the behavior manifold $\mathcal{B}$ , averaged over observations. Its pairwise term is a proper distance on behaviors (a pseudometric, and a metric whenever policies are distinguishable). In practice we estimate it over the $B$ behaviors present in a batch of parallel environments:

\widehat{\mathrm{NMD}}\big(\{\pi_m\}_{m=1}^{B}\big) = \frac{2}{B(B-1)\,|\mathcal{O}_{\mathrm{ep}}|}\sum_{m=1}^{B}\sum_{n=m+1}^{B}\sum_{o\in\mathcal{O}_{\mathrm{ep}}} W_2\big(\pi_m(o),\,\pi_n(o)\big)

\begin{gathered}\widehat{\mathrm{NMD}}\big(\{\pi_m\}_{m=1}^{B}\big) = \frac{2}{B(B-1)\,|\mathcal{O}_{\mathrm{ep}}|}\\[4pt] \times\sum_{m<n}\ \sum_{o\in\mathcal{O}_{\mathrm{ep}}} W_2\big(\pi_m(o),\,\pi_n(o)\big)\end{gathered}

The kernel is the same as in System Neural Diversity (SND). The difference is what it is applied to: SND compares policies indexed by agent, while NMD describes the manifold itself, regardless of which agent runs which behavior, how many agents there are, or in what order.

Event-driven hypernetwork with LoRA

A transformer hypernetwork $g_\theta$ reads one token per agent (its local observation $o_{i,t}$ , the event encoding $e_t$ , and the target diversity $\mathrm{NMD}_{\mathrm{des}}$ ) and emits a low-rank adapter $(C_m, D_m)$ for the last layer of a policy shared by the whole team:

z_m(o_t) = W_{\mathrm{shared}}\,\phi(o_t) + \alpha\, D_m C_m\, \phi(o_t)

The hypernetwork is queried once at the start and then only when an event fires; between events the adapter stays fixed. This avoids both the brittleness of generating behaviors once per episode and the cost and instability of regenerating them every step. Several agents can receive the same adapter, which is why behaviors are indexed by $m$ rather than by agent.

Diversity control by projection

Each behavior is a shared component plus a deviation $u_m(o_t) = D_m C_m\,\phi(o_t)$ . Because the deviation enters linearly, rescaling it by $\alpha$ sets the realized diversity to the target exactly:

\alpha = \frac{\mathrm{NMD}_{\mathrm{des}}}{\widehat{\mathrm{NMD}}\big(\{u_m(o_t)\}_{m=1}^{B}\big)}

This extends Diversity Control (DiCo) from a fixed set of agent indices to the time-varying set of behaviors on the manifold. It also has a useful consequence for learning:

Theorem. The gradient of the expected return with respect to each deviation is a projection of the reward gradient:

\nabla_{u_m} R = \alpha\, P_{u_m} \big(\nabla_{z_m} R\big)^{\top}, \qquad P_{u_m} = I - \frac{u_m \otimes \nabla_{u_m}\widehat{\mathrm{NMD}}}{\widehat{\mathrm{NMD}}}

\begin{gathered}\nabla_{u_m} R = \alpha\, P_{u_m} \big(\nabla_{z_m} R\big)^{\top},\\[6pt] P_{u_m} = I - \frac{u_m \otimes \nabla_{u_m}\widehat{\mathrm{NMD}}}{\widehat{\mathrm{NMD}}}\end{gathered}

Any component of the reward gradient that would change NMD is filtered out before the adapter is updated. Reward maximization and diversity preservation are aligned by construction, with no auxiliary losses or trust regions.

Tasks and baselines

All tasks run in the VMAS simulator. Four are established benchmarks for behavioral diversity; two are new tasks built to require event-driven behavior. We compare against four baselines: HyperMARL (Tessera et al., 2025), a hypernetwork that generates each agent's policy weights from its identity once per episode; CASH (Fu et al., 2025), a hypernetwork that regenerates them at every step from the agent's capabilities, or its identity when capabilities are not available; DiCo (Bettini et al., 2024), which holds diversity between fixed agent indices at a target value (we report the best target from a grid search); and parameter sharing (Gupta et al., 2017), where all agents share one policy. All results are over 5 seeds.

Navigation
Agents explore with LiDAR to find their goals under partial observability.
Dispersion
All agents start at the center and must split up to reach different goals.
Reverse Transport
Agents inside a heavy package push it to the goal together.
Football
Three agents switch between defending and counter-attacking against a scripted opponent.
Pressure Plate New
Agents take turns holding two plates to get the whole team through a locked door.
Wind Flocking New
A larger agent shields a smaller one from a north-to-south wind.

Results

Q1Does it beat established baselines?

Our framework ranks first on all four benchmarks and reaches at least 95% completion on every task. The gap grows with task complexity. On Reverse Transport most methods succeed and the difference shows in episode length, where ours finishes earliest. On Dispersion, CASH drops to roughly 50% and DiCo to roughly 80%. On Football, every baseline collapses to near-zero completion while ours stays close to 100%.

Box plots of final completion rate for Ours, HyperMARL, CASH, DiCo and parameter sharing on Reverse Transport, Navigation, Dispersion and Football. Ours is near 100% on all four; on Football all baselines are near 0%. — **Completion rate against baselines**, 5 seeds. HM: HyperMARL, CH: CASH, DC: DiCo, PS: parameter sharing. Average reward and episode length are reported in the paper.

What each component contributes

We ablate one component at a time on Navigation, Dispersion and Reverse Transport. No single component explains the gains. The full method has the highest median completion on every task (tied on Reverse Transport) and by far the lowest seed-to-seed spread. The variants do not degrade gracefully: they solve the task on some seeds and fail on others.

Component ablation: completion rate in percent, median plus or minus standard deviation over 5 seeds.
Variant	Navigation	Dispersion	Reverse Transport
Ours (full)	96.02 ± 0.77	98.04 ± 0.87	100.00 ± 0.04
(A) No event re-query (single query at t = 0)	87.58 ± 3.18	85.55 ± 4.62	100.00 ± 23.34
(B) NMD replaced by SND	84.90 ± 14.82	89.57 ± 5.65	99.61 ± 30.48
(C) Event-conditioned, no diversity control (α = 1)	84.22 ± 8.93	90.64 ± 2.82	91.56 ± 37.01
(D) Full MLP instead of LoRA	86.74 ± 3.88	88.67 ± 4.28	61.41 ± 34.15
(E) LoRA applied to all policy layers	87.59 ± 3.15	92.52 ± 4.32	100.00 ± 2.24

Completion rate [%], median ± standard deviation over 5 seeds.

Q2Does it generalize zero-shot?

Without retraining, performance holds as the number of agents, their physical capabilities (pushing force, LiDAR range, speed) and the target diversity change. Changing the team size simply means sampling more or fewer points from the learned manifold. Completion stays stable across roughly two orders of magnitude of target NMD, and each task has its own range of good values.

Grid of line plots of completion rate versus number of agents, capability and target NMD for four tasks. Ours stays high inside and outside the training range; baselines degrade or stagnate. — **Zero-shot generalization**, 5 seeds. White: training range; gray: zero-shot. Rows: number of agents, capability (force, LiDAR range, speed), target NMD. Green: ours, yellow: CASH, blue: DiCo, purple: parameter sharing. HyperMARL is excluded because its architecture cannot change team size.

Q3Can it solve tasks that need sequential role changes?

Pressure Plate can only be solved by reassigning behaviors in sequence:

One agent holds the first plate so the others can cross the door.
Those agents find and hold a second plate on the other side, so the first agent can cross too.
All agents move to the goal.

Ours is the only method that solves it, with 98.4% median completion. The same model with a single query at the start fails: each agent keeps one behavior all episode, and the last agent stays stuck behind the door. Because the adapter computed at the last event is held fixed, it acts as a learned, event-indexed memory, and the sequential strategy emerges without a recurrent network.

Box plot of Pressure Plate completion rate: Ours near 100%; Ours SQ, HyperMARL, CASH, DiCo and parameter sharing all at 0%. — **Pressure Plate completion rate**, 5 seeds. SQ: ours with a single query at t = 0.

Q4Does it handle event sequences it never saw?

We remove an agent mid-episode, which never happens during training: in Football at a random time, and in Pressure Plate the agent that first reaches the second plate. The team recovers and redistributes roles. Completion drops, but stays far above what any baseline reaches on the unperturbed task.

Agent removal: completion rate in percent, median with interquartile offsets.
Task	Agent removed	Completion [%]
Pressure Plate	No	98.4+1.4−0.8
Pressure Plate	Yes	81.2+0.9−0.2
Football	No	98.2+1.0−0.8
Football	Yes	88.6+6.6−0.7

Median, with offsets to the upper and lower quartile.

In Wind Flocking we change the target diversity mid-episode, from 0.1 to 0.2 to 0.3 and then down to 0.05. The formation spreads out as the target rises and tightens when it drops, without retraining. Since the target is an input to the hypernetwork, it acts as a direct, interpretable control on team behavior.

Wind Flocking at step 20 with target NMD 0.1: the two agents fly close together. — **Wind Flocking** at steps 20, 70, 120 and 170, as the target NMD changes within one episode. The arrow shows the wind direction.

Step 70 with target NMD 0.2: the agents are further apart. — **Wind Flocking** at steps 20, 70, 120 and 170, as the target NMD changes within one episode. The arrow shows the wind direction.

Citation

@inproceedings{buechi2026events,
  title     = {Events as Triggers for Behavioral Diversity
               in Multi-Agent Reinforcement Learning},
  author    = {B{\"u}chi, Hannes and Flageat, Manon and
               Sebasti{\'a}n, Eduardo and Prorok, Amanda},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}

Acknowledgments

This work is supported by the European Research Council (ERC) Project 949940 (gAIa), by a Leverhulme Trust Research Project Grant, and by the EPSRC funded INFORMED-AI project EP/Y028732/1. We gratefully acknowledge their support.

Events as Triggers for Behavioral Diversity in Multi‑Agent Reinforcement Learning

When should an agent change what it does?