A team scaling reinforcement learning (RL) on a GPU cluster can hit bottlenecks that waste compute and slow down training. Rollout workers generate samples—sometimes as simple as a single response, sometimes as complex as repository edits and tool calls. The slowest rollout can hold up the next training step, and redundant computation or slow weight updates only make things worse.
NVIDIA Dynamo sits in the serving layer, between RL orchestration and inference workers, to address these issues. It isn't the RL framework itself, but the middleware that manages rollout generation, routing, and model weight distribution. Dynamo is already integrated with open-source RL stacks like verl, NeMo RL, Slime, and prime-rl.
Why RL rollouts become a bottleneck
Distributed RL workloads are uneven by nature. Some requests are simple; others are long chains of actions and tool invocations. If you just spread requests evenly, you still get stragglers—slow rollouts that block the next round of training. Recomputing the same prompt context for every attempt burns GPU time, and keeping all workers up to date with the latest model weights can create further delays.
Three main constraints define the bottleneck:
- Rollout completion time: The slowest sample delays the batch.
- Repeated context: Redundant prompt computation eats GPU cycles.
- Weight freshness: Stale weights mean wasted samples or algorithm drift.
KV-aware routing and prompt cache reuse
Dynamo's router aims to balance two goals: reuse cached prompt context (KV cache) and avoid queueing too many requests on a single worker. If you always send similar requests to the same worker, you save on computation but risk creating a bottleneck. Dynamo routes requests to workers that can reuse cache and have spare capacity, so you get earlier rollout completion for the batch.




