Skip to content
paulius.rauba
← Home
← All notes

#3 Notes on system performance of large-scale agentic models

· · 9 min read

1. Anatomy of a single rollout#

In online RL, we often care about throughput, i.e. the number of rollouts that finish per hour. Let's understand the anatomy of throughput and how it works in agentic systems.

There are four broad components of such an interacting system in async RL: (1) a rollout engine that generates tokens; (2) a trainer which trains the model; (3) a sandbox where commands are executed (e.g. tool calls); (4) a harness which is the CPU program that sends model's commands to the sandbox, sends the results back to the model, etc (typically <5% of all time for most rollouts).

Therefore, in agentic coding tasks, one rollout end-to-end will spend most of the time in an inference server (generating tokens) and a sandbox, executing commands. Here is a representative example of a single rollout and how it spends its time between an inference engine and sandbox:

image.png

You can see that most of the time is spent in the inference engine. In this case, there are around 19 sandbox interactions all of them have a fairly fixed time horizon (around 20-25 seconds). Here, a tool call is the time when the sandbox runs the commands and the rollout waits for the results. The time when the model writes the commands (i.e. generates the tokens) is part of the model call (blue).

2. Anatomy of multiple rollouts in async RL#

Now, we do not generate a single rollout at a time since we batch these requests. Ideally, we want to have as many concurrent rollouts being generated at the same time. Let's call these rollouts in flight -- the number of rollouts that are being generated at the same tine in all states: model call, tool call, harness, grading.

For an illustrative example, suppose we are training K3.1. with 2.7T params, one trainer server with 3 nodes and 2 inference servers with 2 nodes each (4 nodes in total for inference). Let's now suppose we set a value of 96 rollouts in flight, i.e. at any given point we're generating at most 96 rollouts. Recall that our inference servers have a batch size, i.e. rollouts that are in a model call at a particular moment, where the engine makes one token for each of them at each step. In our case, the maximum batch size (also called max number of running requests) is 64. In practice, we will have an effective batch size of 31. A few questions might appear here:

  • Why not fit the engine to 64? We could do this, but we would need to have more rollouts in flight. Hoewver, this has two costs: (a) each decode step becomes slower when the batch is larger; and (b) the samples might become older when they reach the trainer.

  • Is a variable size on the inference engine easy to implement? Yes, the inference engine does this automatically, called continuous batching. Therefore, at each step, we can have a variable number of requests in each decode step.

  • Do we lose cache because of this? No, this is because we have cache for each rollout's own prompt prefix.

So, the 2 engines hold about 62 rollouts in a model call at a time; the other 34 of the 96 are in a sandbox, in the harness, in setup or grading, or their slot is empty.

image.png

Notice that the inference engines hold on average around 31 requests (rollouts) at a time, there's always some calls in the sandbox and the harness is always doing some work in the interim too. Also notice:

  • Because we are operating in async RL, 96 rollouts do not start and end together. A rollout ends about every 20 seconds, so at each moment there are some rollouts that have just started or ended. This transition period is why there are empty slots.
  • Each rollout has a setup start time and grading at the end (around 25 seconds for our setup), so if a model finishes early, it moves to grading.

After the slot becomes empty, there's a time lag until a new rollout fills in. This is because we have 12 tasks, each with 8 rollouts per task (for GRPO). Therefore, a new problem needs 8 free slots which start together. If there is an empty slow, we cannot start a new rollout because we need 8 empty slots.

3. How this relates to throughput#

Throughput is defined as: Rollouts in flighttime for one rollout\frac{\text{Rollouts in flight}}{\text{time for one rollout}}. In one of my experimental runs, this is 910.51≈178\frac{91}{0.51} \approx 178 rollouts per hour. Therefore, our throughput depends only on these two numbers.

  • If we have more inference engines, we can increase how many rollouts we have in flight assuming that these do not increase the time for one rollout (e.g. because sandbox capacity is limited)
  • If we scale number of requests on the same inference engine, then the time per rollout might increase and this might result in throughput not increasing as fast or even decreasing.

One way to build up intuition toward engine throughput is by reasoning about how quickly we generate tokens within each engine. First, the speed of one request depends on the number of concurrent requests in a model call (which in this case is equal to the batch size for that forward pass). In our implementation, this ranges from around 15 tok/s to around 5 tok/s.

image.png

However, as we have more concurrent requests, we can generate more tokens and the throughput of one engine can increase because the higher batch size with the slower tok/s results in a larger product than a smaller batch size and higher tok/s. Naturaly, if we have two engines, the throughput of two engines will be even higher if we have more requests.

image.png

4. How does this affect the trainer?#

As we're generating rollouts, the trainer is taking these rollouts and learning on them. The metric to think about is updates per hour, i.e. how many times are the weights updated. Given that our batch size is 64, this value -- updates per hour -- ends up being the smaller value of one of the two:

  • Supply of the rollouts divided by 64, i.e. 185/64 = 2.9 (updates per hour)
  • Trainer capacity: one update takes about 8 minutes, so the trainer can do around 60/8 ≈\approx 7 updates per hour.

In our runs, we end up doing around 2.8 updates per hour. This shows an under-utilized trainer, where the supply of rollouts is the limit.

image.png

Here, you see the trainer capacity as a dot line -- this is how quickly we could train a model and update it if the trainer were always full. However, because our trainer waits for rollouts a ong part of the time, it does fewer updates. The the supply, i.e. completed rollouts, are represented by the blue line. Notice that the updates start in batches of 64 rollouts. These rollouts therefore accummulate with the trainer being idle until there are 64 rollouts that fill up the trainer.

How could you improve this? Use more inference engines. If you use more engines, you can obtain more updates per hours (up to the trainer capacity). However, if your trainer becomes the limit, the rollouts might be waiting in the queue, become stale, then we would end up dropping them because they're stale and they would end up being not usable. The example on the left has hmuch more unsused trainer capacity; and the example on the right has fully used up the trainer capacity at the beginning. After about hour 5 the green line falls below the dashed line because each update became slower because the samples become longer.

image.png

A run is balanced when the rollout supply is near trainer capacity. What is the difference between a balanced run and an unbalanced run? The two runs come from very different workloads, so let's look at some key properties of this:

Run example 1 Run example 2
Rollout supply (rollouts per hour) 185 240
Trainer capacity (rollouts per hour) 461 351
Supply ÷ capacity 0.40 0.68
Result: trainer waits for rollouts 61% 28%

How to interpret this:

  • A run is balanced when the supply / capacity ratio is near 1. Below 1, the trainer iss waiting and above 1, a completed rollout is wiating in the queue. In run 1, the run is not balanced, as the rollout supply is not high enough. Run 2 is nearly balanced: supply is about 70% capacity. More engines would give some gains, but then the trainer becomes the limit.

In this case, supply comes from the number of engines and the rollouts each engine completes per hour. Capacity comes from the training batch divided by the trainer cycle (and trainer cycle is update time + weight-sending time which can add up if you're doing cross-cluster communication).

There are some more details behind this:

Add-on: the details behind the two rates

Run example 1 Run example 2
Behind the supply
Inference nodes 4 (2 engines) 12 to 22 (6 to 11 engines)
Rollouts in flight 96 288
Behind the capacity
Trainer nodes 3 16
Training batch 64 (8 tasks × 8 rollouts) 192 (12 tasks × 16 rollouts)
Time for one update 494 s 1,119 s
Time for one weight send 6 s 852 s
Trainer cycle (update + weight send) 8 minutes 33 minutes
Other ratios and checks
Inference nodes for each trainer node 1.3 about 1.1
Rollouts in flight ÷ training batch 1.5 1.5
Stale groups dropped 0 0

A few important notes from this table:

  • In both cases, we're not dropping any stale groups which is great because we're not losing any rollouts we've generated.
  • Notice that inference nodes for each trainer node are very similar in both cases (1.3 and 1.1 respectively), so the main difference is explained by other two properties: (a) more rollouts arrive in example 2 (240 against 185) and (b) each rollout costs the trainer more time.

5. How does this relate to the time it takes to do a full training run?#

A training run is the full end-to-end of generating rollouts + grading them + updating the model, etc., so these components interact.

image.png