Skip to content
paulius.rauba
← Home
← All notes

#4 The exact learning recipe is not always so important in long-horizon async RL

· · 4 min read

It's tempting to think that nailing the exact learning recipe is very important for online RL. While true in principle, there is a space of recipes that all give very similar solutions, even in long-horizon rollouts.

Consider the following setting: an agentic web-bench development task where an agent has to solve coding tasks for designing websites. We are doing online GRPO. We will have a control run and a few target runs to evaluate the differences of different interventions.

1. Our control run#

Setup. Our control run will be three seeds of a standard recipe: Qwen-35B, LoRA rank 32, fully asnc, 8 tasks with 8 rollouts each, policy staleness 4, multi-harness training (3 harnesses sampled at random), advantage without standard deviation, standard reinforce with score centering on top-128 candidates, importance weight cut at 2 with CISPO (gradients are those at the cut-off), routing replay, adamW on LoRA, learning rate 5e-5, beta2 0.98, gradient clip 1.0, no weight decay, KL or entropy, with 80 updates each. We also have a 65K token window. The score is how many automated browser tests our model passes, from 0 (none) to 1 (all of them).

This is a fairly standard modern setup. The question is: (a) how much variability within a single rollout do we have and (b) how much does a control win or lose against this recipe, where a control is any change to the learning recipe?

First, let's establish the variability. We run three seeds, where a seed changes: (a) the LLM sampler; (b) the trainer; (c) the task order. After 80 updates, these are three rollouts

image.png

In all cases, the three seeds improved our learning outcomes with 18.9, 28.4, and 12.1

2. Ablating components#

It's very easy to ablate components en masse within an A/B setup. I've asked an agent to conduct thorough research on best practices, identify the top 30 design choices that other labs are implementing and implement each as a flag, focusing on ones which are useful for (and have been used in) similar long-horizon multi-turn agentic setups. Here are all the 30 ablations, some with multiple seeds, all colored in grey

image.png

Here are the validation results on edge cases (which are chosen problems which discriminate model performance better than an iid split)

image.png

Here are some top candidates among those that are evaluated.

NB: This is not an entirely fair way to represent this as many ablations do not aim to increase reward per-se (e.g. group-relative length penalty makes rollouts shorter or IPO masking only matters if we have big policy staleness which we do not), so in many cases we wouldn’t expect to see a difference here anyway. So, looking only at reward is shortsighted as we don’t look at the behavior it induces. Nonetheless, mildly useful too.

Rank Run Params or change against the control Mean gain Seeds Range of gains
Control REINFORCE + score centering, importance weight cut at 2, group-mean advantage without std, token-mean loss, lr 5e-5, LoRA rank 32, 8 x 8 sessions per update, lag 4, three harnesses 0 (reference) 3 Its own seeds are −7.7 to +8.6 around their mean
1 IPO mask No gradient for a token whose probability moved more than 0.3 +1.6 1 +1.6
2 Prompt-mean loss Loss mean for each prompt group +0.1 1 +0.1
3 Group-relative length penalty Reward minus a penalty for a session that is longer than its group −1.2 1 −1.2
4 CIS Importance weight cap that is tighter for a confident token −1.6 1 −1.6
5 CISPO Detached importance weight cut at 2, no score centering −1.7 2 −10.8 to +7.5
6 Sequence mask No gradient for a sequence with negative advantage that is far from the rollout policy −2.0 2 −7.7 to +3.8
7 Rising cap Token window 40,960 for updates 0-39, then 61,440 −2.3 3 −18.3 to +7.6
8 Std-normalized advantage Advantage divided by the reward std of the group −3.4 1 −3.4
9 Linear length penalty Reward minus a deduction that grows with the session length −3.5 2 −11.1 to +4.1
10 Log-ratio penalty Adds 0.001 x (log ratio)² for each token −3.6 2 −12.4 to +5.3

The key takeaway is that among the design space of possible algorithmic solutions, there are many which result in stable rollouts with within-noise variability, which is why focus on data, environments, in addition to learning algorithm design, can be so powerful.