# #4 The exact learning recipe is not always so important in long-horizon async RL

Author: Paulius Rauba
Canonical URL: https://www.pauliusrauba.com/notes/4-the-exact-learning-recipe-is-not-always-so-important-in-long-horizon-async-rl
Published: 2026-11-17T16:54:30.922Z
Updated: 2026-11-17T17:14:17.029Z

It's tempting to think that nailing the exact learning recipe is very important for online RL. While true in principle, there is a space of recipes that all give very similar solutions, even in long-horizon rollouts.

Consider the following setting: an agentic web-bench development task where an agent has to solve coding tasks for designing websites. We are doing online GRPO. We will have a control run and a few target runs to evaluate the differences of different interventions.

# 1. Our control run
**Setup**. Our control run will be three seeds of a standard recipe: Qwen-35B, LoRA rank 32, fully asnc, 8 tasks with 8 rollouts each, policy staleness 4, multi-harness training (3 harnesses sampled at random), advantage without standard deviation, standard reinforce with score centering on top-128 candidates, importance weight cut at 2 with CISPO (gradients are those at the cut-off), routing replay, adamW on LoRA, learning rate 5e-5, beta2 0.98, gradient clip 1.0, no weight decay, KL or entropy, with 80 updates each. We also have a 65K token window. The score is how many automated browser tests our model passes, from 0 (none) to 1 (all of them).

This is a fairly standard modern setup. The question is: (a) how much variability within a single rollout do we have and (b) how much does a control win or lose against this recipe, where a control is any change to the learning recipe?

First, let's establish the variability. We run three seeds, where a seed changes: (a) the LLM sampler; (b) the trainer; (c) the task order. After 80 updates, these are three rollouts

![image.png](https://www.pauliusrauba.com/notes/files/7500fcdd-199b-4a64-9586-0187e015f0b3)

In all cases, the three seeds improved our learning outcomes with 18.9, 28.4, and 12.1


# 2. Ablating components
It's very easy to ablate components en masse within an A/B setup. I've asked an agent to conduct thorough research on best practices, identify the top 30 design choices that other labs are implementing and implement each as a flag, focusing on ones which are useful for (and have been used in) similar long-horizon multi-turn agentic setups. Here are all the 30 ablations, some with multiple seeds, all colored in grey

![image.png](https://www.pauliusrauba.com/notes/files/6528a4a8-7e71-4a06-9d54-543c37184150)

Here are the validation results on edge cases (which are chosen problems which discriminate model performance better than an iid split)

![image.png](https://www.pauliusrauba.com/notes/files/47f79f46-e4fd-41d4-9fae-e883fd6cd009)

Here are some top candidates among those that are evaluated. 

NB: This is not an entirely fair way to represent this as many ablations do not aim to increase reward per-se (e.g. group-relative length penalty makes rollouts shorter or IPO masking only matters if we have big policy staleness which we do not), so in many cases we wouldn’t expect to see a difference here anyway. So, looking only at reward is shortsighted as we don’t look at the behavior it induces. Nonetheless, mildly useful too.

| Rank | Run | Params or change against the control | Mean gain | Seeds | Range of gains |
|---|---|---|---|---|---|
| | **Control** | REINFORCE + score centering, importance weight cut at 2, group-mean advantage without std, token-mean loss, lr 5e-5, LoRA rank 32, 8 x 8 sessions per update, lag 4, three harnesses | 0 (reference) | 3 | Its own seeds are −7.7 to +8.6 around their mean |
| 1 | IPO mask | No gradient for a token whose probability moved more than 0.3 | +1.6 | 1 | +1.6 |
| 2 | Prompt-mean loss | Loss mean for each prompt group | +0.1 | 1 | +0.1 |
| 3 | Group-relative length penalty | Reward minus a penalty for a session that is longer than its group | −1.2 | 1 | −1.2 |
| 4 | CIS | Importance weight cap that is tighter for a confident token | −1.6 | 1 | −1.6 |
| 5 | CISPO | Detached importance weight cut at 2, no score centering | −1.7 | 2 | −10.8 to +7.5 |
| 6 | Sequence mask | No gradient for a sequence with negative advantage that is far from the rollout policy | −2.0 | 2 | −7.7 to +3.8 |
| 7 | Rising cap | Token window 40,960 for updates 0-39, then 61,440 | −2.3 | 3 | −18.3 to +7.6 |
| 8 | Std-normalized advantage | Advantage divided by the reward std of the group | −3.4 | 1 | −3.4 |
| 9 | Linear length penalty | Reward minus a deduction that grows with the session length | −3.5 | 2 | −11.1 to +4.1 |
| 10 | Log-ratio penalty | Adds 0.001 x (log ratio)² for each token | −3.6 | 2 | −12.4 to +5.3 |

The key takeaway is that among the design space of possible algorithmic solutions, there are many which result in stable rollouts with within-noise variability, which is why focus on data, environments, in addition to learning algorithm design, can be so powerful.

