Consider training a coding agent purely on the basis of pairwise preferences between its own rollouts—that is, with no access to test outcomes, no absolute score, and no signal other than which of two final programs a judge prefers. We have this relative-judge setting whenever verifiable rewards are unavailable.
One solution is simply to substitute the judge’s win probability for the test-based reward in an otherwise unchanged policy-gradient recipe; but such a substitution presumes that what the judge prefers is what the tests verify.
So, to examine this, we train two arms on Web-Bench with Kimi K3.1 (rank-32 LoRA, learning rate 5e-5, 8 groups of 8 rollouts per update), and we change the reward. We'll have two recipes/arms.
- Arm A is rewarded by the fraction of hidden tests passed.
- Arm B is rewarded by a Bradley-Terry win probability within each group, fitted to 24 ballots from a judge. Since this reward has mean 0.5 by construction, we log the tests’ reward in both arms, and additionally score validation rollouts against two frozen rollouts of the base model.
The two rewards move the policy in different directions
How can we interpret this? First, within a group, the rank correlation between the trained reward and the tokens written is +0.12 to +0.32 for the judge and +0.02 to +0.07 for the tests; since the update credits every token of a rollout with positive advantage, a reward correlated with length is—in effect—a gradient on length. Second, we find that our code judg increases our reasoning share a significant amount. In addition, we see higher policy drift and KL. This is in a setting where our judge sees only the text and is constructed to judge which code quality is better. In the spirit of reporting negative findings, a relative judge is not an immediate substitute to verifiable rewards.