Where the gradient goes to die

One replaced inference engine added ten minutes to every weight push after it, and the only trace was a warning line in 1.1 GB of logs. At 8,192 sandboxes, the next training step took 25 minutes to start while one process collected routing data. Thousands of pods asked the image registry for their images at once, failed, and retried.

RL360 is the RL codebase we used to post-train IFM's K2 Horizon models. Most of it is Miles, Megatron-LM, SGLang, SMG and Harbor; the rest is the code we had to write to make them train coding agents together. I lead RL360 and led coding-agent post-training for K2 Horizon. This post starts with what broke when we scaled it, then goes through eight places where that code decides whether a step's gradient can be trusted.

The three stories A replaced engine slowed every weight push R3 held up every step Sandboxes were the death of us

What RL360 is

A training step goes like this:

1. Miles samples a batch of task groups and opens a session on a session server for each attempt.
2. The agent function POSTs the task and the session's OpenAI-compatible URL to the Harbor server's /run. Harbor starts a sandbox (Daytona or Docker), runs the agent, then the task's verifier, and returns the reward and an exit status.
3. Every model call from the agent goes through the session server, which records token IDs and log-probs, and then through SMG to an SGLang engine.
4. Miles collects one merged sample per session, attaches Harbor's reward, and filters groups.
5. Megatron trains on what is left and pushes weights to SGLang.

What broke at scale

The public run trains on 256 attempts per step. Production ran up to 8,192 sandboxes at once. Routing data for replay (R3) and the sandboxes themselves cost us the most, but the weight push goes first, because it shows how quietly this kind of thing happens.

The production numbers below come from one recent 375B run: 25 hours and 17 training steps on a mix of coding and other agent tasks. This is what a step of it looked like, with the R3 and sandbox fixes already in:

A replaced engine slowed every weight push

The weight push took about a minute for the first 13 steps of the 375B run and 9 to 11 minutes for the last four.

Generation stops during a push, so the engines sat idle for about 11% of each of those steps. GPU utilization, memory, temperatures and the engine count looked the same before and after. The push was doing more work: each engine received 41 update calls per push before the change and 1,432 after.

Both counts come from how the weights travel. The broadcast sends 512 MB at a time, and 375B at BF16 is about 750 GB, nearly all of it experts. Our internal branch broadcasts only the non-expert weights, about 20 GB in 40 rounds. An expert lives on one GPU of each engine, so the trainer ranks holding experts send them straight to those GPUs, which takes one more call per engine and about 10 seconds.

That plan is built for one fixed set of engines. When the set changed, the code retired the plan for the rest of the job and went back to the path the public release's recipe uses: gather the experts and broadcast all of them to every GPU, 512 MB at a time.

The swap that did it was the uneventful one. An engine died overnight, and its replacement joined at the next push. That push was rejected and retried, by which time a second engine was down, and the retry path throws the plan away and builds a new one, so the direct path survived.

One step later the second engine's replacement joined without an error, and that is the case that retired the plan. The log records it as one warning line in 1.1 GB of output, and no metric says which path a push took.

The job ended on the push after those four, in the same code. Another replacement joined, a dead engine never confirmed its cleanup, and the code refused to retry because it couldn't confirm the old receivers were gone.

The fix was in before the next long run: a changed engine set now retires the old plan and builds a new one. A later run hit it after a large engine swap, and the next four pushes took 117, 45, 39 and 36 seconds. The release has no direct path for experts, and its recipe uses the broadcast, so at 375B every push takes the long way.

R3 held up every step

At 8,192 trajectories, the next step took forever to start. After the last rollout of a step finished, collecting its R3 routing data took about another 25 minutes before training could begin.

All of it went through one rollout manager.

Rollout routing replay (R3) ships every token's expert choices from inference to training. K2-Horizon-375B has 61 layers with 8 experts per token (the first three layers are dense, but the format keeps a row for every layer), stored as int32: 1,952 bytes a token, 2.6 KB once base64-encoded into JSON. A 58K-token trajectory, the public run's typical length, carries 151 MB. Every response carries routing for the whole conversation so far, old tokens included.

The bytes hurt less than the CPU it takes to serialize and parse them. The session server returns /merged through FastAPI's default response, which serializes with the standard-library json, and the rollout manager parses it with the standard-library json that httpx uses. The alternative on both ends is orjson, which RL360 already pins.

Parsing alone is about the right size. At 0.22 s each, one process working through 8,192 samples of that length takes 30 minutes, close to the 25 we saw.

That is a scale check, not a profile: we never split those minutes between parsing, serialization and the network, and real trajectories vary in length. But it points at the design. One event loop collecting every trajectory at the step boundary doesn't work at this scale.

The release changes the collection. It runs 256 session servers as Ray actors spread across nodes (the default is one), merges each trajectory on the server that holds it, and sends the rollout manager one sample with only the last turn's routing, which already covers every token. That spreads the serialization 256 ways and keeps the earlier turns' copies off the rollout manager.

In the 375B run, preparing R3 takes the trainer 2 seconds a step. The trainer still waits about 22 minutes of each 84-minute step, but that wait looks like generation. In a typical two-minute window 0.05% of the rollout engines were idle, and at any moment about 70% of the trajectories in flight were waiting on the model.

The largest cost left is per turn. Every turn's response still carries routing for the whole conversation from SGLang through SMG to the session server, which parses it. Then it goes on to the agent, which never uses it: the agent function asks for routing on every call, and the session server hands back the engine's body as is. On the public run's shape, 61 turns whose prompts add up to 2.6M tokens, that is about 6.8 GB of routing per trajectory, parsed twice.

Stripping routing from the response before it reaches the agent removes one of the two parses. Upstream Miles has since added a mode that asks SGLang only for the rows a session doesn't already hold, and Periodic describes the same idea as delta router replay.

One open question: the routing we keep for a token comes from whichever forward last wrote its KV slot. With a cache hit that is often the decode that produced the token; after a cache miss it is a later prefill, which can route differently. The release doesn't check how often.

Sandboxes were the death of us

Three changes did most of the work.

Waves. Our first launcher started every sandbox at the start of a batch, so thousands of pods asked the image registry for their images at once; pulls conflicted, pods failed, and every failed pod came back as a retry that pulled again. Admitting rollouts in timed waves (--rolling-start-size) keeps pulls under what the registry can serve, and engines get work as soon as the first wave lands.

Reservations. Sandboxes also share nodes, and one heavy build could take the CPU and memory its neighbours needed, slowing them down or getting one of them OOM-killed. Reserving resources per pod keeps each sandbox to its share.

Pinning. Sandboxes that landed on arbitrary nodes kept pulling their images cold, so we pinned groups of images to specific nodes, and most starts now hit a warm local cache.

At any moment of the 25-hour 375B run, about 0.5% of the trajectories in flight were waiting on sandbox setup. The console log agrees:

The biggest infrastructure loss had moved to the inference engines, and those losses came in bursts, with 1,535 in one 83-minute window.

From token to gradient

The rest is slower going: eight details from the loop itself, with numbers from the public run.

1Tokens in, tokens out

The agent talks to /sessions/{id}/v1 like any OpenAI client and never sees a token ID.

The session server keeps the token IDs of every turn and sends the next request to SGLang as input_ids: the stored IDs plus the new tool or user message, tokenized on its own. The trainer gets exactly the tokens the sampler produced. The flags that make this work are assigned outright, so no agent config can switch them off:

request_body["logprobs"] = True
request_body["return_prompt_token_ids"] = True
request_body["return_meta_info"] = True
request_body["return_sampling_mask"] = session.capture_sampling_mask
if getattr(args, "use_rollout_routing_replay", False):
    request_body["return_routed_experts"] = True
# Must be False so stop tokens are trimmed from output: otherwise the
# agent sees stop-token text in content, and the accumulated checkpoint
# would duplicate structural delimiters that the chat template also emits.
request_body["no_stop_trim"] = False

This only works if the chat template is append-only, meaning a message renders the same way whatever follows it. Qwen3's template isn't: it branches on loop.last and expects a newline after <|im_end|> that the model never generates, so Miles swaps in a fixed template and patches the boundary.

We built the K2V3 template so the problem can't happen: it emits nothing between <|ifm|im_end|> and the next <|ifm|im_start|>, and the stored tokens already equal the canonical render.

2One sample per trajectory

When a rollout ends, its turns are merged into one training sample.

The loss mask is 1 on tokens the policy sampled and 0 on the prompt, tool output and anything else appended. Each turn's rollout log-probs and weight version come along.

If a turn stopped early (finish_reason is length), every turn after it is dropped, because those turns were conditioned on a reply that was cut off. The cut turn itself stays, and the sample still carries the verifier's reward for the whole trial, which the dropped turns helped earn.

3Round-robin throws the cache away

Each request carries X-SMG-Routing-Key: <session_id>, which SMG's manual policy uses to send a session back to the engine holding its KV cache.

The release recipe uses round_robin, which ignores the key, so a conversation can land on a different engine every turn and prefill from scratch.

The public run shows what that costs. A trajectory averaged 61 turns, and because every turn resends the conversation so far, its prompts added up to 2.6M tokens. The prefix-cache hit rate was 20%, so about 2.1M of those tokens were prefilled again.

4Two 3,600-second clocks

The client wraps the whole /run call in AGENT_TIMEOUT_SECS (3,600 s), and Harbor pins the agent's own budget at a flat 3,600 s as well.

The client's deadline also has to cover queueing, sandbox setup and the verifier, so an agent that uses its whole budget hits the client deadline first, and its sample comes back with no reward, which gets scored as 0 (detail 6).

5Generate the next batch while training on this one

train_async.py starts generating batch k+1 before training on batch k, then waits for that generation to finish before pushing new weights, so weights never change mid-rollout.

if (rollout_id + 1) % args.update_weights_interval == 0:
    # sync generate before update weights to prevent update weight in the middle of generation
    rollout_data_curr_ref = (await x) if (x := rollout_data_next_future) is not None else None
    rollout_data_next_future = None
    await actor_model.update_weights()

Every update therefore trains on data from the policy one version back. With --keep-old-actor, Megatron keeps a queue of weight copies, so the PPO ratio's "old" policy is exactly the one the engines sampled with, and the PPO ratio becomes a staleness ratio. Truncated importance sampling (TIS) on top, capped at 2, only has to absorb the numerical gap between SGLang and Megatron.

On the public run that gap was small: the mean per-token |Δ log p| between trainer and sampler stayed between 0.015 and 0.018 for all 99 steps, the TIS cap touched about 2 tokens in a million, and one step of staleness clipped 0.02% of tokens (median pg_clipfrac). That's a dense 7B; the MoE case, where the gap is known to be worse, isn't in the public run.

On the public run the overlap saved about 81 s of a 40-minute step. Training took 81 s and the weight push 3.8 s, so the trainer sat idle about 97% of the time. The rollouts themselves were mostly tail: the mean trajectory took 10.4 minutes, and the step waited another half hour for the slowest ones.

6Telling infrastructure failures from policy failures

A sandbox that times out, a verifier that crashes or a session that vanishes says nothing about the policy, and at 8,192 sandboxes these are routine.

RL360 marks a sample as an infrastructure failure if it was aborted, if its Harbor exit status is on a list (AgentTimeout, EnvStartTimeout, VerifierTimeout, NotFoundError and others), if an LLM judge call failed, or if it has no reward at all. Those samples get a zero loss mask, and an empty aborted attempt borrows a valid sequence only so the tensors line up. A group trains only if at least two valid attempts disagree:

def check_valid_reward_nonzero_std(args, samples: list[Sample], **kwargs) -> DynamicFilterOutput:
    """Keep groups with varying valid rewards despite individual infrastructure failures."""
    valid_samples = [sample for sample in _flatten_samples(samples) if not is_infra_failure(sample)]
    if len(valid_samples) < 2:
        return DynamicFilterOutput(keep=False, reason="insufficient_valid_rewards")
    return check_reward_nonzero_std(args, valid_samples, **kwargs)

The weak spot is that it's a list. A failure the list doesn't name still scores 0: an unexpected exception inside Harbor comes back as reward 0 with exit status Error: RuntimeError, and a client-side timeout returns nothing, which the reward function turns into 0 before the filter runs. The safer default is to trust a reward only if a verifier wrote it.

Counting AgentTimeout as infrastructure is also a judgment call, since running out the clock is often the policy's fault. DeepSWE masks timed-out trajectories too, and SWE-Master reports that the same masking made its reward degrade until training collapsed.

7GRPO, and what filtering does to its baseline

--advantage-estimator grpo --disable-grpo-std-normalization --use-tis
--use-kl-loss --kl-loss-coef 0 --kl-loss-type low_var_kl
--entropy-coef 0 --eps-clip 0.2 --eps-clip-high 0.28

Filtering changes the batch size every step.

--use-dynamic-global-batch-size sets the global batch to the surviving sample count, rounded down to a multiple of the data-parallel size, so each rollout is exactly one optimizer step.

The varying size has a side effect if you reuse the recipe: the code reshapes rewards into groups only when the batch is full, so on a step where the filter dropped a group, the GRPO baseline is the mean over the whole batch. On the public run that was 96 of 99 steps.

That mean also counts masked samples, at a reward of 0 (step 70 shows what that does). The 375B run doesn't hit the fallback, but its group baselines count masked samples at 0 too.

Dr. GRPO is only half applied. Per-group std normalization is off, but the loss is still a mean over each sample, which is the length bias that paper removes with a constant normalizer.

8Log the reward before the filter

The filter drops all-pass and all-fail groups, so the trained batch's mean reward moves when easy groups disappear, even if the policy hasn't changed.

RL360 stashes every group before filtering and logs reward/unfiltered_mean next to reward/kept_mean, along with per-domain means and masked and trained sample counts.

Plot only the kept mean and the policy looks like it stopped learning at step 40.

Step 70

Step 70 is the pink spike in Figure 1 and the dip in Figure 14: a burst of infrastructure failures, gone by the next step. One step of that reaches further than the samples it loses.

Of the 56 samples the filter kept, 23 were masked. Three numbers then moved on their own:

× 0.59

on every gradient. The loss is a mean over kept samples, and masked ones count in the denominator at zero. Adam absorbs part of that, but the step's batch still counts for less than its neighbours, for a reason that has nothing to do with the data.

+ 0.31

on the advantage of every trained sample, compared with a baseline over valid samples. The whole-batch baseline (detail 7) came to 0.44 against a kept mean of 0.75.

0.49

logged as mean reward, down from 0.89 and back at 0.79 the next step, while the kept mean held at 0.75. The unfiltered mean scores every infrastructure failure as 0, so that step's failures were averaged in as wrong answers.

Other logged values take the same hit. Each is a mean over samples with that denominator, so train/tis, which should sit near 1, equals 1 − masked/kept on every one of the 99 steps, and the logged |Δ log p| dips to 0.009 on this one (the range in detail 5 is corrected for it). What we should have logged is the mean over valid samples, with the infrastructure failure rate next to it.

Conclusion

Very little of this post is about the objective. Most of it is bookkeeping: making sure the trainer sees exactly the tokens the sampler produced, keeping broken sandboxes out of the loss, and knowing which reward you are looking at.

If I were starting over, I would get three things right before anything else:

• Tokens in, tokens out, with a chat template that only appends (details 1 and 2), because everything downstream assumes it.
• A strict line between a failed environment and a failed policy, where a reward counts only if a verifier actually wrote it and a failure stays out of the baseline as well as the loss (details 6 and 7).
• Reward logged before the filter and over valid samples only (detail 8), so neither the filter nor a sandbox outage can make a learning policy look stuck or regressing.

The next change is to the schedule. The release trains one update behind and waits for every rollout before pushing weights. Prime Intellect updates weights while generation is still in flight, and its INTELLECT-3 report says steps took more than twice as long without that. Periodic does the same with PipelineRL. Reflection tags every token with the policy version that produced it and reports stable numerics on data 107 versions old. The release already records the weight version on every turn, which is the bookkeeping that needs.

Sources

• RL360, commit 9a9522a (2026-10-22): github.com/ifm-ai/RL360. Every code link in this post points to that commit.
• Public W&B run wm7usgst: wandb.ai/mbzuai-llm/rl360-public. K2-Horizon-7B, a dense model, overfitting 32 tasks (27 coding, 5 STEM questions), so it doesn't touch the MoE paths. Medians and ranges are over its 99 logged steps. It was created on 15 September 2026, five weeks before that commit. Its config records no commit, though its flags match the release recipe, and W&B lists its state as crashed.
• Production numbers: one 375B run of our internal branch, 28–29 October 2026. Step medians are over the 16 steps after it resumed from a checkpoint, and the rollout figures come from about 730 two-minute windows. The weight-push counts and sandbox phases come from the job's console log, and the pushes after the fix from a follow-up run on 30 October.
• Model cards: K2-Horizon-7B, K2-Horizon-375B-A23B. IFM, Introducing K2 Horizon.
• Other write-ups cited: Yao et al., Your efficient RL framework secretly brings you off-policy RL training; Dr. GRPO; rollout routing replay; DeepSWE; SWE-Master; Prime Intellect, INTELLECT-3; Periodic, AI Infrastructure at Periodic; Reflection, Introducing Beam.
• Bundled projects: Miles, Harbor, Megatron-LM, SGLang.