On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation

Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) to consolidate capabilities from multiple teachers. Although prior work reports near-perfect in-distribution (IND) distillation performance, we find that such performance can mask incomplete capability consolidation beyond the supervised distribution. A preliminary synthetic reasoning study shows incomplete transfer of teacher generalization. We then experiment on real-world extrapolative and compositional generalization across mathematical reasoning, coding, agent, and long-context capabilities using Qwen3 models. Across extrapolative settings, OPD and MOPD reach 96.04-102.73% of their teachers’ mean IND scores, yet their mean out-of-distribution (OOD) scores remain 2.45-16.25% below the base. Across three compositional settings, MOPD improves mean IND performance by 0.48-22.46% over the base but remains 18.19-24.34% below the best teachers on composed tasks. We analyse teacher capability and data coverage as two potential bottlenecks, where jointly easing them eliminates OOD forgetting in our experiments and shows potential for generalization beyond the teacher. Moreover, relative to their RL baselines, OPD and MOPD reduce OOD forgetting on average. Geometry analysis finds more compact OPD and MOPD updates on average than those of their RL baselines, along with greater shared update structure under MOPD. Based on the analysis, we propose geometry-based patching and base-preserving replay to improve OOD performance while retaining most IND performance. Our findings identify a blind spot in capability consolidation via MOPD and motivate distillation algorithms that explicitly support OOD retention and generalization.

OPD can achieve near-perfect IND distillation within relatively few training steps [40, 41]. Yet grokking shows that generalization can emerge long after a model fits its training data [50]. This motivates asking whether rapid IND convergence in OPD is accompanied by transfer of the teacher’s generalization. We examine this question in a controlled synthetic mathematical reasoning study.

Findings. Figure 1 shows near-perfect IND distillation alongside incomplete OOD transfer. While the distillation ratio of OPD is approximately 96-104% on pretraining IND and 93-96% on post-training IND, it falls to 76-86% on OOD problems. Thus, when the teacher generalizes beyond the supervised range, OPD does not fully transfer that generalization despite strong IND performance. This motivates testing whether the same limitation persists when consolidating capabilities on real-world tasks, where useful behavior can extend far beyond narrow distillation data and require distinct capabilities together.

We examine whether near-perfect IND distillation yields extrapolative generalization in mathematical reasoning, using self-RL specialist teachers and external generalist teachers.

Overall findings. Across both teacher setups, near-perfect IND distillation coexists with OOD forgetting. OPD and MOPD reduce average forgetting relative to the self-RL teachers and mixed RL, respectively, but still fall below the base on OOD evaluations.

Across the three distillation datasets, OPD achieves near-perfect IND distillation, reaching 99.34% of the teacher’s IND score (0.62 points below it), yet remains 3.73% below the base on OOD evaluations (2.96 points). Stronger teacher performance alone therefore does not ensure OOD transfer or retention under narrow supervision. § 7.1 further examines whether broader distillation data improves these outcomes.

Overall findings. Across the three settings, MOPD improves IND performance by 9.24% over the base (3.97% below the corresponding teachers), but remains 23.82% below the best teachers on composed tasks (Figures 5 to 7). Mixed RL shows a similar compositional gap (26.10%); MOPD’s mean composed-task score is 3.08% (1.07 percentage points) higher than mixed RL’s.

Code and Agent. Coding and tool-use supervision improves SWE-bench Verified from the base’s 6.63 to 8.05, above mixed RL (7.79) but below the shared teacher (9.84; Figure 5). MOPD also improves over the base on all component benchmarks except HumanEval+.

Math and Long Context. MOPD improves LongBench-v2 from 40.50 to 56.20 and GSM8K from 83.40 to 95.53, yet the composed-task outcomes diverge (Figure 6). DocMath improves from 55.38 to 60.63, whereas GSM-Infinite falls from 50.88 to 49.75; both remain below the best teacher. MOPD nevertheless outperforms mixed RL on all displayed benchmarks.

Code and Long Context. MOPD improves all three component benchmarks, yet LongBench-v2 Code remains at the base’s 44.00, below both teachers (50.00 and 58.00; Figure 7). Mixed RL raises this composed-task score to 46.00 but lowers the LongBench-v2 component score from 40.50 to 38.84, illustrating a trade-off between component retention and composed-task performance.

The code plus agent, math plus long-context, and code plus long-context evaluations show a compositional generalization limitation: a student can acquire improvements on separately supervised capabilities without obtaining the expected performance when those capabilities must be combined.

Under narrow supervision, students can remain below the base on OOD evaluations with either self-RL specialists (§ 5.1) or stronger external generalists (§ 5.2). Thus, stronger teacher capability alone is insufficient for OOD retention (panels a-b of Figure 8). To relax both bottlenecks (panel c), we keep Qwen3-32B as teacher and train on 90% GSM8K plus 10% OpenR1 [45]. Compared with GSM8K-only distillation, it exceeds the base on all five benchmarks. The student matches the teacher’s AIME24/25 pass@8 (86.67/83.33). On AMC23, it improves to 100.00, exceeding the teacher’s 95.00 (Figure 9).

Together, these results support considering teacher capability and data coverage jointly: a stronger teacher makes OOD capabilities available, while broader supervision helps the student retain or acquire them. Easing both bottlenecks can reduce OOD forgetting and potentially encourage further generalization.

Both OPD and MOPD produce smaller updates on average than their corresponding RL baselines (Table 1). Across the extrapolative experiments, at approximately matched IND performance, OPD’s update norm is 55.4% of RL’s (0.154 versus 0.278). For MOPD, the mean update norm is also smaller than that of mixed RL.

OPD updates are also more spectrally concentrated. OPD’s top 64 singular directions capture 69.0% of update energy, compared with 39.9% for its RL teachers.

MOPD is better explained by the single-OPD span than mixed RL is by the single-RL span. The respective spans explain 11.0-18.6% of MOPD’s squared update norm and less than 0.240% of mixed RL’s (Table 1). Using the same single-RL teacher span for both methods also yields a higher R2 for MOPD than for mixed RL. These measurements suggest that MOPD is an algorithm with more shared update structure.

Please note that these connections remain a preliminary interpretation; a causal link between these geometric properties and generalization has yet to be established.

Since near-perfect IND distillation can coexist with an OOD performance gap, improving generalization also requires preserving capabilities beyond the training distribution. We investigate two explicit interventions: geometry-based patching after distillation and base-preserving replay during distillation.

Geometry-Based Patching

The compact, spectrally concentrated OPD updates in § 7.2 motivate post-hoc editing: OPD students can forget less than self-RL teachers at similar IND performance. We hypothesize that suppressing update components that harm OOD retention but contribute little to IND performance can reduce forgetting; these components may arise from optimization noise.

Base-Preserving Replay

KL-regularized RLVR constrains policy drift from a reference model [52]. In contrast, the (M)OPD objectives in § 3.1 match task teachers without an explicit constraint to preserve the base model’s behavior. We hypothesize that this missing constraint contributes to OOD forgetting. Motivated by replay methods for capability preservation during RL post-training [49], we introduce base-preserving replay for (M)OPD. Base-preserving replay mixes IND examples supervised by the task teachers with additional prompts supervised by the base model, using the same OPD loss. We test coding and instruction-following data as replay sources outside the specialist’s mathematical training distribution; Figure 11 reports coding replay for the GSM8K OPD student distilled from a self-RL teacher.

Relative to plain OPD, coding replay improves five of eight OOD scores, including a 6.67-point increase in AIME25 avg@8 to 67.50, surpassing both the student base and RL teacher with only a 0.18-point reduction in GSM8K accuracy. These results suggest that replay can support both retention and generalization beyond the replay domain.

Limitations

Our main post-training experiments are on Qwen3-8B students, and our largest external teacher is Qwen3-32B given computation constraints. Fixing the student supports within-model comparisons, but leaves dependence on student capacity, teacher strength, and model family under-explored. Our main findings on the relative performance of MOPD and mixed RL concern the tested LLM architecture and practical post-training tasks. At the algorithmic level, this ordering need not hold for simpler models and tasks with different generalization requirements, which we additionally discuss in § 9.2.

Public datasets make our comparisons inspectable, but do not reproduce the breadth or adaptive data selection of full post-training pipelines in frontier model development. Our OOD labels are relative to post-training data, not pretraining exposure. Broader training data from the real world and held-out evaluations with graded difficulty would test generalization under more demanding conditions.

Mixed RL and MOPD both combine component recipes without extensive tuning of reward balancing, mixture weights, or curricula (§ A.1). They may not establish each objective’s best achievable performance. However, we have tuned hyperparameters for single OPD and single RL recipes to ensure optimal performance. Considering the much larger tuning space for Mixed RL and MOPD, we view this as an out-of-the-box usage and see this comparison as a fair setting.

Computational constraints prevent us from repeating every experiment with multiple sampling seeds. For mathematical reasoning, we repeat evaluation with four sampling seeds on a 10% subset of the evaluation questions and observe consistent qualitative conclusions across these checks. These checks assess variability in sampled responses; they do not establish robustness across independent training runs.

Arithmetic Study

MOPD achieves higher mean composed-task accuracy than mixed RL in our main comparisons on practical post-training tasks (§ 6), but this ordering need not hold across all possible settings. Conceptually, MOPD learns from teacher distributions without an independent correctness signal, whereas RL uses verifier feedback on its training tasks. There might be a soft bound on teacher OOD accuracy when teacher exists inaccuracy at the first place, which might be harder for MOPD to generalize beyond either teacher.

These results reveal that, considering different model architectures, task learnability, interference between tasks, etc., there can be different balances between component performance and compositional generalization. Isolating their effects across architectures and training objectives remains future work. Our main conclusions concern the LLMs and practical post-training tasks studied here.

Future Work

The connection between update geometry, OOD retention, and compositional generalization remains a hypothesis. Causal tests should manipulate update directions while controlling update magnitude, training budget, and IND performance. Beyond post-hoc patching, step-level parameter-space control could project or regularize updates during training, with random-direction controls. Future work should also jointly study teacher assignment, data coverage, and base-preserving replay under a fixed budget, testing retention and further generalization beyond expanded training coverage.