On-Policy Self-Distillation without Any Supervision

Yijiang Li$^{\spadesuit}$ Bingyang Wang$^{\heartsuit}$ Yijun Liang$^{\diamondsuit}$ Yunjie Tian$^{\clubsuit}$
Di Fu$^{\clubsuit}$ Nuno Vasconcelos$^{\spadesuit}$

$^{\spadesuit}$ UC San Diego $^{\heartsuit}$ Georgia Institute of Technology
$^{\diamondsuit}$ University of Maryland, College Park $^{\clubsuit}$ ByteDance

{yijiangli, nuno}@ucsd.edu

Abstract

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions a teacher distribution on the shortest pseudo-solution and distills it into prefixes of the model's longest incorrect completion, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively.

Executive Summary: The paper introduces unsupervised on-policy self-distillation (U-OPSD), a method that improves large language models' reasoning performance using only the models' own outputs. Existing post-training approaches for reasoning, including supervised fine-tuning, reinforcement learning with verifiable rewards, and on-policy distillation, all depend on external labels such as ground-truth solutions or stronger teacher models. This dependence raises costs and limits use on unlabeled problems. The work asks whether genuine self-distillation is possible without any such external input.

The authors propose U-OPSD to answer that question. For each unlabeled problem, the model generates multiple independent solutions. It forms a pseudo-solution from the majority vote among those that agree, provided the agreement exceeds a consistency threshold. The method then trains the model to match the next-token distribution of the shortest agreeing solution when generating the prefixes of its own disagreeing solutions. This process supplies dense, corrective supervision exactly where the model is inconsistent with its own consensus. Experiments used Qwen3 models from 4B to 30B parameters on five math benchmarks, training for 150 steps with eight rollouts per prompt and comparing against supervised baselines that had access to ground-truth labels.

U-OPSD raised average performance by 8.5–10.7 points over the base models in non-thinking mode and by 1.9–2.2 points in thinking mode. It also outperformed or matched the supervised versions of on-policy self-distillation and group-relative policy optimization on the same data, with gains of up to 3.2 points over the supervised self-distillation baseline at the 4B scale. Improvements appeared consistently across model sizes and held when the base models were already instruction-tuned. Ablations showed that full-distribution distillation, longer reasoning traces for the teacher, and moderate rollout counts contributed most to the gains.

These results indicate that majority agreement among a model's own generations can replace ground-truth supervision for token-level distillation. The approach therefore removes a major scalability barrier for post-training on domains where labels are scarce or expensive. It focuses training effort on the model's competence frontier—problems where consensus exists but errors remain—producing a natural curriculum without external difficulty labels.

Organizations training reasoning models should test U-OPSD on their unlabeled data before investing in large-scale annotation. Immediate next steps include applying the method to non-mathematical domains that allow answer extraction or softer consensus measures, increasing rollout counts where compute permits, and exploring adaptive teacher updates. Further validation on open-ended tasks and larger models would strengthen confidence before broad deployment.

The main limitations are the restriction to problems with extractable final answers, larger gains observed only in weaker non-thinking regimes, and dependence on the base model's existing competence for reliable votes. Results rest on single runs per configuration; replicated error bars are not yet available. Readers should treat the reported improvements as promising but provisional until broader testing confirms stability across seeds and domains.

1. Introduction

Section Summary: Post-training methods have improved large language models’ reasoning by using techniques such as supervised fine-tuning and reinforcement learning, but most still depend on external guidance from stronger models or labeled data. On-policy self-distillation tries to remove the stronger teacher, yet it continues to rely on ground-truth answers or other outside signals, limiting its use on unlabeled problems. The paper introduces an unsupervised variant called u-OPSD that lets a single model generate multiple answers to a question, treat the majority answer as a reliable reference, and distill improved behavior from its own agreeing and disagreeing outputs without any external supervision.

Post-training has emerged as a key driver of advances in the reasoning capabilities of large language models (LLMs), with progress largely propelled by supervised fine-tuning (SFT) ([1, 2]), knowledge distillation from stronger teachers ([3, 4]), and reinforcement learning with verifiable rewards (RLVR) ([5, 6, 7]).

Among the many recipes, on-policy distillation bridges SFT and RL by training on the model's own generations, reducing the train-inference mismatch ([8, 9]) and catastrophic forgetting ([10]) of teacher-forced SFT, while retaining dense token-level supervision instead of sparse scalar rewards. Subsequent work has continued the effort of OPD along its objective, supervision, and systems dimensions. DistiLLM ([11]) introduces skewed KL, while DistiLLM-2 applies asymmetric objectives to teacher- and student-generated responses ([12]). Other work stabilizes long-horizon OPD by restricting supervision to teacher-supported tokens and masking unreliable signals ([13]), or reduces its systems overhead by precomputing teacher scores under a teacher-consistent offline pipeline ([14]). More recent analyses further identify teacher–student mismatch and length exploitation as key failure modes, motivating clipped and log-compressed token-level guidance ([15]). Nevertheless, these methods still heavily rely on a separate strong teacher model for guidance.

On-policy self-distillation (OPSD) ([16]) pushes this further by removing the need for a stronger teacher model – a single LLM plays both roles, where the teacher (student itself) conditions on the ground-truth solution, whereas the student sees only the problem. Subsequent work identifies information leakage and instability ([17]), redesigns the privileged context ([18, 19, 20]), and extends it to agentic setups ([21, 17]).

The privileged context, however, is also its own bottleneck. Existing OPD and OPSD methods rely on external supervision, such as ground-truth solutions, environmental feedback, or guidance from larger models. Thus, the model is self-distilled only in the sense that the teacher and student share parameters; the information that makes the teacher more capable still comes from outside the model. This dependence limits scalability to unlabeled problems and restricts applicability in domains where supervision is costly, unreliable, or unavailable. We therefore ask: Does the teacher in on-policy self-distillation need a ground-truth solution? Can a model construct its own privileged context and perform genuine self-distillation?

We answer with $\textsc{u-OPSD}$, an unsupervised on-policy self-distillation method that requires no external supervision. Our key observation is that although an individual rollout may be unreliable, agreement among multiple independently sampled rollouts provides an endogenous confidence signal. This makes it possible to derive both the teacher reference and the student trajectories entirely from the model's own on-policy samples. Formally, for each unlabeled problem $x\sim\mathcal{U}$, we sample $G$ independent rollouts $y^{(1)}, \ldots, y^{(G)} \overset{\mathrm{i.i.d.}}{\sim} \bar\pi(\cdot\mid x)$, where $\mathcal{U}$ is the unlabeled problem distribution, $G$ is the number of rollouts per problem, and $\bar\pi\triangleq\pi_{\operatorname{sg}[\theta]}$ denotes a stop-gradient copy of the current policy $\pi_\theta$. We extract the final answers from each rollout as $a^{(g)}=\mathrm{Ans}(y^{(g)})$, where $\mathrm{Ans}(\cdot)$ denotes the answer-extraction and canonicalization function, and obtain the vote-based pseudo-answer $\tilde{a}(x)=\operatorname*{arg, max}{a}\sum{g=1}^{G}\mathbb{1}!\left[a^{(g)}=a\right]$, where $\mathbb{1}[\cdot]$ is the indicator function.

The rollouts are then partitioned into agreeing and disagreeing multisets $\mathcal{Y}^{+}{x}={y^{(g)}:a^{(g)}=\tilde{a}(x)}, \qquad \mathcal{Y}^{-}{x} = {y^{(g)}:a^{(g)}\neq\tilde{a}(x)}$. When the winning vote fraction reaches a self-consistency threshold $\tau$, the shortest agreeing rollout $y^{+} \in \operatorname*{arg, min}{y\in \mathcal{Y}^{+}{x}} |y|$ serves as the pseudo-solution, while the disagreeing rollouts in $\mathcal{Y}^{-}_{x}$ serve as the student trajectories. By distilling the teacher distribution conditioned on the pseudo-solution into the student along prefixes of the model's disagreeing completions, $\textsc{u-OPSD}$ achieves continual improvement through on-policy self-distillation without any external supervision.

$ \begin{aligned}\mathcal{L}{\textsc{u-OPSD}}(\theta) = \mathbb{E}{x\sim\mathcal{U}}, \mathbb{E}{{y^{(g)}}{g=1}^{G}\sim \bar\pi(\cdot\mid x)} &\Bigg[\mathbb{1}!\left[\mathcal{Y}^{-}{x}\neq\emptyset\right] \frac{1}{|\mathcal{Y}^{-}{x}|} \sum_{y^{-}\in\mathcal{Y}^{-}{x}} \frac{1}{|y^{-}|} \sum{n=1}^{|y^{-}|}\&\qquad D_{\beta}!\left(\bar\pi!\left(\cdot\mid x, , y^{+}, , y^{-}{<n}\right) , \middle|, \pi\theta!\left(\cdot\mid x, , y^{-}_{<n}\right) \right) \Bigg].\end{aligned}\tag{1} $

Here, $|y^{-}|$ denotes the token length of a disagreeing rollout, and $y^{-}{<n}$ denotes its prefix before token position $n$. The function $D{\beta}(P, |, Q)$ denotes a divergence between the teacher and student next-token distributions, such as the forward KL divergence $D_{\mathrm{KL}}(P, |, Q)$.

We evaluate $\textsc{u-OPSD}$ on five mathematical reasoning benchmarks across six Qwen3 configurations: 4B and 8B models in both non-thinking and thinking modes, together with Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507. Across all configurations, $\textsc{u-OPSD}$ improves over the base model, by $8.5$ – $10.7$ points in non-thinking mode, $1.9$ – $2.2$ in thinking mode, and $1.7$ – $1.8$ on the Instruct models. $\textsc{u-OPSD}$ also improves over prior self-rewarding RL methods by a large margin, i.e., on average $7.0$ – $11.3%$ in non-thinking mode and $0.8$ – $1.4%$ in thinking mode, showing that consensus is substantially more effective as conditioning context for token-level distillation than as a scalar reward for policy optimization. Without any external supervision, $\textsc{u-OPSD}$ is still on par with or better than supervised SFT, GRPO, and OPSD with GT labels, with gains of up to $10.9$, $8.9$, and $3.2$ points, respectively, while tying OPSD only in the 8B thinking setting.

2. Related work

Section Summary: Recent research on self-rewarding reinforcement learning has explored ways to train language models without relying on external labels, instead drawing reward signals from the model's own confidence in its outputs or from agreement across multiple sampled answers. Parallel efforts in on-policy self-distillation and self-training let a model learn from its own generated trajectories, using techniques such as asymmetric conditioning or consistency-based preferences to provide supervision without separate teachers or verified data. The approach described here builds on these ideas by turning agreement among the model's rollouts into a teacher signal for token-level distillation and using points of disagreement to select correction targets, all without any annotations or external verification.

Self-rewarding Reinforcement Learning. Although RLVR has proven effective at strengthening LLM reasoning ([22, 6]), it hinges on curated ground-truth labels, whose cost and scarcity quickly become the limiting factor ([23]). A growing body of work replaces external verification with reward signals the model derives from its own behavior on unlabeled data. Among these intrinsic signals, confidence-based rewards are especially appealing: a model's certainty in its own output—captured directly as self-certainty ([24, 25]) or inversely through predictive entropy ([26, 27])—yields a dense, label-free reward that requires no extra sampling infrastructure and correlates well with answer correctness. Agreement across sampled solutions offers a complementary signal, as exploited by majority voting ([28]) and test-time training on self-consistency ([29]). These methods build on earlier self-rewarding language models ([30]) and self-play supervision ([31]), and the paradigm has since broadened to unsupervised self-training ([32, 33]), self-correction guided by the model's own judgments ([34]), and zero-data self-evolution in which models construct their own curricula ([35, 36, 37]).

On-policy (self-) distillation. On-policy distillation trains on student-generated trajectories, allowing a teacher to provide dense token-level supervision at states visited by the student and reducing the train–inference mismatch of off-policy distillation ([8, 9]). On-policy self-distillation removes the separate teacher by letting the same model act under asymmetric contexts, with the teacher conditioned on verified solutions ([16]), demonstrations or in-context examples ([38]), privileged observations ([19]), or environmental feedback ([39]). Subsequent work explores alternative conditioning signals ([40]), derives process supervision from externally verified successful and unsuccessful trajectories ([41]), or modulates token-level supervision according to teacher reliability ([42, 43]). $\textsc{u-OPSD}$ instead constructs the teacher reference from rollout agreement and identifies correction targets through rollout disagreement, without annotations, demonstrations, environmental feedback, gold answers, or externally verified outcomes.

Self-training, self-distillation and consistency. Self-training improves models using supervision derived from their own generations. Existing methods select or refine self-generated data using answer verification and reward signals ([44, 45, 46]), as well as confidence, consistency, and iterative self-improvement ([47, 48]). More broadly, self-distillation transfers knowledge across different instances or views of the same model, including previous model generations and intermediate optimization snapshots ([49, 50]). Other approaches distill across architectural branches or augmented views ([51, 52]), while recent work extends self-distillation to foundation models using diverse reasoning traces or multiple self-teachers ([53, 54]). Self-consistency originally aggregates multiple reasoning paths through majority voting at inference time ([28]), while subsequent methods turn agreement into sequence-level preferences ([55]), pseudo-labels or scalar reinforcement-learning rewards ([29, 56]), or model-generated feedback and preferences ([57]). $\textsc{u-OPSD}$ connects self-consistency with on-policy self-distillation by using consensus to construct an instance-specific self-teacher and transferring dense next-token distributional supervision along prefixes of consensus-disagreeing rollouts.

3. Method

Section Summary: The method section introduces u-OPSD, an unsupervised variant of on-policy self-distillation for training language models on reasoning tasks. It starts from existing approaches like GRPO, which relies on gold answers for sparse rewards, and OPSD, which uses ground-truth solutions to create a privileged teacher that guides the student policy via token-level divergence on the model's own rollouts. u-OPSD removes all external supervision by instead generating pseudo-labels through the model's majority-vote consensus on unlabeled problems, allowing the same model to serve as its own teacher without reference answers or stronger external models.

3.1 Preliminaries

Notation.

Let $\pi_\theta$ be an LLM defining next-token distributions $\pi_\theta(\cdot\mid c)$ over a vocabulary $\mathcal{V}$ for a context $c$, and let $\bar\pi\triangleq\pi_{\operatorname{sg}[\theta]}$ be the same network with gradients detached, so that sampling from or scoring under $\bar\pi$ carries no gradient. For a sequence $y$, $y_{<t}$ denotes its prefix, and $\mathrm{Ans}(y)\in\mathcal{A}\cup{\varnothing}$ the final answer parsed from it, where $\mathcal{A}$ is the space of admissible answers and $\varnothing$ marks a generation from which no answer can be parsed (Appendix C). Supervised post-training draws from a labeled corpus $\mathcal{S}={(x, y^{\star})}$ of problems with reference solutions and gold answers $a^{\star}=\mathrm{Ans}(y^{\star})$; $\textsc{u-OPSD}$ uses only unlabeled problems $x\sim\mathcal{U}$. Throughout, $D_{\beta}$ denotes the generalized Jensen–Shannon divergence between distributions on $\mathcal{V}$, whose $\beta\to 0$ limit is the forward KL.

Group Relative Policy Optimization: sparse signal by verifiable reward.

GRPO ([22]) optimizes the policy with a verifiable reward normalized within a group of sampled rollouts: for each problem it samples $G$ rollouts $y^{(1)}, \ldots, y^{(G)}$, scores them with $r^{(g)}=\mathbb{1}!\left[\mathrm{Ans}(y^{(g)})=a^{\star}\right]$, and normalizes within the group, $A^{(g)}=\big(r^{(g)}-\operatorname{mean}{r^{(j)}}{j=1}^{G}\big)\big/ \operatorname{std}{r^{(j)}}{j=1}^{G}$, so that all tokens of a rollout share one sequence-level advantage. The policy is then updated on the clipped surrogate

$ \begin{aligned}\mathcal{L}{\mathrm{GRPO}}(\theta) =-, \mathbb{E}{x\sim\mathcal{S}}, \mathbb{E}{{y^{(g)}}{g=1}^{G}\sim\bar\pi(\cdot\mid x)}, \frac{1}{G}\sum_{g=1}^{G}\frac{1}{|y^{(g)}|}\sum_{t=1}^{|y^{(g)}|} \min!\Big(&\rho^{(g)}{t}A^{(g)}, \&\operatorname{clip}\big(\rho^{(g)}{t}, , 1{-}\varepsilon, , 1{+}\varepsilon\big)A^{(g)}\Big), \end{aligned}\tag{2} $

where $\rho^{(g)}{t}=\pi\theta\big(y^{(g)}{t}\mid x, y^{(g)}{<t}\big)\big/ \bar\pi\big(y^{(g)}{t}\mid x, y^{(g)}{<t}\big)$ is the importance ratio to the behaviour policy and $\varepsilon$ is the clipping range. The supervision is the gold answer $a^{\star}$, and the signal is sparse: it is sequence-level, and vanishes whenever all $G$ rewards coincide.

On-policy distillation: dense signal by external teacher.

On-policy distillation (OPD) addresses this sparsity by retaining the model's own rollouts as training trajectories while replacing the scalar reward with dense supervision from a teacher's next-token distribution. The student is conditioned only on the problem $x$, matching inference time, while the teacher receives a possibly richer context $c$. Writing $\pi_{T}$ for the teacher, these methods share the form

$ \mathcal{L}{\mathrm{OPD}}(\theta) =\mathbb{E}{x}, \mathbb{E}{y\sim\bar\pi(\cdot\mid x)} \sum{t=1}^{|y|} D_{\beta}\big(\pi_{T}(\cdot\mid c, , y_{<t}), \big|, \pi_\theta(\cdot\mid x, , y_{<t})\big),\tag{3} $

which yields dense token-level supervision at states the student actually visits. Variants differ only in the choice of teacher $\pi_{T}$ and the context $c$ it receives. OPD ([8, 9, 4]) takes $\pi_{T}$ to be a separate, typically stronger model with $c=x$; the supervision is dense, but it resides in an external teacher that must already surpass the model being trained.

On-policy self-distillation.

OPSD ([16]) takes a further step by constructing the teacher and student from the same language model under different conditioning contexts, removing the external teacher and supplying a gold solution in its place. The student policy observes only the problem statement $x$, matching inference time, while the teacher policy conditions on both the problem statement $x$ and the GT solution $y^{\star}$:

$ \pi_{S}(\cdot \mid x) \triangleq \pi_\theta(\cdot \mid x), \qquad\qquad \pi_{T}(\cdot \mid x, y^{\star}) \triangleq \bar\pi(\cdot \mid x, y^{\star}). $

Given a problem $x$ paired with a GT solution $y^{\star}$, OPSD samples the full next-token distribution from both policies, then minimizes a token-level divergence between them:

$ \mathcal{L}{\mathrm{OPSD}}(\theta) =\mathbb{E}{(x, y^{\star})\sim\mathcal{S}}, \mathbb{E}{y\sim\bar\pi(\cdot\mid x)} \sum{t=1}^{|y|} D_{\beta}\big(\pi_{T}(\cdot\mid x, , y^{\star}, , y_{<t}), \big|, \pi_{S}(\cdot\mid x, , y_{<t})\big).\tag{4} $

Because the teacher sees $y^{\star}$, its next-token distribution along the student's trajectory encodes what a model that already knows the solution would do at each position, and matching it distills solution-conditioned behavior into the solution-free policy. Following the strongest OPSD configuration, we use $\beta = 0$ (forward KL, teacher $\rightarrow$ student) with per-token point-wise clipping for stability. Teacher and student share parameters, yet OPSD still requires external supervision, i.e., the reference solution $y^{\star}$.

Across Equation 3–Equation 4, external supervision enters as the gold answer $a^{\star}$, the external teacher $\pi_{T}$, or the reference solution $y^{\star}$.

3.2 u-OPSD: On-Policy Self-Distillation without Any Supervision

Despite on-policy training, these methods remain externally supervised. RLVR relies on gold answers for verification, OPD on a stronger teacher, and OPSD and related work on privileged teacher contexts such as GT solutions ([16]), demonstrations ([38]), or environmental feedback ([39]) (see the left of Figure 1). $\textsc{u-OPSD}$ removes this dependence by enabling the model to construct its own privileged context through internal consistency, i.e., a majority vote over its own rollouts that identifies a pseudo-solution in place of $y^{\star}$, while the resulting conditional teacher distribution is distilled into the student on the disagreeing rollouts (Figure 1, right).

(1) Sample. For each prompt, draw $G$ rollouts $y^{(1)}, \dots, y^{(G)} \sim \bar\pi(\cdot \mid x)$ at the training temperature and parse $a^{(g)}=\mathrm{Ans}(y^{(g)})\in\mathcal{A}\cup{\varnothing}$ (i.e., extract the final answer $a^{(g)}$ from boxed... with normalization and canonicalization; see Appendix C). $\varnothing$ marks rollouts without a parsable final answer (usually truncations), which we call invalid.

(2) Vote. Among the valid answers, we define the pseudo-answer $\tilde{a}(x)$ as the plurality vote, breaking ties uniformly at random ([28]). This vote partitions the rollouts into agreeing and disagreeing sets $\mathcal{Y}^{+}{x}={y^{(g)}:a^{(g)}=\tilde{a}(x)}, \qquad \mathcal{Y}^{-}{x} = {y^{(g)}:a^{(g)}\neq\tilde{a}(x)}$. Invalid rollouts belong to neither set, as an incomplete generation provides evidence of neither a correct nor an incorrect belief. We quantify vote confidence using the self-consistency score $c(x)=\frac{1}{G}\sum_{g=1}^{G}\mathbb{1}!\left[a^{(g)}=\tilde{a}(x)\right].$ The score is normalized by the total number of sampled rollouts $G$, rather than only by the number of valid rollouts where truncated generations reduce confidence. If $c(x)<\tau$, the prompt is treated as unlabeled and contributes no gradient in the training step. We set $\tau=\frac{1}{2}$ for an absolute majority.

(3) Distill. Given a valid majority vote, we select an agreeing rollout from $y^{+}\in\mathcal{Y}^{+}{x}$ as the teacher reference in place of $y^{\star}$ and distill the teacher into the student on the disagreeing rollouts $\mathcal{Y}^{-}{x}$. Substituting the pseudo solution $(x, y^{+})$ into Equation 4 yields

$ \mathcal{L}{\textsc{u-OPSD}}(\theta) ;=; \sum{i \in W_k} \sum_{t=1}^{|y^{(i)}|} D_{\beta}!\Big(\bar\pi\big(\cdot \mid x, y^{+}, y^{(i)}{<t}\big) ;\Big|; \pi\theta\big(\cdot \mid x, y^{(i)}_{<t}\big) \Big).\tag{5} $

The above skips two classes of samples: those for which the model cannot form a sufficiently consistent vote ($c(x)<\tau$), making the pseudo-answer unreliable, and those for which all valid rollouts already agree ($\mathcal{Y}^{-}_{x}=\emptyset$), leaving no disagreement to correct. Training therefore focuses on the model's competence frontier: prompts for which it can consistently identify a plausible solution but still assigns substantial probability to conflicting trajectories. This curriculum emerges directly from the model's own voting statistics, without external difficulty labels or scheduling.

Unlike OPSD, $\textsc{u-OPSD}$ requires no labeled pair $(x, y^{\star})$; its privileged context is constructed entirely from the model's own on-policy rollouts. Unlike conventional majority-vote self-training, it does not simply imitate the selected response under teacher-forced prefixes. Instead, it transfers the solution-conditioned next-token distribution along the model's own disagreeing trajectories, providing dense corrective supervision precisely at the prefixes that lead toward answers inconsistent with the model's consensus. In this way, $\textsc{u-OPSD}$ enables iterative on-policy self-evolution and self-improvement.

Require: prompt $x$; student $\pi_\theta$; detached teacher $\bar\pi$; rollouts $G$; threshold $\tau$
sample $y^{(1)},\ldots,y^{(G)}\overset{\mathrm{i.i.d.}}{\sim}\bar\pi(\cdot\mid x)$ // on-policy, training temperature
$a^{(g)}\gets\mathrm{Ans}(y^{(g)})$ for all $g$ // $\varnothing$ if unparsable
$\tilde{a}(x)\gets$ plurality of $\{a^{(g)}:a^{(g)}\neq\varnothing\}$; $c(x)\gets\frac{1}G\sum_{g}\mathbb{1}[a^{(g)}=\tilde{a}(x)]$
$\mathcal{Y}^{+}_{x}\gets\{y^{(g)}:a^{(g)}=\tilde{a}(x)\}$; $\mathcal{Y}^{-}_{x}\gets\{y^{(g)}:a^{(g)}\notin\{\tilde{a}(x),\varnothing\}\}$
if $c(x)<\tau$ or $\mathcal{Y}^{-}_{x}=\emptyset$ then
  return // vote not trusted, or nothing to correct
end if
select $y^{+}\in\mathcal{Y}^{+}_{x}$ and $\mathcal{B}^{-}_{x}\subseteq\mathcal{Y}^{-}_{x}$ // design choices; defaults: shortest agreeing, one uniform disagreeing
minimize $\displaystyle\frac{1}{|\mathcal{B}^{-}_{x}|}\sum_{y^{-}\in\mathcal{B}^{-}_{x}}\ \sum_{t=1}^{|y^{-}|} D_{\beta}\big(\bar\pi(\cdot\mid x,y^{+},y^{-}_{<t})\,\|\,\pi_\theta(\cdot\mid x,y^{-}_{<t})\big)$

4. Experiments

Section Summary: In the experiments, researchers tested their u-OPSD method on four variants of the Qwen3 language models using a 30,000-problem subset of math questions, training without access to correct solutions and evaluating on five challenging benchmarks such as AIME24, AIME25, and MATH500. The approach, which generates multiple model responses per question and distills from the most consistent ones, was compared against supervised baselines like SFT and GRPO as well as the original OPSD method, with separate runs in both thinking and non-thinking modes. Results showed u-OPSD delivering larger gains than the baselines—up to 10 points over base models in non-thinking mode and smaller but consistent improvements in thinking mode—while also boosting performance on larger instruction-tuned models without needing extra tuning.

4.1 Setup

Models and datasets. We adopt four variants of Qwen3 ([58]), including Qwen3-4B, Qwen3-8B, Qwen3-30B-A3B-Instruct-2507 and Qwen3-4B-Instruct-2507. For Qwen3-4B and Qwen3-8B, we train them in both non-thinking and thinking modes. In all experiments, the same checkpoint serves as both student and teacher. For the training set, we adopt a 30k subset of OpenThoughts ([59]) following OPSD; $\textsc{u-OPSD}$ uses only the problem statements, never the solution field. We compare against three supervised baselines trained on the same prompts: SFT on their gold solutions, GRPO with binary outcome rewards verified against the gold answer, and OPSD, which conditions the teacher on the gold solution.

Evaluation and benchmarks. We adopt five competition-level math reasoning benchmarks: AIME24, AIME25, and HMMT25 at avg@12, and MATH500 ([60, 61]) and AMC23 at avg@4. We follow OPSD's protocol for decoding using vLLM at temperature 1.0 with maximum generation length 38k, in the same reasoning mode the model was trained in.

For $\textsc{u-OPSD}$ and OPSD, we evaluate checkpoints every 25 steps up to 150 steps and report the best score. For GRPO and SFT, we follow OPSD's setup: for GRPO, we report the peak performance within 500 training steps. For SFT, we train on the same number of samples as OPSD. For TTRL, RENT and Intuitor, we adopt the same training setup as $\textsc{u-OPSD}$, i.e., training with the same rollout budget and the same prompt set.

Implementation details. We follow most of OPSD's training recipe ([16]): we use forward KL ($\beta{=}0$) over the full vocabulary with per-token pointwise clipping as the objective. We also adopt a teacher fixed to the initial policy rather than the running one to ensure fair comparison. Both choices are supported by ablations in [16]. We use LoRA of rank 64 ($\alpha{=}128$) on all attention and MLP projections, a learning rate of $5{\times}10^{-6}$, gradient-norm clipping at 0.1, and sampling at temperature 1.1 with top- $p$ 0.95 and top- $k$ 20. All runs are trained for 150 steps with a checkpoint every 25.

For $\textsc{u-OPSD}$, we introduce two unique hyperparameters: for each prompt, we generate $G{=}8$ rollouts independently, filtered by confidence threshold $\tau{=}0.5$ (default unless specified otherwise), where supervised OPSD uses 32 prompts and a single rollout. We also increase the maximum completion length from 1, 024 to 4, 096 tokens, as $\textsc{u-OPSD}$ requires rollouts to reach a boxed final answer; [16] report comparable performance between these two token budgets in their setting. For reasoning-mode experiments, following OPSD, we keep the teacher in reasoning mode and distill its behavior into a student in non-reasoning mode, while evaluating the resulting model in reasoning mode. For non-reasoning experiments, both the teacher and student are in non-reasoning mode during training, and evaluation is likewise conducted in non-reasoning mode.

::: {caption="Table 1: Performance comparison on math reasoning benchmarks for Qwen3 models with non-thinking mode."}

:::

4.2 Main results

::: {caption="Table 2: Thinking mode, per benchmark at step 150, under the protocol of Table 1. Shading and bold as in that table."}

:::

Non-thinking mode.

Table 1 reports the results in non-thinking mode. $\textsc{u-OPSD}$ achieves average scores of 49.49 and 54.31 on Qwen3-4B and Qwen3-8B, improving over the corresponding base models by 8.5 and 10.7 points. Despite using no ground-truth solutions, $\textsc{u-OPSD}$ outperforms all supervised baselines, including SFT, GRPO, and OPSD, exceeding OPSD by 3.2 and 2.3 points at the two model scales. These gains are consistent across benchmarks: $\textsc{u-OPSD}$ obtains the best result on four of the five benchmarks for both models. In contrast, the label-free RL baselines improve over the base models by at most 1.5 points under the same rollout budget. This comparison highlights the advantage of using consensus-derived solutions as privileged teacher context for token-level distillation, rather than reducing them to scalar rewards.

Thinking mode.

Table 2 presents the results in thinking mode. $\textsc{u-OPSD}$ reaches 77.05 on Qwen3-4B and 77.99 on Qwen3-8B, improving over the base models by 2.2 and 1.9 points, respectively. It matches or slightly exceeds supervised OPSD and consistently outperforms GRPO in average performance. The smaller gains relative to non-thinking mode suggest that consensus-based self-distillation is most effective when the base model is sufficiently reliable to produce informative consensus while retaining substantial room for improvement; the stronger thinking-mode models leave less such headroom.

Instruction-tuned models.

Table 3 further evaluates $\textsc{u-OPSD}$ on two instruction-tuned checkpoints using pass@1. $\textsc{u-OPSD}$ improves Qwen3-30B-A3B-Instruct-2507 from 75.77 to 77.46 and Qwen3-4B-Instruct-2507 from 67.00 to 68.78, achieving the highest average performance on both models. It also surpasses supervised OPSD by 1.1 and 1.7 points on average, though on Qwen3-4B-Instruct-2507 OPSD leads on HMMT25, MATH500 and AMC23. The result on Qwen3-30B-A3B-Instruct-2507 demonstrates that the same training recipe transfers effectively to a substantially larger mixture-of-experts model without model-specific hyperparameter tuning.

::: {caption="Table 3: Qwen3-30B-A3B-Instruct-2507, non-thinking, scored as pass@1 rather than average@ n : generation i of every problem forms one single-sample run, and i = 1, 2, 3 give three estimates, reported as mean ± population standard deviation. Each arm is shown at the checkpoint with the best five-benchmark mean under this metric. Because the metric differs from Table 1 and Table 2, the two are not comparable and the numbers are kept apart. Bold marks the best value in each column."}

:::

Training curves.

Figure 3 plots the first four checkpoints of the Qwen3-4B runs on AIME24, AIME25 and MATH500 in both modes. The two modes separate immediately. In thinking mode every method stays within about two points of the base model and the ordering changes from checkpoint to checkpoint, with the whole MATH500 panel spanning less than a point. The label-free baselines track the base model closely in both $\textsc{u-OPSD}$ and in OPSD. In non-thinking mode $\textsc{u-OPSD}$ is above every other method at every checkpoint on both AIME benchmarks, against the supervised and the label-free baselines alike, and its curves peak at step 50 rather than at the last checkpoint; on MATH500 the other methods catches up to the same level by step 100.

4.3 Ablation and analysis

Pseudo-label quality.

On 64 training prompts under the full training configuration ($G{=}8$, 4096-token budget, $\tau{=}0.5$), 96.3% of rollouts yield a parsable boxed answer, 94.0% of prompts clear the self-consistency threshold and receive a pseudo-label, and 86.7% of those pseudo-labels match the gold answer from data source (used in supervised methods). Fewer than 10% of valid rollouts disagree with their prompt's vote, so the distillation set is small and concentrated.

Divergence computation strategy.

Full-vocabulary logit distillation ([9]) evaluates $D_{\beta}$ over the entire vocabulary at each position, giving a proper token-level divergence between the two policies. Sampled-token distillation ([4]) evaluates teacher and student log-probabilities only at the token the student sampled, and uses that term as a scalar advantage inside a policy-gradient objective. OPSD ([16]) reports the full-distribution objective to be the stronger of the two under gold supervision, and because $\textsc{u-OPSD}$ changes the teacher's conditioning context rather than its objective, the open question is whether that preference survives once the privileged context is a pseudo-solution. Table 6 shows that it does, and by a wider margin: the full-distribution objective leads by 17.8 points on AIME25 and 7.5 on HMMT25, against 2.0 and 2.7 under gold supervision. Exposing the student to the entire teacher distribution therefore matters at least as much when that teacher is conditioned on the model's own consensus.

Self-consistency threshold.

The threshold $\tau$ decides which prompts are allowed to supervise. Figure 4 (left) sweeps it and the ranking is monotone: accepting every prompt ($\tau{=}0$) is best at 59.21, against 53.93 for the default and 44.40 at $\tau{=}0.9$ at step 150, with the best-checkpoint series ordered the same way. Accuracy falls monotonically as the threshold rises, so the filtering costs more than the label noise it removes.

Rollouts per prompt.

$G$, number of rollouts per prompt, controls both the resolution of the vote and the pool from which disagreeing rollouts are drawn, and it is the dominant cost term, since the sampling cost grows linearly in $G$. Figure 4 (middle) shows the return is real but soon saturating: $G{=}4$ and $G{=}8$ are indistinguishable at 56.99 and 57.10 at the best checkpoint, $G{=}12$ gains 4.7 points over the default, and $G{=}16$ gives back half of that. We keep $G{=}8$ elsewhere because it is OPSD's own generation budget and makes the comparison in Table 1 and Table 2 cost-matched, but $G{=}12$ is the better operating point when generation is not the constraint.

Teacher update.

OPSD ([16]) freezes the teacher at the initial policy, which under LoRA means evaluating the base weights with the adapter disabled. We report the frozen teacher elsewhere because it is OPSD's setting and keeps the comparison in Table 1 and Table 2 matched. Yet, Figure 4 (right) and Table 12 show that the EMA settings offer more room for improvement: the best, at decay 0.995, gains 2.4 points over the frozen default at the best checkpoint and 4.1 at step 150, and the two faster decays land within 0.2 points of each other at step 150.

::: {caption="Table 4: Performance under combinations of the teacher-reference selection (rows) and the distillation-target selection (columns), on Qwen3-8B non-thinking, G=8, tau=0.5, k=1. Each cell is the five-benchmark average at the best checkpoint in 25–150, with the step given in parentheses. 'label-only' strips the teacher's reference down to the boxed pseudo-label."}

:::

Teacher reference and distillation target.

Table 4 crosses which agreeing rollout the teacher is conditioned on against which disagreeing rollout is distilled. Both axes favour the longest rollout, and the best cell conditions the teacher on the longest agreeing rollout. When the shortest is distilled the three references fall within 0.8 points of each other. Further stripping the reference to the boxed pseudo-label costs 11.4 to 15.6 points and puts all three label-only cells at or below the base model's 43.57. Effective distillation requires the teacher to condition on the longer, full reasoning trace rather than only the final answer.

::: {caption="Table 5: Ablation of the disagreeing-rollout selection policy (matched decay schedule). Each cell shows step 150 / best checkpoint in 25–150. All variants use G=8, tau=0.5."}

:::

Selection of disagreeing rollouts.

Table 5 sweeps the number of disagreeing rollouts cap $k \in {1, 2, 3, \text{select all no cap}}$ and longest 1 rollout under random selection. Every capped variant exceeds the supervised baseline in every cell, so $\textsc{u-OPSD}$ 's effectiveness does not depend on the student-rollout selection tactic. Among them disagree-1 is the most consistent, leading in four of the five benchmarks tested at step 150, while disagree-3 attains the single best AIME25 checkpoint and ties longest-1 for the best on HMMT25, at the cost of larger swings. Removing the cap is clearly worst: disagree-all leads no column, and its step-150 HMMT25 score of 10.00 falls below the base model at 10.83, one of only two step-150 cells in the table below base.

::: {caption="Table 6: Comparison of divergence computation strategy: Full vocabulary is logit distillation over every token ([9]); sampled token evaluates the two policies only at the token the student drew ([4]); top- k rows truncate the teacher to its k largest entries. We report on Qwen3-8B non-thinking at the best checkpoint."}

:::

::: {caption="Table 7: Divergence family $D_b$eta under u-OPSD, on Qwen3-8B non-thinking, longest-1, G=8, tau=0.5. Each arm is shown at its best checkpoint in 25–150. Objective names follow [16], who report the same comparison under gold supervision."}

:::

Teacher vocabulary truncation.

Table 6 tests $k$, the number of teacher entries the token-level divergence is restricted to, from 20 to the full vocabulary. The best checkpoints fall under top-50/100/200 and do not improve further with the full vocabulary. The top- $k$ truncation is a practical optimization that reduces the cost of computing the divergence. The student token-only variant is the sampled-token objective of [4], which is 13.7 points below the default and 15.6 below the best checkpoint, so it is not competitive under pseudo-label supervision.

Divergence choice.

OPSD uses forward KL with $\beta{=}0$ and it rewards a student that concentrates on one mode of the teacher and tolerates dropping the rest. Table 7 shows the preference towards forward KL is stronger under $\textsc{u-OPSD}$. Using symmetric Jensen–Shannon divergence (JSD) costs 13.8 points, which puts it level with the untrained model, and reverse KL does not produce a converged model. The collapse under reverse KL is gradual and one-directional: generations average 2.7k characters at step 25, 12k at step 50, 76k at step 75, and 99k — the token ceiling — from step 125 on, while the fraction of rollouts that yield a parsable boxed answer falls from 99% to 33%. Inspection of the completions shows a loss of termination rather than cohesive reasoning (e.g., a phrase, a LaTeX{} command pair, or a nesting bracket repeated until the token budget runs out).

5. Limitations

Section Summary: The experiments are restricted to a single family of models and to competition math problems that have clear final answers, so the voting approach would need substantial changes to handle open-ended text generation. Gains from the method are much larger when the model is not using its built-in reasoning mode than when it is, and performance cannot exceed whatever answer the original model already favors most often. While multiple checks were used to reduce random variation in the results, full repeated runs with different random seeds have not yet been completed.

Scope. Our experiments cover one model family (Qwen3, at 4B and 8B and in both reasoning modes) and one domain (competition mathematics) with automatically checkable final answers. The voting mechanism requires an extractable, canonicalizable answer; extending $\textsc{u-OPSD}$ to open-ended generation would require replacing exact-match voting with a softer consensus.

The size of the advantage is regime-dependent. The gain over supervised OPSD is large in non-thinking mode and less in thinking mode. $\textsc{u-OPSD}$ exceeds OPSD by 3.2 and 2.3 points at 4B and 8B in Table 1, and by 0.9 and 0.02 points in Table 2; the improvement over the base model shrinks in the same way, from 8.5 and 10.7 points to 2.2 and 1.9. Thinking-mode base accuracy is already 74.9 and 76.1, and thinking-mode rollouts are far longer, so a 150-step budget covers many fewer completed votes per token spent. Readers should therefore treat consensus as a stronger replacement for gold solutions in the non-thinking regime we measured and as a match for them in the thinking regime, rather than as a uniform improvement on on-policy self-distillation.

Dependence on base-model competence. Majority-vote supervision reproduces whichever answer the base model already produces most often, so it is bounded by that answer's accuracy. Our probe measured 13.3% wrong pseudo-labels in-domain, and we did not measure training dynamics under deliberately corrupted votes. Mechanisms for detecting or down-weighting low-quality consensus (e.g., by vote margin) are natural next steps.

Variance and repeats. We mitigated variance by evaluating every checkpoint with 12 samples, spanning five benchmarks, and confirming the main effect under two schedules and three method variants, but seed-replicated error bars for every training setup are pending and will be added in a revision.

6. Conclusion

Section Summary: The researchers found that on-policy self-distillation works just as well, and sometimes better, when the model uses its own majority-vote answers in place of human-provided correct solutions. By treating consistent rollouts as the teaching signal and applying distillation only to inconsistent ones, the label-free approach improved performance over the supervised version across benchmarks and model sizes. This indicates that the main limit on progress is not the lack of external answers but the ability to detect and correct the model's internal disagreements.

We showed that the ground-truth solution in on-policy self-distillation can be replaced by the model's own majority-vote consensus: the agreeing rollout serves as the teacher's reference, and distillation is applied only to self-inconsistent rollouts. With the OPSD recipe otherwise untouched, this label-free variant outperforms its supervised counterpart on the five-benchmark average, and a grid over two model sizes and both reasoning modes shows the advantage reproduces at both scales in non-thinking mode and a tie in thinking mode, where the base model is already strong enough that little headroom remains. The result suggests that, in the regime of on-policy self-distillation—a base model competent enough to vote well and fallible enough to have headroom—the binding constraint is not access to gold solutions but the machinery for surfacing and correcting the model's own inconsistencies, a machinery that requires no supervision.

Appendix

Section Summary: The appendix supplies supporting checks and details for the main experiments. It first confirms that the authors' reproduction of prior OPSD results closely matches the published numbers, then tests whether GRPO performance gains depend on using the same reasoning mode during training and evaluation, finding that matched modes work better. It also describes the automatic answer-extraction process, gives hardware and runtime information, and presents additional ablation tables that break results down by benchmark.

A. Reproduction of OPSD

Reproduction check.

Before any unsupervised experiment we reproduced OPSD itself with the released code and hyperparameters. Across the 30 (checkpoint, benchmark) points published in the OPSD repository, our reproduction matches to within $\pm 4$ points avg@12 with no systematic direction (Table 8), consistent with sampling noise at 12 samples on 30-problem benchmarks. All supervised OPSD numbers below are from our reproduction under conditions identical to the unsupervised runs.

::: {caption="Table 8: Reproduction of published OPSD results with the released code and hyperparameters (avg@12, temperature 1.0). 'pub.' denotes the numbers published in the official OPSD repository; 'ours' our rerun. Checkpoint columns are steps 50/75/100/150, matching the checkpoints published for this configuration."}

:::

B. Does GRPO's gain survive a change of reasoning mode at evaluation time?

Every arm in Table 1 and Table 2 is trained and evaluated in the same reasoning mode (both off or both on), which leaves open whether a method's gain is a property of the model or of the mode it was tuned in. We ran GRPO in all combinations of modes, and its LoRA update is small enough that the model remains usable under either prompt format. We therefore take the two thinking-trained GRPO runs and score them again with the non-thinking prompt, against the non-thinking base. Table 9 reports the best five-benchmark performance of each combination.

::: {caption="Table 9: GRPO under matched and mismatched reasoning modes. Each row is the best of ten checkpoints (steps 50–500) by five-benchmark mean, scored at temperature 1.0 against the base model in the evaluation mode of that row. The two 'thinking → non-thinking' rows reuse the checkpoints of the rows above them; only the evaluation prompt differs."}

:::

At both scales the mismatched arm is beaten by the arm trained in the evaluation mode, by $1.8$ points at 4B ($+2.62$ vs $+4.43$) and by $2.3$ at 8B ($+1.20$ vs $+3.48$) — gaps larger than GRPO's entire same-mode gain in thinking mode ($+1.79$ and $+0.66$). Bold marks the better of the two non-thinking rows within each model, and the non-thinking-trained arm takes every contested column at 8B and four of five at 4B.

Two details are worth flagging for anyone reading the mismatched rows closely. First, the 4B mismatched peak rests on a single checkpoint: steps 50–450 all sit within one point of base ($39.98$ – $42.27$) and only step 500 jumps to $44.06$, so that $+2.62$ should not be read as a trend. The 8B mismatched arm is better behaved, clearing base at nine of ten checkpoints. Second, the transfer is not symmetric across benchmarks — AIME24 improves at every 8B checkpoint under the mismatched prompt, while AIME25 falls below base at six of ten. We did not run the reverse direction (non-thinking-trained checkpoints scored with the thinking prompt), so this appendix bounds mode dependence in one direction only.

C. Answer extraction and canonicalization

Answers are extracted from the last boxed... span with a brace-matching scanner. Specifically, the methods below are applied in order: stripping ... wrappers, thin-space and spacing commands, and trailing units; splitting trailing equalities (" $x = 2021$ " $\rightarrow$ " $2021$ "); normalizing frac, dfrac, and plain-slash fraction forms to a common representation; and mapping terminating decimals to canonical fractions (e.g., $0.5 \rightarrow \frac{1}{2}$). Two answers are considered equal if their canonical forms match exactly. On the training distribution this parser extracts an answer from 96.3% of rollouts at the 4096-token budget (Section 4.3).

D. Additional experimental details

Hardware and runtime.

All training runs use 8 GPUs (80 GB class) with colocated vLLM generation. An $\textsc{u-OPSD}$ run (150 steps, $G{=}8$, 4096-token budget) takes roughly 9 hours; supervised OPSD roughly 3.5 hours. Evaluating one (checkpoint, benchmark) cell takes 0.5–1.5 hours on 4 GPUs with vLLM (12 samples per problem on the 30-problem contest sets; 4 samples on MATH500/AMC23).

Skipped prompts.

With $G{=}8$ and $\tau{=}0.5$, roughly 94% of prompts receive a pseudo-label; among labeled prompts, those whose valid rollouts all agree contribute no distillation loss. The per-prompt distilled-row count is therefore at most $k$ and can be zero; batches are formed at the prompt level so optimizer steps remain synchronized across data-parallel workers.

Sequence budgets.

Student rollouts and teacher scoring use a 20,000-token maximum context. $\textsc{u-OPSD}$ raises the completion budget from OPSD's 1024 to 4096 tokens because voting requires completed answers; Section 4.3 quantifies extraction quality at this budget.

E. Additional ablation tables with per-benchmark breakdown

See Table 10 for the self-consistency threshold $\tau$ ablation, Table 11 for the number of rollouts per prompt $G$ ablation, Table 12 for the teacher update ablation, and Table 13 for the learning-rate ablation.

::: {caption="Table 10: Self-consistency threshold tau on Qwen3-8B non-thinking, longest-1, G=8. Each cell shows step 150 / best checkpoint in 25–150. tau is the fraction of valid rollouts that must agree for a prompt to receive a pseudo-label; tau=0 accepts every prompt."}

:::

::: {caption="Table 11: Number of rollouts per prompt G on Qwen3-8B non-thinking, longest-1, tau=0.5. Each cell shows step 150 / best checkpoint in 25–150."}

:::

::: {caption="Table 12: How the teacher is updated, on Qwen3-8B non-thinking, longest-1, G=8, tau=0.5. Each cell shows step 150 / best checkpoint in 25–150. 'fixed' freezes the teacher at the initial policy (the base model with the adapter disabled), which is OPSD's own setting and ours everywhere else; the EMA rows let the teacher track the student at the given decay."}

:::

::: {caption="Table 13: Learning-rate schedule ablation: best AIME24 checkpoint (avg@12) per method under an effectively-constant LR (30-epoch horizon, ≈ 5×1$0^-6$ throughout) vs. the matched 150-step linear decay used throughout Table 1 and Table 2. The constant-LR column is OPSD's native configuration, so the released figure is directly comparable there; no released run exists under the decayed schedule."}

:::

References

Section Summary: This references section compiles a bibliography of recent academic papers and technical reports, primarily from 2024 to 2026, that explore methods for enhancing reasoning abilities in large language models. The cited works cover topics such as reinforcement learning, knowledge distillation, self-training, and test-time computation, drawing from conferences like ICLR, ACL, and ICML as well as arXiv preprints. They represent foundational sources likely referenced throughout the main document to support discussions on efficient AI model development.

[1] Yixin Ye et al. (2025). LIMO: Less is More for Reasoning. In Second Conference on Language Modeling. https://openreview.net/forum?id=T2TZ0RY4Zk.

[2] Wen et al. (2025). Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track). pp. 318–327. doi:10.18653/v1/2025.acl-industry.24. https://aclanthology.org/2025.acl-industry.24/.

[3] Marah Abdin et al. (2025). Phi-4-reasoning Technical Report. https://arxiv.org/abs/2504.21318. arXiv:2504.21318.

[4] Lu, Kevin and Thinking Machines Lab (2025). On-Policy Distillation. Thinking Machines Lab: Connectionism. doi:10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation.

[5] MiniMax et al. (2025). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. https://arxiv.org/abs/2506.13585. arXiv:2506.13585.

[6] DeepSeek-AI et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948.

[7] 5 Team et al. (2025). GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. https://arxiv.org/abs/2508.06471. arXiv:2508.06471.

[8] Gu et al. (2024). MiniLLM: Knowledge Distillation of Large Language Models. In The Twelfth International Conference on Learning Representations. https://arxiv.org/abs/2306.08543.

[9] Agarwal et al. (2024). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=3zKtaqxLhW.

[10] Idan Shenfeld et al. (2026). RL's Razor: Why Online Reinforcement Learning Forgets Less. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=7HNRYT4V44.

[11] Ko et al. (2024). DistiLLM: Towards Streamlined Distillation for Large Language Models. In Proceedings of the 41st International Conference on Machine Learning. pp. 24872–24895. https://proceedings.mlr.press/v235/ko24c.html.

[12] Ko et al. (2025). DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. In Proceedings of the 42nd International Conference on Machine Learning. pp. 31044–31062. https://proceedings.mlr.press/v267/ko25a.html.

[13] Fu et al. (2026). Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. arXiv preprint arXiv:2603.25562. doi:10.48550/arXiv.2603.25562. https://arxiv.org/abs/2603.25562.

[14] Wu et al. (2026). Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation. arXiv preprint arXiv:2604.13010. doi:10.48550/arXiv.2604.13010. https://arxiv.org/abs/2604.13010.

[15] Wang et al. (2026). Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations. arXiv preprint arXiv:2607.13399. doi:10.48550/arXiv.2607.13399. https://arxiv.org/abs/2607.13399.

[16] Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv preprint arXiv:2601.18734.

[17] Yang et al. (2026). Self-Distilled RLVR. arXiv preprint arXiv:2604.03128.

[18] Ye et al. (2026). On-Policy Context Distillation for Language Models. arXiv preprint arXiv:2602.12275.

[19] Penaloza et al. (2026). Privileged Information Distillation for Language Models. arXiv preprint arXiv:2602.04942.

[20] Sang et al. (2026). CRISP: Compressed Reasoning via Iterative Self-Policy Distillation. arXiv preprint arXiv:2603.05433.

[21] Liu et al. (2026). HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation. arXiv preprint arXiv:2606.11559.

[22] Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300.

[23] Yue et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. arXiv preprint arXiv:2504.13837.

[24] Zhao, Xuandong and others (2025). Learning to Reason without External Rewards. arXiv preprint arXiv:2505.19590.

[25] Li et al. (2025). Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models. arXiv preprint arXiv:2506.06395.

[26] Prabhudesai, Mihir and others (2025). Maximizing Confidence Alone Improves Reasoning. arXiv preprint arXiv:2505.22660.

[27] Zhang et al. (2025). Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization. arXiv preprint arXiv:2504.05812.

[28] Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations (ICLR).

[29] Zuo, Yuxin and others (2025). TTRL: Test-Time Reinforcement Learning. arXiv preprint arXiv:2504.16084.

[30] Yuan et al. (2024). Self-Rewarding Language Models. arXiv preprint arXiv:2401.10020.

[31] Chen et al. (2024). Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. In International Conference on Machine Learning (ICML).

[32] Xu et al. (2025). Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning. In Annual Meeting of the Association for Computational Linguistics (ACL).

[33] Fang et al. (2025). SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data. arXiv preprint arXiv:2505.20347.

[34] Xiong et al. (2025). Self-Rewarding Correction for Mathematical Reasoning. arXiv preprint arXiv:2502.19613.

[35] Zhao et al. (2025). Absolute Zero: Reinforced Self-play Reasoning with Zero Data. arXiv preprint arXiv:2505.03335.

[36] Huang et al. (2025). R-Zero: Self-Evolving Reasoning LLM from Zero Data. arXiv preprint arXiv:2508.05004.

[37] Liu et al. (2025). SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning. arXiv preprint arXiv:2506.24119.

[38] Shenfeld et al. (2026). Self-Distillation Enables Continual Learning. arXiv preprint arXiv:2601.19897.

[39] Hübotter et al. (2026). Reinforcement Learning via Self-Distillation. arXiv preprint arXiv:2601.20802.

[40] Sang et al. (2026). On-Policy Self-Distillation for Reasoning Compression. arXiv preprint arXiv:2603.05433.

[41] Tan, Zhiquan and Hong, Yinrong (2026). Self-Supervised On-Policy Distillation for Reasoning Language Models. arXiv preprint arXiv:2605.17497.

[42] Ke et al. (2026). Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning. arXiv preprint arXiv:2605.13255.

[43] Liu et al. (2026). When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning. arXiv preprint arXiv:2605.21606.

[44] Zelikman et al. (2022). STaR: Bootstrapping Reasoning With Reasoning. arXiv preprint arXiv:2203.14465.

[45] Yuan et al. (2023). Scaling Relationship on Learning Mathematical Reasoning with Large Language Models. arXiv preprint arXiv:2308.01825.

[46] Gulcehre et al. (2023). Reinforced Self-Training (ReST) for Language Modeling. arXiv preprint arXiv:2308.08998.

[47] Huang et al. (2022). Large Language Models Can Self-Improve. arXiv preprint arXiv:2210.11610.

[48] Liang et al. (2026). Self-Evolving Visual Questioner. arXiv preprint arXiv:2606.13929.

[49] Furlanello et al. (2018). Born Again Neural Networks. In Proceedings of the 35th International Conference on Machine Learning. pp. 1607–1616.

[50] Yang et al. (2018). Snapshot Distillation: Teacher–Student Optimization in One Generation. arXiv preprint arXiv:1812.00123.

[51] Zhang et al. (2019). Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3713–3722.

[52] Caron et al. (2021). Emerging Properties in Self-Supervised Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9650–9660.

[53] Wu et al. (2025). SDRT: Enhance Vision-Language Models by Self-Distillation with Diverse Reasoning Traces. arXiv preprint arXiv:2503.01754.

[54] Jin et al. (2026). UniSD: Towards a Unified Self-Distillation Framework for Large Language Models. arXiv preprint arXiv:2605.06597.

[55] Prasad et al. (2024). Self-Consistency Preference Optimization. arXiv preprint arXiv:2411.04109.

[56] Shafayat et al. (2025). Can Large Reasoning Models Self-Train?. arXiv preprint arXiv:2505.21444.

[57] Yuan et al. (2024). Self-Rewarding Language Models. arXiv preprint arXiv:2401.10020.

[58] Yang et al. (2025). Qwen3 Technical Report. arXiv preprint arXiv:2505.09388.

[59] Guha et al. (2025). OpenThoughts: Data Recipes for Reasoning Models. arXiv preprint arXiv:2506.04178. doi:10.48550/arXiv.2506.04178. https://arxiv.org/abs/2506.04178.

[60] Hendrycks et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track.

[61] Lightman et al. (2024). Let's Verify Step by Step. In International Conference on Learning Representations (ICLR).