A Primer in Post-Training Reasoning Data: What We Know About How It Works

Yaoming Li$^{1}$, Guangxiang Zhao$^{1,3}$ $^{\dagger}$, Qilong Shi$^{2}$, Lin Sun$^{3}$, Xiangzheng Zhang$^{3}$, Tong Yang$^{1}$ $^{*}$
$^{1}$State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University
$^{2}$Tsinghua University
$^{3}$Qiyuan Tech
[email protected], [email protected], [email protected]
Project Repository: https://github.com/RenBing-Sumeru/Awesome-LLM-Reasoning-Data

$^{*}$ Corresponding authors.
$^{\dagger}$ Project lead.

Abstract

Post-training has become a primary driver of recent progress in large reasoning models, and reasoning data are often the key variable determining whether this stage succeeds. Work on post-training reasoning data has grown rapidly, yet this literature remains scattered across dataset papers, reinforcement-learning recipes, reward-model studies, benchmarks, and frontier system reports. This paper is the first primer to synthesize over 150 key public studies and system reports on post-training reasoning data. We organize the field around four questions: what data objects exist, what makes them useful, how they are constructed, and how they scale. Together, this organization provides an attribution framework for future reasoning-data releases and post-training recipes.

Executive Summary: Post-training has emerged as a major driver of progress in large language models that can reason step by step, yet the field lacks a shared framework for understanding how reasoning data actually works. Practitioners often rely on intuitive but flawed assumptions, such as the belief that longer reasoning traces or harder problems always produce better results. These gaps matter because post-training data now often influences model performance more than the choice of training algorithm.

This primer set out to synthesize the scattered literature on post-training reasoning data. It reviews more than 150 public studies, system reports, and benchmarks to answer four practical questions: what kinds of reasoning data exist, what makes particular items useful, how the data are built, and how reported gains should be attributed when models improve.

The authors organize the material around a verifier-anchored taxonomy rather than domain labels. They classify data by the kind of feedback that can be checked—programmatic rules, environmental states, or human or model judgments—and then examine how that feedback enters a trajectory. They also review construction practices across prompt sourcing, trace writing, search substrates, self-play anchors, and reward channels, drawing on public releases from systems such as DeepSeek-R1, Qwen3, and others.

The review surfaces several recurring patterns. Long chain-of-thought traces frequently rationalize answers rather than reveal the factors that actually drove them. A problem’s usefulness depends heavily on the base model, verifier, and sampling settings; items that are unreachable or already solved yield little signal. Coverage depends on source mixture, filtering rules, teacher lineage, and decontamination status, not simply on dataset size. Clean success traces in agent settings often erase the failures, retries, and state changes needed for credit assignment. Finally, scaling gains must be separated into changes in the reachable ceiling versus improvements in the speed of approach; many reported improvements reflect one but not the other.

These patterns imply that gains reported for new models or recipes are difficult to interpret or reproduce without detailed metadata. The same optimizer or scale can mask very different prompt pools, trace sources, verifiers, and evaluation budgets. Better attribution practices would reduce wasted effort on ineffective data and lower the risk of hidden contamination or judge-specific preferences.

Future releases should therefore report concrete fields such as base-model pass rates, verifier versions, filtering rules, lineage risks, replayable episodes for agent data, and explicit audit fields for each construction layer. A standard attribution ledger would make it possible to compare and test claims across papers.

The synthesis rests entirely on publicly available evidence. Proprietary pipelines, undocumented mixtures, and omitted audit details are necessarily absent. The work is a question-driven review rather than a formal meta-analysis, so readers should treat the framework as a practical reporting tool that will need updating as new agentic and multimodal settings appear.

1. Introduction

Section Summary: The training of large language models occurs in two main stages, with post-training growing especially important for improving capabilities through better data rather than new algorithms. Despite data's central role, no prior survey has systematically examined how post-training data is built and used, so this paper reviews 150 studies to provide the first comprehensive overview and practical guidance. It focuses on reasoning data, cautioning against common misconceptions about what makes data effective and framing each example as a detailed record that links problems, model actions, feedback, and explanations for performance gains.

The training of large language models typically consists of two stages: pre-training and post-training. With the rise of o1-style test-time scaling paradigms and thinking models, post-training has become increasingly important for further advancing model capabilities ([1, 2]). Recent studies suggest that, within the post-training pipeline, the quality and construction of training data often have a greater impact on model performance than other components, such as training algorithms or optimization strategies ([3, 4, 5]). However, despite the central role of data in post-training, there has not yet been a systematic survey dedicated to post-training data. This paper fills this gap. To the best of our knowledge, it is the first survey focused on post-training data, reviewing 150 key papers in this area and synthesizing practical insights into how post-training data is constructed, curated, and utilized.

::: {caption="Table 1: Counterintuitive lessons for reasoning-data attribution. The table distills recurring traps in post-training reasoning-data work, why they fail, links each takeaway to the section where it is developed, and lists the future reporting fields needed to make gains attributable."}

:::

**Figure 1:** **Beyond prompt–response pairs.** A reasoning-data item packages a problem or state, model behavior, judging feedback, and attribution metadata. The map previews four questions: objects, usefulness, construction, and gain attribution.

Post-training reasoning data is therefore not just a collection of datasets, but a set of evidence-bearing records for explaining why a model improves. As summarized in Table 1, many intuitive assumptions about reasoning data are misleading: long CoT traces do not necessarily imply faithful reasoning ([6, 7, 8]); harder data is useful only relative to a base model and verifier ([4, 9, 10]); larger corpora may still provide poor coverage when source mixture, leakage, or lineage are uncontrolled ([3, 5, 11]); and, in agentic settings, overly cleaned success trajectories can remove precisely the failures, retries, and state changes needed for credit assignment ([12, 13, 14]). These pitfalls reveal a missing common language for connecting data objects, feedback signals, construction recipes, optimizers, base models, scaffolds, and budgets. We organize the primer around four questions—what reasoning-data objects exist, what makes them useful, how they are constructed, and how their gains scale. As Figure 1 shows, this framing moves beyond prompt–response pairs and treats each item as a feedback-bearing record linking task context, model behavior, judging feedback, and attribution metadata. The goal is not just to enumerate datasets, but to help readers inspect future releases and attribute reported gains.

2. What Data Do We Need? A Verifier-Anchored Taxonomy

Section Summary: The section argues that reasoning datasets should be organized not by topic or domain but by their "verification contract"—the specific mechanism that allows outputs to be checked and the feedback signal to be made trainable and auditable. It outlines three main contract types: programmatic verification for automatically checkable results such as code or proofs, environmental verification for interactive tasks with tools or systems, and judgment-based verification for cases requiring human-like evaluation. The authors add cross-cutting considerations such as where in a trajectory feedback is applied, how behavior boundaries are defined, and how synthetic data lineages are tracked, showing that these factors determine what a model can actually learn from a given sample.

Reasoning corpora are usually named by domain, but the learning signal is defined by what can be checked. We therefore sort data by verification contract: where feedback enters, what becomes trainable, and which fields make the signal auditable. The unit is a verifier-bearing sample, not a domain-labelled prompt–response pair ([22, 23]). Figure 2 gives the taxonomy as an attribution lens: it specifies what supervision a sample exposes, not whether the trained model later uses that supervision at inference time.

**Figure 2:** **Verifier-anchored taxonomy.** Verification contracts, rather than domains, define the native signal, trainable object, failure surface, and required audit fields of a reasoning sample.

2.1 Three Verification Contracts

Programmatic verification.

Programmatic contracts make artifacts checkable: answers can be normalized, code executed, and proof states verified, enabling scalable math, code, and Lean corpora ([24, 25, 26, 27]). Signals are local: verifiers may certify a final artifact without certifying robustness, non-spuriousness, or movement beyond the base policy ([28, 29, 30]). Since small curated sets can elicit latent skills ([31, 32]), releases should report checked object, extractor, filter, and base-conditioned difficulty.

Environmental verification.

Environmental contracts make interaction trainable. Tool, web, app, OS, and repository tasks expose goals, actions, observations, files, state transitions, and terminal predicates ([12, 33, 34, 35, 36, 37, 38, 39]). Their strength is state-based success; their failure is transcript compression. Clean SFT traces often remove failed actions, retries, recoveries, and sibling rollouts—the branches where credit assignment is visible. Such data should be replayable episodes, not only successful transcripts.

Judgment-required verification.

When no deterministic verifier exists, the reusable unit is an auditable judgment record. Medical, factuality, safety, and rubric-reward datasets attach criteria, evidence, risk, provenance, judge traces, or disagreement fields ([40, 41, 42, 43, 44]). This expands scope, but makes the judge part of the data object: hidden prompts, rubrics, calibration sets, or judge identities can turn gains into judge-specific preferences ([45]). Releases should ship the judgment contract itself.

2.2 Three Cross-Cutting Axes

The contract says what verifies a sample; the next question is where that feedback enters the trajectory. Figure 3 shows four entry points: feedback on the final answer, an intermediate reasoning step, a state–action transition, or the full episode. Moving the feedback point changes the trainable object even when the underlying task is the same.

**Figure 3:** **Where supervision enters the trajectory.** The same task becomes different post-training data when feedback targets the final answer, an intermediate step, a state–action transition, or the full episode.

Supervision granularity.

Outcome labels check endpoints; step labels localize valid reasoning; continuation scores estimate stability; formal and tool traces make operations reproducible ([22, 23, 46, 47, 48]). This separates capability from mechanism: accuracy may show that a task is solved, while the teachable signal may live in steps, rollouts, or execution.

Behaviour bounds.

Behaviour-bounding data define when to answer, refuse, abstain, provide safe completion, or expose a trace ([49, 50, 43, 51]). The caution is that abstention, verbosity, or exposed chains need not imply reliability ([7, 8]); releases should separate intent, risk, allowed content, response type, trace visibility, and mismatch.

Cross-generational lineage.

Synthetic flywheels make data cheap, but what propagates may be trace style, decoding policy, filter, or teacher preference rather than the question distribution ([3, 52]). Scale can coexist with narrowing, hidden trait transfer, leakage, or search contamination ([11, 53, 54, 55]). Lineage should therefore be sample-level metadata.

Together, these axes explain why contract labels alone do not determine how the signal behaves. A sample can be programmatically verified but answer-only, environmentally verified but scaffold-inherited, or judgment-required but shaped by teacher lineage.

The taxonomy yields four downstream gaps: value is base-relative; environmental signal may live in branches and failures; verifier errors must be attributable to the model, verifier, scaffold, or schema ([30, 56]); and leakage must be audited beyond answer strings. They motivate the next questions: what makes a sample useful, how it is built, and how scaling claims should be attributed.

3. What Makes Reasoning Data Good?

Section Summary: Reasoning data quality cannot be reduced to simple properties like answer strings or trace length, because correctness, difficulty, trace validity, and coverage are each relative to a different piece of supporting evidence. Correctness depends on the exact verifier contract including extraction, normalization, and known failure modes, while difficulty is only meaningful relative to a given base model, prompt format, and sampling regime. Traces must be audited for local validity and grounding rather than length, and any release needs to document its full lineage to control leakage and inheritance effects.

Section 2 treated reasoning data as verifier-bearing samples; the quality question is therefore which attached fields license a claim about the sample. Recent releases expose different parts of this claim: DeepMath-103K and DAPO make extraction and rule verification operational ([4, 10]); PRM800K and Math-Shepherd move supervision into steps and rollouts ([22, 23]); Skywork-OR1 makes difficulty model-aware ([57]); and OpenThoughts makes data construction ablatable ([3]). Figure 4 summarizes the view: correctness is verifier-relative, difficulty is base-relative, trace quality is trajectory-relative, and coverage is lineage-relative.

**Figure 4:** **Quality support matrix.** No single field licenses all quality claims: correctness, difficulty, trace quality, and coverage each require different verifier, base, trajectory, and lineage evidence.

3.1 Correctness Is Verifier-Relative

Correctness is a versioned verifier contract, not an answer string. DeepMath-103K and DAPO show that extraction, normalization, rule checking, invalid-output handling, and pass-rate filtering determine which rollouts become trainable ([4, 10]). Yet verifiers are also failure surfaces: rule verifiers can reject equivalent answers, and model judges can be steered by superficial cues or verifier-gaming policies ([58, 59, 45, 30]). Correctness claims should therefore report the extractor, verifier version, filtering rule, and known failure modes.

3.2 Good Traces Are Not Just Long Traces

A visible trace is suggestive evidence, not proof of mechanism. Rationales may omit the factors that drove the answer or rationalize decisions shaped by biased features ([8, 7]). Figure 5 makes the distinction explicit: useful traces are auditable, locally valid, task-relevant, and grounded in labels or execution context.

$**Figure 5:** **Two common quality traps: long trace $\neq$ good trace, hard item $\neq$ useful item.** Trace quality depends on validity and grounding, while useful difficulty lies between always-fail and always-pass regimes for a given base and sampling protocol.$

Process supervision makes intermediate validity observable, but it does not remove the need for audit: PRM800K, Math-Shepherd, and OmegaPRM differ in whether the signal comes from human labels, rollout values, or refined process estimates ([22, 23, 46]). ProcessBench and PRMBench further show that PRMs must be tested for localization, soundness, sensitivity, and robustness ([60, 61]). For tool traces, sandbox, timeout, package version, output, and execution lineage are part of the sample ([24]).

3.3 Difficulty Is Base-Relative

Difficulty is an item–model–sampling relation. Figure 5 illustrates the corresponding trap: a hard item is not necessarily a useful item, because it may be unreachable for one base, gradient-producing for another, and saturated for a third.

Skywork-OR1 estimates difficulty against model variants, while DAPO implements the same idea by removing prompt groups with accuracy $0$ or $1$ ([57, 10]). Thus "medium difficulty" hides the base model, prompt format, rollout count, temperature, and verifier. Because scores can shift with seeds, rollouts, and token budgets, and because RLVR may reweight base-supported trajectories rather than expand the frontier, difficulty should be released as a dated base-conditional estimate ([21, 29]).

3.4 Coverage, Leakage, and Lineage

Coverage is a recipe, not a count. OpenThoughts ablates source, mixture, filter, generator, and teacher choices ([3]); Structure Trumps Size shows that data structure can matter more than conventional cleanliness or scale ([5]). Synthetic data make this recipe inherit across generations: corpora can narrow recursively, transmit hidden teacher traits, leak private fields through traces, or contaminate benchmarks through search channels ([11, 62, 53, 55]). A useful release should therefore report generator, filter, split, teacher, verifier of record, decontamination status, and known inheritance risks.

4. How Is Post-Training Reasoning Data Built?

Section Summary: Post-training reasoning data is assembled through several interconnected layers that shape both the content and the credit assignment for model improvements. These layers include sourcing and filtering prompts from existing problem pools, generating detailed reasoning traces often with teacher models or process supervision, embedding tasks in executable environments for search and tool use, applying self-play or curation rules, and defining reward signals through verifiers or learned models. Reports on recent systems make some of these steps visible, yet the paper stresses that accurate attribution of performance gains requires tracking which components are actually modified versus merely reported.

Public reports have made the RL scaffold increasingly visible: long-CoT SFT, distillation, RLVR, reward modelling, and GRPO-style optimization recur across DeepSeek-R1, Kimi K1.5, Qwen3, Magistral, Phi-4-reasoning, and Llama-Nemotron ([2, 63, 64, 65, 66, 67, 18]). But optimizer visibility is not causal isolation. DAPO changes filtering and loss geometry, MiniMax-M1 changes importance-weight control, and Qwen3-Coder-style systems move signal into long-horizon tool interaction ([10, 68, 69]). Table 2 therefore treats reasoning-data construction as an attribution ledger: each upstream layer exposes a visible field, hides a different confound, and determines which metadata are needed to interpret the final gain.

\begin{tabularx}{\cclcmcccccc}{@
>{\raggedright\arraybackslash}p{0.27\columnwidth}
>{\raggedright\arraybackslash}X
ccccc@}
\toprule
\textbf{Build layer} 
& \textbf{Attribution handle} 
& \textbf{Q} & \textbf{T} & \textbf{E} & \textbf{V} & \textbf{L} \\
\midrule
Prompt sourcing
& problem support / pass-rate band
& $\checkmark$ & -- & -- & $\triangle$ & $\triangle$ \\

Trace writing
& inherited reasoning style
& $\triangle$ & $\checkmark$ & -- & $\triangle$ & $\checkmark$ \\

Search substrate
& exploration and replayability
& $\triangle$ & $\triangle$ & $\checkmark$ & $\checkmark$ & $\triangle$ \\

Self-play anchor
& where curation re-enters
& $\checkmark$ & $\triangle$ & $\triangle$ & $\checkmark$ & $\checkmark$ \\

Reward/verifier
& what counts as success
& $\triangle$ & $\triangle$ & $\triangle$ & $\checkmark$ & $\checkmark$ \\

Frontier pipeline
& where reports appear to converge
& $\checkmark$ & $\checkmark$ & $\checkmark$ & $\checkmark$ & $\checkmark$ \\
\bottomrule
\end{tabularx}

4.1 Where Do Reasoning Prompts Come From?

Prompt sourcing is where a recipe first becomes data-specific. OpenThoughts makes source, mixture, filtering, answer generation, and teacher choice explicit ablation variables ([3]); math and code corpora turn olympiad pools, web-mined problems, executable tests, and model-conditioned solve rates into prompt-side fields ([9, 24, 4, 70, 25]). Small-set results such as LIMO and s1 provide the counterpoint: few demonstrations can elicit strong behaviour only when the skill already lies within the base policy's support ([32, 31, 29, 71]). Prompt releases should therefore report source, filter, verifier, per-base pass rate, and estimate date, not only domain and count.

4.2 Who Writes the Trace?

Trace construction fixes the author, process field, and grounding of the trajectory. Teacher traces can transfer decomposition style, uncertainty expression, formatting, tool use, and stopping behaviour, not merely answers ([3, 72, 66, 48]). Process-supervised variants make hidden structure more explicit: PRM800K labels steps, Math-Shepherd estimates rollout value, OmegaPRM searches for first errors, PRIME derives implicit process reward, and PROF filters incoherent chains ([22, 23, 46, 73, 74]). The dispute is therefore not whether traces matter, but whether outcome-only RL, explicit PRMs, or trainable verifiers expose the relevant process field ([75, 76]).

4.3 Where Do Search-Time Gains Live?

Agentic reasoning moves search from decoding into an executable substrate. Repository, web, app, OS, and tool benchmarks define samples through states, actions, observations, files, tests, and terminal predicates ([13, 39, 36, 37, 14, 47]); theorem-proving systems collapse action and verification into Lean-state transitions ([27, 77]). The lesson is not that every task needs an environment, but that environment data should be replayable. Successful transcripts alone erase failed actions, recoveries, retries, state diffs, and hidden predicates—the branches where credit assignment is visible.

4.4 Can Self-Play Eliminate Curation?

Self-play does not eliminate curation; it moves curation into the anchor that turns generated behaviour into trainable feedback. STaR anchors rollouts to external answers, R-Zero separates Challenger and Solver roles, Absolute Zero uses the Python interpreter, TTRL turns test-time majority into reward, and multi-agent variants distribute these functions across roles ([15, 78, 16, 17, 79]). These systems differ less in agent count than in what makes a trajectory admissible: answer availability, executable feedback, majority selection, role-mediated challenge, or archive-based evaluation.

The optimistic reading is autonomous curriculum growth. The caution is that anchors also define support bounds and label reliability: on-policy RLVR may conservatively reweight trajectories already supported by the base policy, and pseudo-label accuracy can degrade as generated tasks harden ([71, 78]). AlphaEvolve illustrates the counter-pressure: discovery is auditable precisely because the evaluator and evolutionary archive remain explicit anchors ([80]). Self-play releases should therefore report the anchor, selection rule, verifier, replay policy, and failure modes, not only generated tasks or final success rates.

4.5 What Does the Policy Train Against?

The reward channel is also a data object: a "reward" may be a formal checker, process verifier, learned reward model, rubric judge, or closed-loop selection rule, as summarized in Figure 6.

**Figure 6:** **Verifier families and failure surfaces.** A reward channel is itself a data object: formal checkers, process verifiers, learned reward models, rubric judges, and implicit selectors expose different use cases, failure modes, and audit fields.

Master-key attacks, spurious rewards, GSM-Symbolic perturbations, and verifier-gaming tests all show that reward signals can be broad yet brittle ([45, 81, 82, 30]). Rubric systems extend the same problem to domains without symbolic checking: HealthBench, RaR, OnlineRubrics, Omni-RM, AutoRubric, RubricArm, and PoP make criteria, elicitation, modality, and judge-policy interaction part of the reward-data pipeline ([40, 83, 84, 85, 86, 87, 88]). CoVerRL and DeepSeekMath-V2 suggest co-evolving generators and verifiers, but co-evolution raises the need for versioning rather than removing it ([89, 76]).

4.6 What Does a Frontier Pipeline Look Like?

A frontier pipeline is an orchestration of the fields above. Distill-then-RL systems buy cold-start speed through teacher traces; small-warmup multi-stage RL systems let RL reshape the policy; pure-RL lines reduce teacher inheritance but increase dependence on verifier and prompt support ([66, 67, 2, 64, 65, 90]). Agent-native systems add tool calls, execution feedback, long contexts, and software environments to the post-training distribution itself ([91, 68, 69]). The optimizer is therefore the wrong final unit of comparison: a reported gain is interpretable only after the prompt support, trace teacher, substrate, anchor, verifier, scaffold, and inference budget are declared.

5. Scaling: Asymptotes on the Data Substrate, Efficiency on the Optimiser

Section Summary: Recent scaling analyses treat performance gains not as a uniform curve but as a split between the highest reachable level and the speed of getting there. Choices around data sources, environments, and verifiers tend to raise or lower that ceiling, while adjustments to loss functions, reuse rates, and sampling mainly speed up progress toward whatever limit is set. The result is that reported improvements must be read by asking which constraint was actually relaxed rather than assuming every change expands capability itself.

Recent scaling sweeps make reasoning post-training look less like a single law than a ledger of what changed. [19] separate asymptotic performance from compute efficiency in large-scale RL runs, while [20] show that model size, compute, and data reuse interact across the Qwen2.5 dense series. We use their equations as a reading device rather than a universal law:

$ \begin{gathered} R(C) = R_0 + \frac{A-R_0}{1+(C_{\mathrm{mid}}/C)^B} \ \log L(N, C) = E(N) - k(N)\log C \end{gathered}\tag{1} $

Here $A$ is the reachable ceiling, while $B$ or $k(N)$ captures approach efficiency. A benchmark gain is therefore not self-explanatory: it may move the ceiling, improve approach speed, or change the measurement surface.

**Figure 7:** **Asymptote–efficiency decomposition.** A scaling result can change what the data substrate makes reachable, how efficiently training approaches that frontier, or both.

\begin{tabularx}{\cclcmcccccc}{@
>{\raggedright\arraybackslash}p{0.34\columnwidth}
cc
>{\raggedright\arraybackslash}X@}
\toprule
\textbf{Scaling knob} 
& \textbf{$A$} 
& \textbf{$B/k$} 
& \textbf{Release fields} \\
\midrule
Prompt/filter pool  
& $\checkmark$ & $\triangle$  
& source, filter, pass-rate band \\

Loss / clipping / entropy 
& $\triangle$ & $\checkmark$ 
& objective, KL, batch, weighting \\

Reuse / epochs  
& $\triangle$ & $\checkmark$ 
& $D_{\mathrm{unique}}$, $\tau$, replay rule \\

Search / TTT topology  
& $\triangle$ & $\checkmark$ 
& budget rule, selector, verifier, metric \\

Environment substrate  
& $\checkmark$ & $\triangle$  
& state, actions, replay, terminal check \\

Verifier refresh  
& $\checkmark$ & $\checkmark$ 
& verifier card, attacks, refresh log \\
\bottomrule
\end{tabularx}

5.1 Asymptotes and Efficiency

The useful commonality between the Khatri and Tan laws is the separation in Figure 7: some choices change the reachable frontier, while others change the path toward it ([19, 20]). Table 3 gives the audit reading. Data substrate, verifier quality, support coverage, context, architecture, and search topology can move the ceiling; loss design, sampling, rollout budget, curriculum, precision, and warm-start distillation more directly alter efficiency.

This distinction also organizes the RLVR debate. [29] and [71] read current RLVR as sharpening trajectories already accessible to the base policy. The counter-literature relaxes different closures: ProRL extends horizon length ([92]); RL-PLUS injects external rollouts ([93]); CoT-Pass@K changes the success metric ([94]); and PASS@ $(k, T)$ adds interaction depth ([95]). Thus "RL expands capability" is not a scalar claim; it specifies which closure was broken.

5.2 Data Uniqueness and Stage Composition

Tan et al.'s decomposition $D_{\mathrm{total}}=D_{\mathrm{unique}}\times\tau$ makes reuse part of the scaling surface rather than a preprocessing detail ([20]). Figure 8 reframes the small-pool versus large-pool debate: what matters is whether examples fall inside the gradient-producing band for a given base model.

**Figure 8:** **Small-pool versus large-pool coverage.** Small curated pools can be effective when they repeatedly sample the active band of a capable base; larger pools matter when useful gradients lie in the tail or must cover multiple bases, verifiers, or domains.

Small-pool releases such as LIMO and s1 show hundreds or thousands of examples can elicit strong behaviour when the base already supports the skill ([32, 29, 71]). OpenThoughts and Big-Math expose the opposite regime, where source mixture, filtering, teacher choice, and base-conditional solve rates determine coverage ([3, 9]). Cross-base studies sharpen the point: cognitive behaviours and even spurious rewards are base-conditional rather than generic ([96, 81]). Distillation and stage composition decide what enters RL before scaling is measured, since teacher traces, grounded thoughts, mid-training, SFT, and RL prefer different quality–diversity trade-offs ([97, 98, 99]). Scaling is therefore parameter count under a base prior, uniqueness budget, teacher lineage, and stage order.

5.3 Inference-Time Compute and Trace Topology

Inference-time scaling changes the observable, not merely the decoding setting. Repeated sampling can scale coverage when candidates are verifiable ([100]); s1 makes budget forcing an explicit intervention on thinking length ([31]); and parallel-distill-refine, TTRL, and MiGrATe make the deployment-time selector, verifier, or mixed policy part of the reported data object ([19, 17, 101]). Long-context reasoning adds another topology: more context can raise the ceiling while reducing efficiency, whereas Markovian Thinking changes the environment so reasoning proceeds through fixed-size chunks and learned state ([19, 102]). A pass@N score under serial extension, repeated sampling, parallel refinement, test-time training, or Markovian chunking is therefore not the same observable.

5.4 Verifier Scaling and Release Lineage

The verifier scales with the policy, but not necessarily in the right direction. Master-key attacks, GSM-Symbolic perturbations, and verifier-robustness studies show that reward signals can be broad yet brittle: LLM judges may reward trigger tokens, math accuracy may collapse under surface changes, rule verifiers can false-reject, and learned verifiers can become hackable ([45, 82, 59]). Process-verifier corpora follow the same logic: PRM800K, Math-Shepherd, OmegaPRM, and Qwen2.5-Math-PRM differ in whether supervision comes from human labels, Monte-Carlo scores, first-error localization, or judge consensus ([22, 23, 46, 103]). CoVerRL and DeepSeekMath-V2 suggest co-evolving generators and verifiers, but co-evolution increases the need for versioning rather than removing it ([89, 104]).

The same versioning problem appears in benchmark reporting. Scores can shift with seed and budget ([21]); search agents can retrieve benchmark-adjacent answers during evaluation ([55]); and teacher traits can pass through data even when semantic cues are stripped ([62]). Release time is therefore not a neutral timestamp but a proxy for a versioned data object: prompt pools, verifier versions, search budgets, contamination audits, teacher lineages, evaluation protocols, and refresh cadences may all have changed. A useful scaling report should disclose these fields before attributing a score difference to data scale, optimizer efficiency, or model size.

6. Conclusion

Section Summary: Recent progress in reasoning models relies heavily on specialized post-training data that is now being released at a rapid pace. The key reusable element is no longer a simple prompt and answer pair but a broader feedback system that includes a verifier and depends on many interacting factors such as the base model and training setup. The central open problem is figuring out which of those factors actually produces gains, and this overview seeks to make such improvements easier to track and test.

Post-training reasoning data have become the hidden substrate of reasoning-model progress, and the stream of releases is accelerating. The evidence reviewed here suggests that the reusable unit is not a prompt–response pair, but a verifier-bearing feedback interface whose value depends on the verifier, base model, lineage, optimizer, scaffold, and inference budget. The central unresolved question is therefore attribution: when a model improves, which part of that interface changed? This primer helps the community move from reporting gains to making them inspectable, comparable, and testable.

7. Limitations

Section Summary: This primer is constrained by the limited public information available on post-training reasoning methods, since many pipelines remain proprietary and reports frequently omit crucial details such as data sources, compute budgets, and contamination checks. The authors relied on a question-driven selection of studies rather than a formal meta-analysis, drawing from a mix of papers, preprints, and model cards without independently verifying results or running experiments themselves. As a result, the taxonomy is presented as a flexible reporting framework that will need ongoing updates as new techniques and settings emerge.

This primer is limited by the public evidence available for post-training reasoning data. Closed pipelines, proprietary data mixtures, and undocumented release practices are necessarily absent, and many public reports omit lineage cards, verifier versions, compute and inference budgets, and contamination audits. Our synthesis is also question-driven rather than a formal meta-analysis: we include work that exposes reasoning-data objects, feedback interfaces, construction recipes, or scaling surfaces, which may underrepresent work that reports only aggregate benchmark gains or uses non-public artifacts. In addition, the evidence base is heterogeneous, spanning peer-reviewed papers, arXiv preprints, technical reports, benchmarks, and model cards; we use these sources to document public recipes and release practices, not to establish causal rankings. Finally, we do not independently re-run training recipes, audit contamination, or validate every verifier. The taxonomy should therefore be read as an attribution and reporting framework whose fields will need updating as reasoning post-training expands to new multimodal, multilingual, agentic, and co-evolving verifier-generator settings.

8. Ethics Statement

Section Summary: This paper does not release any new data or models. The approaches it examines do touch on sensitive topics including private user information, expert guidelines, safety limits, autonomous actions by AI agents, and expensive training runs. For that reason, future work in this area will need to pay close attention to matters such as licensing, data origins, the well-being of human reviewers, and clear reporting of computing resources.

This work releases no new data or models. Still, the practices surveyed involve privacy-bearing traces, expert rubrics, safety boundaries, agentic actions, and high-cost RL pipelines, making license, provenance, annotator welfare, verifier hacking, contamination, and compute disclosure central ethical requirements for future releases.

Appendix

Section Summary: The appendix supplies supporting figures, tables, and notes that expand on the primer’s discussion of reasoning post-training without adding essential new arguments. It describes the overall scope and selection rules, explaining that sources are included only when they explicitly release or detail data objects, verifiers, construction recipes, or scaling information, while non-peer-reviewed reports are used mainly for context rather than causal claims. Additional coverage tables then list clusters of related work in areas such as math, code, agent environments, and scientific reasoning to show the survey’s bibliographic boundaries.

A. Supplementary Figures and Audit Notes

This appendix provides supplementary material for the primer. The main text is self-contained; the figures and tables below provide expanded schematics, audit fields, source-placement rules, and a brief AI-assistance disclosure. The technical material follows the order of the main text: verification contracts, construction layers, scaling attribution, agent-trajectory audit fields, and source placement.

A.1 Scope, Methodology, and Inclusion Criteria

This primer is a question-driven synthesis rather than a formal meta-analysis. We focus on public work that makes at least one component of reasoning post-training data explicit: the data object, verifier, trace source, reward channel, environment, scaling rule, or release metadata. The goal is not to exhaustively enumerate all reasoning datasets, but to identify the recurring feedback interfaces that determine whether a reported gain can be attributed.

We included work when it satisfied at least one of four criteria. First, the work releases or describes reasoning-oriented post-training data, including math, code, theorem proving, tool use, agent trajectories, medical or safety reasoning, and rubric-based evaluation. Second, it exposes a verifier, reward model, process label, environment predicate, judge, or feedback rule that determines what counts as success. Third, it describes a construction recipe, such as prompt sourcing, trace generation, distillation, filtering, self-play, or verifier refresh. Fourth, it contributes scaling evidence for reasoning post-training, including data reuse, test-time compute, RLVR, verifier scaling, or inference-budget effects.

We excluded work that only reports leaderboard performance without describing the data object, feedback interface, construction recipe, or evaluation budget. We also treat technical reports, arXiv preprints, benchmark papers, and model cards differently from peer-reviewed empirical papers: non-peer-reviewed sources are used primarily for documenting public recipes and release practices, not for settling causal claims. This distinction is important because the paper's central question is not simply whether a model improves, but which data object, verifier, scaffold, or inference budget carries the improvement.

\begin{tabularx}{\cclcmcccccc}{@
>{\raggedright\arraybackslash}p{0.25\columnwidth}
>{\raggedright\arraybackslash}X
>{\raggedright\arraybackslash}p{0.25\columnwidth}@}
\toprule
\textbf{Source type} & \textbf{Included when it exposes} & \textbf{Used for} \\
\midrule
Model report 
& post-training recipe, optimizer, verifier, or release budget 
& construction and scaling context \\

Dataset / benchmark 
& task source, split, verifier, contamination audit, or evaluation rule 
& data-object taxonomy \\

Verifier / reward work 
& checker, PRM, judge, rubric, or failure mode 
& quality and audit fields \\

Agent / environment work 
& state, action, observation, terminal predicate, or replay log 
& environmental verification \\

Scaling study 
& compute, reuse, inference budget, or asymptote-efficiency claim 
& scaling attribution \\
\bottomrule
\end{tabularx}

A.2 Supplementary Source Coverage

The main text cites sources at the point where they support an argument. To make the survey boundary explicit, Table 5 lists additional source clusters used as coverage checks for the primer. These sources are not treated as interchangeable evidence: benchmarks identify task and evaluation surfaces, data releases expose trainable objects, verifier papers expose reward channels, and agent environments expose stateful interaction fields. The purpose of this table is therefore bibliographic coverage rather than causal aggregation.

\begin{tabularx}{\ccccccccc}{@
>{\raggedright\arraybackslash}p{0.18\textwidth}
>{\raggedright\arraybackslash}p{0.31\textwidth}
>{\raggedright\arraybackslash}X@}
\toprule
\textbf{Coverage cluster} & \textbf{What it contributes to the primer} & \textbf{Representative sources} \\
\midrule
General reasoning and live evaluations
& contamination-aware or high-difficulty evaluation surfaces for testing whether reported reasoning gains are robust beyond static benchmark reuse
& ([105, 106, 107, 108]) \\

Math and code benchmarks
& canonical answer- or execution-checkable tasks that make programmatic verification reusable for post-training and evaluation
& ([109, 110, 111, 112, 113, 114]) \\

Formal proving environments
& proof-state, theorem-proving, and proof-assistant feedback surfaces where verifier and substrate can collapse into the same object
& ([115, 116, 117, 118, 119, 120]) \\

Instruction, preference, and alignment data
& earlier post-training and preference-learning precedents that clarify how demonstrations, comparisons, reward models, and feedback traces become trainable data objects
& ([121, 122, 123, 124, 125, 126, 127]) \\

Tool use and agent environments
& action schemas, tool calls, web or software environments, and stateful interaction protocols needed to analyze environmental verification
& ([128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139]) \\

Scientific and biomedical reasoning
& evidence-grounded, retrieval-heavy, and expert-sensitive domains where correctness is often evidential rather than a simple answer string
& ([140, 141, 142, 143, 144, 145]) \\

Financial and legal reasoning
& domain-specific reasoning settings where tabular evidence, contracts, holdings, and expert annotation make provenance and rubric fields central
& ([146, 147, 148, 149, 150, 151, 152]) \\

Safety, judges, and reward-model audits
& reward-channel and judge-failure evidence for why verifier cards, attack tests, and refresh logs should be part of reasoning-data releases
& ([153, 154, 155, 156, 157, 158, 159, 160, 161]) \\
\bottomrule
\end{tabularx}

A.3 Supplementary Taxonomy View

Section 2 organizes reasoning post-training data by verification contract. Figure 9 expands that taxonomy by showing why contract labels alone are not sufficient. A sample can be programmatically verified but still answer-only; environmentally verified but scaffold-inherited; or judgment-required but shaped by self-play, rubric design, or teacher lineage. The three cross-cutting axes—supervision granularity, behaviour bounds, and cross-generational lineage—therefore need to be reported orthogonally to the contract itself.

**Figure 9:** **Cross-cutting axes over verifier contracts.** Verification contracts describe what can be checked, whereas granularity, behaviour bounds, and lineage describe how the signal behaves once it enters training. They should therefore be reported orthogonally to programmatic, environmental, and judgment-required contracts.

The diagnostic implication is that a data release should not only name the domain or verifier family. It should specify where supervision enters the trajectory, which behaviours are allowed or bounded, and which generator, teacher, filter, split, or decontamination procedure produced the sample. Without these fields, two superficially similar datasets may instantiate different learning problems.

A.4 Supplementary Construction Schematics

Section 4 compresses the construction discussion into Table 2. The figures below provide schematic views of the same construction problem. They should be read as explanatory diagrams rather than additional claims: each one isolates a different place where gain attribution can become ambiguous.

Figure 10 gives the full construction stack. Its main point is that frontier systems can converge at the visible RL scaffold while diverging upstream in prompt sourcing, trace writing, search substrate, self-play anchors, and verifier design. This is why optimizer comparisons alone cannot determine where the gain came from.

**Figure 10:** **The reasoning-data construction stack.** Reasoning-data recipes accumulate hidden variables before the downstream optimizer is applied. Prompt sourcing determines problem support; trace writing determines inherited reasoning style; search substrates preserve or erase failed branches; anchors determine where curation re-enters; and verifiers determine what counts as success.

Figure 11 isolates trace authorship. The same final answer can be paired with very different process fields depending on whether the trace is written by a human expert, teacher model, self-distillation loop, process-labeling procedure, tool-grounded execution, or self-play trajectory. This matters because traces do not merely supply intermediate text; they can transmit formatting conventions, decomposition style, stopping behaviour, uncertainty expression, and tool-use habits.

**Figure 11:** **Who writes the trace?** Trace authorship changes both the ceiling and the attribution risk. Human experts, teacher models, self-distillation, process labels, tool grounding, and self-play trajectories differ in cost, scalability, provenance, and the degree to which they expose the mechanism rather than only the answer.

Figure 12 expands the self-play discussion. Self-play is often described as reducing human supervision, but it does not remove curation. Instead, it moves curation to the anchor that turns rollouts into trainable signals: an external answer, an interpreter, a verifier, a majority vote, an archive, or a role split between agents. For this reason, self-play releases should report not only the generated tasks, but also the anchor, selection rule, verifier, replay policy, and failure cases.

**Figure 12:** **Self-play relocates curation.** STaR, R-Zero, Absolute Zero, TTRL, and AlphaEvolve-style systems differ less in whether they are "self-generated" than in the anchor that converts rollouts into feedback. The anchor defines which generated behaviours become trainable and which are discarded.

A.5 Supplementary Scaling Views

Section 5 uses the asymptote–efficiency lens to separate what a data substrate makes reachable from how efficiently an optimizer approaches that frontier. Figure 13 expands this separation by listing typical ceiling movers and efficiency movers. The distinction is not absolute: some interventions, especially verifier refresh or environment redesign, may affect both. The figure is therefore a reading device for scaling claims rather than a universal scaling law.

**Figure 13:** **What moves the ceiling, what moves efficiency.** Ceiling movers change what is reachable under a data substrate, verifier, environment, or context; efficiency movers change how quickly the policy approaches that frontier through optimizer design, sampling, curriculum, precision, or stage composition.

Figure 14 treats release time as part of the scaling object. Reasoning-model comparisons often mix model-family changes, data refreshes, verifier updates, benchmark audits, and evaluation-budget changes. A release date is therefore not a neutral timestamp: it can mark a different prompt pool, a new verifier, a changed inference budget, or a new contamination audit.

**Figure 14:** **Release timeline as a scaling object.** A release tick is a versioned data artifact, not a neutral timestamp. Without prompt-pool, verifier-refresh, contamination, lineage, and evaluation-budget fields, score differences can collapse data-substrate changes, optimizer changes, and measurement changes.

A.6 Agent Trajectory Audit Fields

Environmental reasoning data require a richer audit schema than prompt–answer datasets because the trainable object is a trajectory. A successful final transcript is often insufficient: it may hide failed tool calls, retries, state diffs, invalid actions, scaffold interventions, or terminal predicates. Table 6 lists fields that make agent trajectories replayable and auditable.

\begin{tabularx}{\ccccccccc}{@
>{\raggedright\arraybackslash}p{0.16\textwidth}
>{\raggedright\arraybackslash}p{0.30\textwidth}
>{\raggedright\arraybackslash}X@}
\toprule
\textbf{Audit field} & \textbf{What should be released} & \textbf{Why it matters} \\
\midrule
Task state 
& initial state, files, UI state, repository snapshot, environment version 
& determines whether the episode can be reset and replayed \\

Goal and constraints 
& user goal, hidden tests, policy constraints, allowed/disallowed actions 
& separates task success from scaffold compliance \\

Action schema 
& tool/API schema, action grammar, argument format, timeout and permission rules 
& defines the action space available to the model \\

Observations 
& tool outputs, browser/app observations, terminal logs, screenshots, error messages 
& exposes the feedback actually available during the episode \\

State diffs 
& file changes, patches, database changes, UI transitions, environment deltas 
& makes credit assignment and recovery behaviour inspectable \\

Failures and retries 
& invalid actions, failed calls, rejected patches, recovery attempts, backtracking 
& preserves the branches usually erased by cleaned SFT traces \\

Terminal predicate 
& success condition, unit tests, grader, judge, environment predicate 
& defines what counts as completion or reward \\

Scaffold metadata 
& planner, tool wrapper, memory, prompt template, agent loop, stopping rule 
& prevents scaffold changes from being mistaken for model changes \\

Budget and sampling 
& rollout count, token budget, time limit, temperature, pass@ $k$ or selector 
& makes inference-time compute comparable across systems \\

Lineage and split 
& generator, teacher, verifier version, filtering rule, split, contamination audit 
& supports attribution across generations and releases \\
\bottomrule
\end{tabularx}

The table also clarifies why environmental verification is not reducible to final success. If a release includes only the successful path, it removes the branches where tool misuse, recovery, exploration, and credit assignment become visible. For post-training, those branches may be the most informative part of the data.

A.7 Source Placement Guide

The paper uses sources in different roles depending on what each source makes visible. Model reports are useful for documenting public recipes and optimizer scaffolds, but they often under-specify data lineage. Dataset releases are useful for identifying data objects and verifier contracts, but they may not isolate causal training effects. Benchmark and evaluation papers reveal failure modes and measurement surfaces, while scaling studies help separate ceiling movement from approach efficiency. Table 7 summarizes how sources are placed in the primer.

\begin{tabularx}{\ccccccccc}{@
>{\raggedright\arraybackslash}p{0.16\textwidth}
>{\raggedright\arraybackslash}p{0.27\textwidth}
>{\raggedright\arraybackslash}p{0.25\textwidth}
>{\raggedright\arraybackslash}X@}
\toprule
\textbf{Source role} & \textbf{What it contributes} & \textbf{Typical examples} & \textbf{Placement rule} \\
\midrule
Model reports 
& public post-training scaffold, recipe description, inference budget 
& DeepSeek-R1, Kimi K1.5, Qwen3-style reports ([2, 63, 64]) 
& use for visible pipeline claims, not alone for data-causal attribution \\

Reasoning-data releases 
& prompt source, answer format, verifier, filter, split, contamination audit 
& DeepMath-103K, DAPO, OpenThoughts ([4, 10, 3]) 
& use for data-object and construction claims \\

Process-supervision work 
& step labels, PRM training, rollout values, first-error localization 
& PRM800K, Math-Shepherd, OmegaPRM ([22, 23, 46]) 
& use for trace-quality and supervision-granularity claims \\

Agent / environment benchmarks 
& state, action, observation, terminal predicate, replay or task scaffold 
& SWE-Gym, AppWorld, OSWorld-style tasks ([13, 36, 14]) 
& use for environmental verification and trajectory-audit claims \\

Verifier and judge studies 
& verifier failure modes, judge bias, reward hacking, false rejection 
& verifier robustness, master-key, spurious reward studies ([59, 45, 81]) 
& use for correctness and audit-risk claims \\

Scaling studies 
& compute, data reuse, asymptote, approach efficiency, inference-time budget 
& RL scaling and post-training scaling analyses ([19, 20]) 
& use for scaling-attribution claims \\
\bottomrule
\end{tabularx}

A.8 Use of AI Assistance

The authors used AI assistants, including ChatGPT, for language polishing, structural editing, and drafting brief disclosure and limitation text. AI assistants were not used to generate empirical results, create datasets, or determine source inclusion, attribution claims, or citations. All claims, citations, and final manuscript wording were reviewed and verified by the authors.

References

Section Summary: The references section compiles a list of recent academic papers and preprints, mostly from 2023–2026, that investigate ways to improve reasoning abilities in large language models. The works focus on techniques like reinforcement learning, chain-of-thought prompting, high-quality math datasets, data filtering methods, and verification strategies, with contributions from groups such as OpenAI and DeepSeek-AI. Many entries are arXiv preprints examining scalable training, benchmarks, and potential pitfalls in model development.

[1] OpenAI (2024). Learning to reason with LLMs.

[2] DeepSeek-AI et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948. doi:10.1038/s41586-025-09422-z. https://arxiv.org/abs/2501.12948. arXiv:2501.12948.

[3] Etash Guha et al. (2025). OpenThoughts: Data Recipes for Reasoning Models. arXiv preprint arXiv:2506.04178. doi:10.48550/arXiv.2506.04178. https://arxiv.org/abs/2506.04178. arXiv:2506.04178.

[4] Zhiwei He et al. (2025). DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning. arXiv preprint arXiv:2504.11456. doi:10.48550/arXiv.2504.11456. https://arxiv.org/abs/2504.11456. arXiv:2504.11456.

[5] Xu et al. (2025). Structure Trumps Size: Rethinking Data Quality for LLM Reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2025. pp. 11489–11513. doi:10.18653/v1/2025.findings-emnlp.616. http://dx.doi.org/10.18653/v1/2025.findings-emnlp.616.

[6] Jason Wei et al. (2023). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv preprint arXiv:2201.11903. doi:10.48550/arXiv.2201.11903. https://arxiv.org/abs/2201.11903. arXiv:2201.11903.

[7] Miles Turpin et al. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. arXiv preprint arXiv:2305.04388. doi:10.48550/arXiv.2305.04388. https://arxiv.org/abs/2305.04388. arXiv:2305.04388.

[8] Tamera Lanham et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv preprint arXiv:2307.13702. doi:10.48550/arXiv.2307.13702. https://arxiv.org/abs/2307.13702. arXiv:2307.13702.

[9] Alon Albalak et al. (2025). Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models. arXiv preprint arXiv:2502.17387. doi:10.48550/arXiv.2502.17387. https://arxiv.org/abs/2502.17387. arXiv:2502.17387.

[10] Qiying Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv preprint arXiv:2503.14476. doi:10.48550/arXiv.2503.14476. https://arxiv.org/abs/2503.14476. arXiv:2503.14476.

[11] Shumailov et al. (2024). AI models collapse when trained on recursively generated data. Nature. 631(8022). pp. 755–759. doi:10.1038/s41586-024-07566-y. http://dx.doi.org/10.1038/s41586-024-07566-y.

[12] Yujia Qin et al. (2023). ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv preprint arXiv:2307.16789. doi:10.48550/arXiv.2307.16789. https://arxiv.org/abs/2307.16789. arXiv:2307.16789.

[13] Jiayi Pan et al. (2025). Training Software Engineering Agents and Verifiers with SWE-Gym. arXiv preprint arXiv:2412.21139. doi:10.48550/arXiv.2412.21139. https://arxiv.org/abs/2412.21139. arXiv:2412.21139.

[14] Tianbao Xie et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv preprint arXiv:2404.07972. doi:10.48550/arXiv.2404.07972. https://arxiv.org/abs/2404.07972. arXiv:2404.07972.

[15] Eric Zelikman et al. (2022). STaR: Bootstrapping Reasoning With Reasoning. arXiv preprint arXiv:2203.14465. doi:10.48550/arXiv.2203.14465. https://arxiv.org/abs/2203.14465. arXiv:2203.14465.

[16] Andrew Zhao et al. (2025). Absolute Zero: Reinforced Self-play Reasoning with Zero Data. arXiv preprint arXiv:2505.03335. doi:10.48550/arXiv.2505.03335. https://arxiv.org/abs/2505.03335. arXiv:2505.03335.

[17] Yuxin Zuo et al. (2025). TTRL: Test-Time Reinforcement Learning. arXiv preprint arXiv:2504.16084. doi:10.48550/arXiv.2504.16084. https://arxiv.org/abs/2504.16084. arXiv:2504.16084.

[18] Zhihong Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300. doi:10.48550/arXiv.2402.03300. https://arxiv.org/abs/2402.03300. arXiv:2402.03300.

[19] Devvrit Khatri et al. (2025). The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv preprint arXiv:2510.13786. doi:10.48550/arXiv.2510.13786. https://arxiv.org/abs/2510.13786. arXiv:2510.13786.

[20] Zelin Tan et al. (2026). Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning. arXiv preprint arXiv:2509.25300. doi:10.48550/arXiv.2509.25300. https://arxiv.org/abs/2509.25300. arXiv:2509.25300.

[21] Andreas Hochlehnert et al. (2025). A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility. arXiv preprint arXiv:2504.07086. doi:10.48550/arXiv.2504.07086. https://arxiv.org/abs/2504.07086. arXiv:2504.07086.

[22] Hunter Lightman et al. (2023). Let's Verify Step by Step. arXiv preprint arXiv:2305.20050. doi:10.48550/arXiv.2305.20050. https://arxiv.org/abs/2305.20050. arXiv:2305.20050.

[23] Peiyi Wang et al. (2024). Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. arXiv preprint arXiv:2312.08935. doi:10.48550/arXiv.2312.08935. https://arxiv.org/abs/2312.08935. arXiv:2312.08935.

[24] Ivan Moshkov et al. (2025). AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset. arXiv preprint arXiv:2504.16891. doi:10.48550/arXiv.2504.16891. https://arxiv.org/abs/2504.16891. arXiv:2504.16891.

[25] Wasi Uddin Ahmad et al. (2025). OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique. arXiv preprint arXiv:2507.09075. doi:10.48550/arXiv.2507.09075. https://arxiv.org/abs/2507.09075. arXiv:2507.09075.

[26] Zhangchen Xu et al. (2025). KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding. arXiv preprint arXiv:2503.02951. doi:10.48550/arXiv.2503.02951. https://arxiv.org/abs/2503.02951. arXiv:2503.02951.

[27] Z. Z. Ren et al. (2025). DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition. arXiv preprint arXiv:2504.21801. doi:10.48550/arXiv.2504.21801. https://arxiv.org/abs/2504.21801. arXiv:2504.21801.

[28] Rulin Shao et al. (2026). Spurious Rewards: Rethinking Training Signals in RLVR. arXiv preprint arXiv:2506.10947. doi:10.48550/arXiv.2506.10947. https://arxiv.org/abs/2506.10947. arXiv:2506.10947.

[29] Yang Yue et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. arXiv preprint arXiv:2504.13837. doi:10.48550/arXiv.2504.13837. https://arxiv.org/abs/2504.13837. arXiv:2504.13837.

[30] Lukas Helff et al. (2026). LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking. arXiv preprint arXiv:2604.15149. doi:10.48550/arXiv.2604.15149. https://arxiv.org/abs/2604.15149. arXiv:2604.15149.

[31] Niklas Muennighoff et al. (2025). s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393. doi:10.48550/arXiv.2501.19393. https://arxiv.org/abs/2501.19393. arXiv:2501.19393.

[32] Yixin Ye et al. (2025). LIMO: Less is More for Reasoning. arXiv preprint arXiv:2502.03387. doi:10.48550/arXiv.2502.03387. https://arxiv.org/abs/2502.03387. arXiv:2502.03387.

[33] Xiang Deng et al. (2023). Mind2Web: Towards a Generalist Agent for the Web. arXiv preprint arXiv:2306.06070. doi:10.48550/arXiv.2306.06070. https://arxiv.org/abs/2306.06070. arXiv:2306.06070.

[34] Xing Han Lù et al. (2024). WebLINX: Real-World Website Navigation with Multi-Turn Dialogue. arXiv preprint arXiv:2402.05930. doi:10.48550/arXiv.2402.05930. https://arxiv.org/abs/2402.05930. arXiv:2402.05930.

[35] Jing Yu Koh et al. (2024). VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. arXiv preprint arXiv:2401.13649. doi:10.48550/arXiv.2401.13649. https://arxiv.org/abs/2401.13649. arXiv:2401.13649.

[36] Harsh Trivedi et al. (2024). AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. arXiv preprint arXiv:2407.18901. doi:10.48550/arXiv.2407.18901. https://arxiv.org/abs/2407.18901. arXiv:2407.18901.

[37] Shunyu Yao et al. (2024). $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045. doi:10.48550/arXiv.2406.12045. https://arxiv.org/abs/2406.12045. arXiv:2406.12045.

[38] Chowdhury et al. (2024). Introducing SWE-bench Verified. OpenAI blog post. https://openai.com/index/introducing-swe-bench-verified/.

[39] Naman Jain et al. (2025). R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents. arXiv preprint arXiv:2504.07164. doi:10.48550/arXiv.2504.07164. https://arxiv.org/abs/2504.07164. arXiv:2504.07164.

[40] Rahul K. Arora et al. (2025). HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv preprint arXiv:2505.08775. doi:10.48550/arXiv.2505.08775. https://arxiv.org/abs/2505.08775. arXiv:2505.08775.

[41] Forrest Sheng Bao et al. (2024). FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs. arXiv preprint arXiv:2410.13210. doi:10.48550/arXiv.2410.13210. https://arxiv.org/abs/2410.13210. arXiv:2410.13210.

[42] Shaona Ghosh et al. (2025). Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails. arXiv preprint arXiv:2501.09004. doi:10.48550/arXiv.2501.09004. https://arxiv.org/abs/2501.09004. arXiv:2501.09004.

[43] Makesh Narsimhan Sreedhar et al. (2025). Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Models. arXiv preprint arXiv:2505.20087. doi:10.48550/arXiv.2505.20087. https://arxiv.org/abs/2505.20087. arXiv:2505.20087.

[44] David Anugraha et al. (2025). R3: Robust Rubric-Agnostic Reward Models. arXiv preprint arXiv:2505.13388. doi:10.48550/arXiv.2505.13388. https://arxiv.org/abs/2505.13388. arXiv:2505.13388.

[45] Yulai Zhao et al. (2025). One Token to Fool LLM-as-a-Judge. arXiv preprint arXiv:2507.08794. doi:10.48550/arXiv.2507.08794. https://arxiv.org/abs/2507.08794. arXiv:2507.08794.

[46] Liangchen Luo et al. (2024). Improve Mathematical Reasoning in Language Models by Automated Process Supervision. arXiv preprint arXiv:2406.06592. doi:10.48550/arXiv.2406.06592. https://arxiv.org/abs/2406.06592. arXiv:2406.06592.

[47] Xingyao Wang et al. (2025). OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arXiv preprint arXiv:2407.16741. doi:10.48550/arXiv.2407.16741. https://arxiv.org/abs/2407.16741. arXiv:2407.16741.

[48] Jeonghye Kim et al. (2026). Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?. arXiv preprint arXiv:2603.24472. doi:10.48550/arXiv.2603.24472. https://arxiv.org/abs/2603.24472. arXiv:2603.24472.

[49] Cui et al. (2025). OR-Bench: An Over-Refusal Benchmark for Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning. pp. 11515–11542.

[50] Polina Kirichenko et al. (2025). AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. arXiv preprint arXiv:2506.09038. doi:10.48550/arXiv.2506.09038. https://arxiv.org/abs/2506.09038. arXiv:2506.09038.

[51] Fengqing Jiang et al. (2025). SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities. arXiv preprint arXiv:2502.12025. doi:10.48550/arXiv.2502.12025. https://arxiv.org/abs/2502.12025. arXiv:2502.12025.

[52] Anhao Zhao et al. (2026). Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation. arXiv preprint arXiv:2605.16826. doi:10.48550/arXiv.2605.16826. https://arxiv.org/abs/2605.16826. arXiv:2605.16826.

[53] Tommaso Green et al. (2025). Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers. arXiv preprint arXiv:2506.15674. doi:10.48550/arXiv.2506.15674. https://arxiv.org/abs/2506.15674. arXiv:2506.15674.

[54] Yixiong Fang et al. (2025). LastingBench: Defend Benchmarks Against Knowledge Leakage. arXiv preprint arXiv:2506.21614. doi:10.48550/arXiv.2506.21614. https://arxiv.org/abs/2506.21614. arXiv:2506.21614.

[55] Ziwen Han et al. (2025). Search-Time Data Contamination. arXiv preprint arXiv:2508.13180. doi:10.48550/arXiv.2508.13180. https://arxiv.org/abs/2508.13180. arXiv:2508.13180.

[56] Andreas Plesner et al. (2026). An Imperfect Verifier is Good Enough: Learning with Noisy Rewards. arXiv preprint arXiv:2604.07666. doi:10.48550/arXiv.2604.07666. https://arxiv.org/abs/2604.07666. arXiv:2604.07666.

[57] Jujie He et al. (2025). Skywork Open Reasoner 1 Technical Report. arXiv preprint arXiv:2505.22312. doi:10.48550/arXiv.2505.22312. https://arxiv.org/abs/2505.22312. arXiv:2505.22312.

[58] Zhangchen Xu et al. (2025). TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning. arXiv preprint arXiv:2505.14625. doi:10.48550/arXiv.2505.14625. https://arxiv.org/abs/2505.14625. arXiv:2505.14625.

[59] Yuzhen Huang et al. (2025). From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning. arXiv preprint arXiv:2505.22203. doi:10.48550/arXiv.2505.22203. https://arxiv.org/abs/2505.22203. arXiv:2505.22203.

[60] Zheng et al. (2025). ProcessBench: Identifying Process Errors in Mathematical Reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1009–1024. doi:10.18653/v1/2025.acl-long.50. https://aclanthology.org/2025.acl-long.50/. arXiv:2412.06559.

[61] Mingyang Song et al. (2025). PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models. arXiv preprint arXiv:2501.03124. doi:10.48550/arXiv.2501.03124. https://arxiv.org/abs/2501.03124. arXiv:2501.03124.

[62] Alex Cloud et al. (2025). Subliminal Learning: Language models transmit behavioral traits via hidden signals in data. arXiv preprint arXiv:2507.14805. doi:10.48550/arXiv.2507.14805. https://arxiv.org/abs/2507.14805. arXiv:2507.14805.

[63] Kimi Team et al. (2025). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv preprint arXiv:2501.12599. doi:10.48550/arXiv.2501.12599. https://arxiv.org/abs/2501.12599. arXiv:2501.12599.

[64] An Yang et al. (2025). Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. doi:10.48550/arXiv.2505.09388. https://arxiv.org/abs/2505.09388. arXiv:2505.09388.

[65] Mistral-AI et al. (2025). Magistral. arXiv preprint arXiv:2506.10910. doi:10.48550/arXiv.2506.10910. https://arxiv.org/abs/2506.10910. arXiv:2506.10910.

[66] Marah Abdin et al. (2025). Phi-4-reasoning Technical Report. arXiv preprint arXiv:2504.21318. doi:10.48550/arXiv.2504.21318. https://arxiv.org/abs/2504.21318. arXiv:2504.21318.

[67] Akhiad Bercovich et al. (2025). Llama-Nemotron: Efficient Reasoning Models. arXiv preprint arXiv:2505.00949. doi:10.48550/arXiv.2505.00949. https://arxiv.org/abs/2505.00949. arXiv:2505.00949.

[68] MiniMax et al. (2025). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv preprint arXiv:2506.13585. doi:10.48550/arXiv.2506.13585. https://arxiv.org/abs/2506.13585. arXiv:2506.13585.

[69] Qwen Team (2025). Qwen3-Coder: Agentic Coding in the World.

[70] Weizhe Yuan et al. (2025). NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions. arXiv preprint arXiv:2502.13124. doi:10.48550/arXiv.2502.13124. https://arxiv.org/abs/2502.13124. arXiv:2502.13124.

[71] Fang Wu et al. (2026). The Invisible Leash: Why RLVR May or May Not Escape Its Origin. arXiv preprint arXiv:2507.14843. doi:10.48550/arXiv.2507.14843. https://arxiv.org/abs/2507.14843. arXiv:2507.14843.

[72] Yunjie Ji et al. (2025). AM-Thinking-v1: Advancing the Frontier of Reasoning at 32B Scale. arXiv preprint arXiv:2505.08311. doi:10.48550/arXiv.2505.08311. https://arxiv.org/abs/2505.08311. arXiv:2505.08311.

[73] Ganqu Cui et al. (2025). Process Reinforcement through Implicit Rewards. arXiv preprint arXiv:2502.01456. doi:10.48550/arXiv.2502.01456. https://arxiv.org/abs/2502.01456. arXiv:2502.01456.

[74] Chenlu Ye et al. (2026). Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training. arXiv preprint arXiv:2509.03403. doi:10.48550/arXiv.2509.03403. https://arxiv.org/abs/2509.03403. arXiv:2509.03403.

[75] Michael Sullivan and Alexander Koller (2026). GRPO is Secretly a Process Reward Model. arXiv preprint arXiv:2509.21154. doi:10.48550/arXiv.2509.21154. https://arxiv.org/abs/2509.21154. arXiv:2509.21154.

[76] Zhihong Shao et al. (2025). DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning. arXiv preprint arXiv:2511.22570. doi:10.48550/arXiv.2511.22570. https://arxiv.org/abs/2511.22570. arXiv:2511.22570.

[77] Yong Lin et al. (2025). Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction. arXiv preprint arXiv:2508.03613. doi:10.48550/arXiv.2508.03613. https://arxiv.org/abs/2508.03613. arXiv:2508.03613.

[78] Chengsong Huang et al. (2026). R-Zero: Self-Evolving Reasoning LLM from Zero Data. arXiv preprint arXiv:2508.05004. doi:10.48550/arXiv.2508.05004. https://arxiv.org/abs/2508.05004. arXiv:2508.05004.

[79] Yixing Chen et al. (2025). Multi-Agent Evolve: LLM Self-Improve through Co-evolution. arXiv preprint arXiv:2510.23595. doi:10.48550/arXiv.2510.23595. https://arxiv.org/abs/2510.23595. arXiv:2510.23595.

[80] Alexander Novikov et al. (2025). AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131. doi:10.48550/arXiv.2506.13131. https://arxiv.org/abs/2506.13131. arXiv:2506.13131.

[81] Rulin Shao et al. (2026). Spurious Rewards: Rethinking Training Signals in RLVR. arXiv preprint arXiv:2506.10947. doi:10.48550/arXiv.2506.10947. https://arxiv.org/abs/2506.10947. arXiv:2506.10947.

[82] Iman Mirzadeh et al. (2025). GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. arXiv preprint arXiv:2410.05229. doi:10.48550/arXiv.2410.05229. https://arxiv.org/abs/2410.05229. arXiv:2410.05229.

[83] Anisha Gunjal et al. (2025). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv preprint arXiv:2507.17746. doi:10.48550/arXiv.2507.17746. https://arxiv.org/abs/2507.17746. arXiv:2507.17746.

[84] MohammadHossein Rezaei et al. (2025). Online Rubrics Elicitation from Pairwise Comparisons. arXiv preprint arXiv:2510.07284. doi:10.48550/arXiv.2510.07284. https://arxiv.org/abs/2510.07284. arXiv:2510.07284.

[85] Zicheng Kong et al. (2026). Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis. arXiv preprint arXiv:2602.00846. doi:10.48550/arXiv.2602.00846. https://arxiv.org/abs/2602.00846. arXiv:2602.00846.

[86] Delip Rao and Chris Callison-Burch (2026). Autorubric: Unifying Rubric-based LLM Evaluation. arXiv preprint arXiv:2603.00077. doi:10.48550/arXiv.2603.00077. https://arxiv.org/abs/2603.00077. arXiv:2603.00077.

[87] Ran Xu et al. (2026). Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training. arXiv preprint arXiv:2602.01511. doi:10.48550/arXiv.2602.01511. https://arxiv.org/abs/2602.01511. arXiv:2602.01511.

[88] Chengyu Huang et al. (2026). Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text. arXiv preprint arXiv:2604.20051. doi:10.48550/arXiv.2604.20051. https://arxiv.org/abs/2604.20051. arXiv:2604.20051.

[89] Teng Pan et al. (2026). CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution. arXiv preprint arXiv:2603.17775. doi:10.48550/arXiv.2603.17775. https://arxiv.org/abs/2603.17775. arXiv:2603.17775.

[90] Jingcheng Hu et al. (2025). Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model. arXiv preprint arXiv:2503.24290. doi:10.48550/arXiv.2503.24290. https://arxiv.org/abs/2503.24290. arXiv:2503.24290.

[91] Kimi Team et al. (2026). Kimi K2: Open Agentic Intelligence. arXiv preprint arXiv:2507.20534. doi:10.48550/arXiv.2507.20534. https://arxiv.org/abs/2507.20534. arXiv:2507.20534.

[92] Mingjie Liu et al. (2025). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv preprint arXiv:2505.24864. doi:10.48550/arXiv.2505.24864. https://arxiv.org/abs/2505.24864. arXiv:2505.24864.

[93] Yihong Dong et al. (2026). RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization. arXiv preprint arXiv:2508.00222. doi:10.48550/arXiv.2508.00222. https://arxiv.org/abs/2508.00222. arXiv:2508.00222.

[94] Xumeng Wen et al. (2025). Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs. arXiv preprint arXiv:2506.14245. doi:10.48550/arXiv.2506.14245. https://arxiv.org/abs/2506.14245. arXiv:2506.14245.

[95] Zhiyuan Zhai et al. (2026). Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysis. arXiv preprint arXiv:2604.14877. doi:10.48550/arXiv.2604.14877. https://arxiv.org/abs/2604.14877. arXiv:2604.14877.

[96] Kanishk Gandhi et al. (2025). Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs. arXiv preprint arXiv:2503.01307. doi:10.48550/arXiv.2503.01307. https://arxiv.org/abs/2503.01307. arXiv:2503.01307.

[97] Dan Busbridge et al. (2025). Distillation Scaling Laws. arXiv preprint arXiv:2502.08606. doi:10.48550/arXiv.2502.08606. https://arxiv.org/abs/2502.08606. arXiv:2502.08606.

[98] Chengying Tu et al. (2025). A Survey on LLM Mid-Training. arXiv preprint arXiv:2510.23081. doi:10.48550/arXiv.2510.23081. https://arxiv.org/abs/2510.23081. arXiv:2510.23081.

[99] Kaixiang Mo et al. (2025). Mid-Training of Large Language Models: A Survey. arXiv preprint arXiv:2510.06826. doi:10.48550/arXiv.2510.06826. https://arxiv.org/abs/2510.06826. arXiv:2510.06826.

[100] Bradley Brown et al. (2024). Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv preprint arXiv:2407.21787. doi:10.48550/arXiv.2407.21787. https://arxiv.org/abs/2407.21787. arXiv:2407.21787.

[101] Phan et al. (2025). MiGrATe: Mixed-Policy GRPO for Adaptation at Test-Time. OpenReview submission, ICLR 2026.

[102] Milad Aghajohari et al. (2025). The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning. arXiv preprint arXiv:2510.06557. doi:10.48550/arXiv.2510.06557. https://arxiv.org/abs/2510.06557. arXiv:2510.06557.

[103] Zhenru Zhang et al. (2025). The Lessons of Developing Process Reward Models in Mathematical Reasoning. arXiv preprint arXiv:2501.07301. doi:10.48550/arXiv.2501.07301. https://arxiv.org/abs/2501.07301. arXiv:2501.07301.

[104] DeepSeek-AI et al. (2025). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv preprint arXiv:2512.02556. doi:10.48550/arXiv.2512.02556. https://arxiv.org/abs/2512.02556. arXiv:2512.02556.

[105] Long Phan et al. (2026). Humanity's Last Exam. arXiv preprint arXiv:2501.14249. doi:10.1038/s41586-025-09962-4. https://arxiv.org/abs/2501.14249. arXiv:2501.14249.

[106] David Rein et al. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv preprint arXiv:2311.12022. doi:10.48550/arXiv.2311.12022. https://arxiv.org/abs/2311.12022. arXiv:2311.12022.

[107] Elliot Glazer et al. (2025). FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI. arXiv preprint arXiv:2411.04872. doi:10.48550/arXiv.2411.04872. https://arxiv.org/abs/2411.04872. arXiv:2411.04872.

[108] Colin White et al. (2025). LiveBench: A Challenging, Contamination-Limited LLM Benchmark. arXiv preprint arXiv:2406.19314. doi:10.48550/arXiv.2406.19314. https://arxiv.org/abs/2406.19314. arXiv:2406.19314.

[109] Karl Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168. doi:10.48550/arXiv.2110.14168. https://arxiv.org/abs/2110.14168. arXiv:2110.14168.

[110] Dan Hendrycks et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. arXiv preprint arXiv:2103.03874. doi:10.48550/arXiv.2103.03874. https://arxiv.org/abs/2103.03874. arXiv:2103.03874.

[111] Mark Chen et al. (2021). Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374. doi:10.48550/arXiv.2107.03374. https://arxiv.org/abs/2107.03374. arXiv:2107.03374.

[112] Dan Hendrycks et al. (2021). Measuring Coding Challenge Competence With APPS. arXiv preprint arXiv:2105.09938. doi:10.48550/arXiv.2105.09938. https://arxiv.org/abs/2105.09938. arXiv:2105.09938.

[113] Terry Yue Zhuo et al. (2025). BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. arXiv preprint arXiv:2406.15877. doi:10.48550/arXiv.2406.15877. https://arxiv.org/abs/2406.15877. arXiv:2406.15877.

[114] Naman Jain et al. (2024). LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv preprint arXiv:2403.07974. doi:10.48550/arXiv.2403.07974. https://arxiv.org/abs/2403.07974. arXiv:2403.07974.

[115] Kunhao Zheng et al. (2022). miniF2F: a cross-system benchmark for formal Olympiad-level mathematics. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. https://openreview.net/forum?id=9ZPegFuFTFv.

[116] Kaiyu Yang et al. (2023). LeanDojo: Theorem Proving with Retrieval-Augmented Language Models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. http://papers.nips.cc/paper_files/paper/2023/hash/4441469427094f8873d0fecb0c4e1cee-Abstract-Datasets_and_Benchmarks.html.

[117] Zhangir Azerbayev et al. (2023). ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics. arXiv preprint arXiv:2302.12433. doi:10.48550/arXiv.2302.12433. https://arxiv.org/abs/2302.12433. arXiv:2302.12433.

[118] Kshitij Bansal et al. (2019). HOList: An Environment for Machine Learning of Higher Order Logic Theorem Proving. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA. pp. 454–463. http://proceedings.mlr.press/v97/bansal19a.html.

[119] Huajian Xin et al. (2024). DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data. arXiv preprint arXiv:2405.14333. doi:10.48550/arXiv.2405.14333. https://arxiv.org/abs/2405.14333. arXiv:2405.14333.

[120] Huajian Xin et al. (2024). DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search. arXiv preprint arXiv:2408.08152. doi:10.48550/arXiv.2408.08152. https://arxiv.org/abs/2408.08152. arXiv:2408.08152.

[121] Yizhong Wang et al. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv preprint arXiv:2212.10560. doi:10.48550/arXiv.2212.10560. https://arxiv.org/abs/2212.10560. arXiv:2212.10560.

[122] Jason Wei et al. (2022). Finetuned Language Models Are Zero-Shot Learners. arXiv preprint arXiv:2109.01652. doi:10.48550/arXiv.2109.01652. https://arxiv.org/abs/2109.01652. arXiv:2109.01652.

[123] Victor Sanh et al. (2022). Multitask Prompted Training Enables Zero-Shot Task Generalization. arXiv preprint arXiv:2110.08207. doi:10.48550/arXiv.2110.08207. https://arxiv.org/abs/2110.08207. arXiv:2110.08207.

[124] Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. doi:10.48550/arXiv.2203.02155. https://arxiv.org/abs/2203.02155. arXiv:2203.02155.

[125] Ganqu Cui et al. (2024). UltraFeedback: Boosting Language Models with Scaled AI Feedback. arXiv preprint arXiv:2310.01377. doi:10.48550/arXiv.2310.01377. https://arxiv.org/abs/2310.01377. arXiv:2310.01377.

[126] Yuntao Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2204.05862. doi:10.48550/arXiv.2204.05862. https://arxiv.org/abs/2204.05862. arXiv:2204.05862.

[127] Rafael Rafailov et al. (2024). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv preprint arXiv:2305.18290. doi:10.48550/arXiv.2305.18290. https://arxiv.org/abs/2305.18290. arXiv:2305.18290.

[128] Shunyu Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv preprint arXiv:2210.03629. doi:10.48550/arXiv.2210.03629. https://arxiv.org/abs/2210.03629. arXiv:2210.03629.

[129] Timo Schick et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv preprint arXiv:2302.04761. doi:10.48550/arXiv.2302.04761. https://arxiv.org/abs/2302.04761. arXiv:2302.04761.

[130] Minghao Li et al. (2023). API-Bank: A Benchmark for Tool-Augmented LLMs. CoRR. abs/2304.08244. doi:10.48550/ARXIV.2304.08244. https://doi.org/10.48550/arXiv.2304.08244.

[131] Shishir G. Patil et al. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv preprint arXiv:2305.15334. doi:10.48550/arXiv.2305.15334. https://arxiv.org/abs/2305.15334. arXiv:2305.15334.

[132] Zuxin Liu et al. (2024). APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets. arXiv preprint arXiv:2406.18518. doi:10.48550/arXiv.2406.18518. https://arxiv.org/abs/2406.18518. arXiv:2406.18518.

[133] Patil et al. (2025). The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In Forty-second International Conference on Machine Learning. https://gorilla.cs.berkeley.edu/blogs/1$3_b$fcl_v3_multi_turn.html.

[134] Jiarui Lu et al. (2025). ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. arXiv preprint arXiv:2408.04682. doi:10.48550/arXiv.2408.04682. https://arxiv.org/abs/2408.04682. arXiv:2408.04682.

[135] Thibault Le Sellier De Chezelles et al. (2025). The BrowserGym Ecosystem for Web Agent Research. arXiv preprint arXiv:2412.05467. doi:10.48550/arXiv.2412.05467. https://arxiv.org/abs/2412.05467. arXiv:2412.05467.

[136] Shuyan Zhou et al. (2024). WebArena: A Realistic Web Environment for Building Autonomous Agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. https://openreview.net/forum?id=oKn9c6ytLx.

[137] Alexandre Drouin et al. (2024). WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?. arXiv preprint arXiv:2403.07718. doi:10.48550/arXiv.2403.07718. https://arxiv.org/abs/2403.07718. arXiv:2403.07718.

[138] Christopher Rawles et al. (2025). AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. arXiv preprint arXiv:2405.14573. doi:10.48550/arXiv.2405.14573. https://arxiv.org/abs/2405.14573. arXiv:2405.14573.

[139] Frank F. Xu et al. (2025). TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv preprint arXiv:2412.14161. doi:10.48550/arXiv.2412.14161. https://arxiv.org/abs/2412.14161. arXiv:2412.14161.

[140] Dasigi et al. (2021). A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 4599–4610. doi:10.18653/v1/2021.naacl-main.365. https://aclanthology.org/2021.naacl-main.365/.

[141] David Wadden et al. (2020). Fact or Fiction: Verifying Scientific Claims. arXiv preprint arXiv:2004.14974. doi:10.48550/arXiv.2004.14974. https://arxiv.org/abs/2004.14974. arXiv:2004.14974.

[142] Michael D. Skarlinski et al. (2024). Language agents achieve superhuman synthesis of scientific knowledge. arXiv preprint arXiv:2409.13740. doi:10.48550/arXiv.2409.13740. https://arxiv.org/abs/2409.13740. arXiv:2409.13740.

[143] Karan Singhal et al. (2022). Large Language Models Encode Clinical Knowledge. arXiv preprint arXiv:2212.13138. doi:10.48550/arXiv.2212.13138. https://arxiv.org/abs/2212.13138. arXiv:2212.13138.

[144] Qiao Jin et al. (2019). PubMedQA: A Dataset for Biomedical Research Question Answering. arXiv preprint arXiv:1909.06146. doi:10.48550/arXiv.1909.06146. https://arxiv.org/abs/1909.06146. arXiv:1909.06146.

[145] George Tsatsaronis et al. (2015). An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinform.. 16. pp. 138:1–138:28. doi:10.1186/S12859-015-0564-6. https://doi.org/10.1186/s12859-015-0564-6.

[146] Patronus AI (2023). FinanceBench: A benchmark for financial question answering.

[147] Zhiyu Chen et al. (2022). FinQA: A Dataset of Numerical Reasoning over Financial Data. arXiv preprint arXiv:2109.00122. doi:10.48550/arXiv.2109.00122. https://arxiv.org/abs/2109.00122. arXiv:2109.00122.

[148] Fengbin Zhu et al. (2021). TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance. arXiv preprint arXiv:2105.07624. doi:10.48550/arXiv.2105.07624. https://arxiv.org/abs/2105.07624. arXiv:2105.07624.

[149] Neel Guha et al. (2023). LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. arXiv preprint arXiv:2308.11462. doi:10.48550/arXiv.2308.11462. https://arxiv.org/abs/2308.11462. arXiv:2308.11462.

[150] Zheng et al. (2021). When does pretraining help?: assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law. pp. 159–168. doi:10.1145/3462757.3466088. http://dx.doi.org/10.1145/3462757.3466088.

[151] Dan Hendrycks et al. (2021). CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. arXiv preprint arXiv:2103.06268. doi:10.48550/arXiv.2103.06268. https://arxiv.org/abs/2103.06268. arXiv:2103.06268.

[152] Masahiro Koreeda and Christopher Manning (2021). ContractNLI: A dataset for document-level natural language inference for contracts. In EMNLP.

[153] Stephanie Lin et al. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv preprint arXiv:2109.07958. doi:10.48550/arXiv.2109.07958. https://arxiv.org/abs/2109.07958. arXiv:2109.07958.

[154] Mantas Mazeika et al. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. arXiv preprint arXiv:2402.04249. doi:10.48550/arXiv.2402.04249. https://arxiv.org/abs/2402.04249. arXiv:2402.04249.

[155] Seungju Han et al. (2024). WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. arXiv preprint arXiv:2406.18495. doi:10.48550/arXiv.2406.18495. https://arxiv.org/abs/2406.18495. arXiv:2406.18495.

[156] Paul Röttger et al. (2024). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. arXiv preprint arXiv:2308.01263. doi:10.48550/arXiv.2308.01263. https://arxiv.org/abs/2308.01263. arXiv:2308.01263.

[157] Nathan Lambert et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling. arXiv preprint arXiv:2403.13787. doi:10.48550/arXiv.2403.13787. https://arxiv.org/abs/2403.13787. arXiv:2403.13787.

[158] Lianghui Zhu et al. (2025). JudgeLM: Fine-tuned Large Language Models are Scalable Judges. arXiv preprint arXiv:2310.17631. doi:10.48550/arXiv.2310.17631. https://arxiv.org/abs/2310.17631. arXiv:2310.17631.

[159] Seungone Kim et al. (2024). Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. arXiv preprint arXiv:2405.01535. doi:10.48550/arXiv.2405.01535. https://arxiv.org/abs/2405.01535. arXiv:2405.01535.

[160] Leo Gao et al. (2022). Scaling Laws for Reward Model Overoptimization. arXiv preprint arXiv:2210.10760. doi:10.48550/arXiv.2210.10760. https://arxiv.org/abs/2210.10760. arXiv:2210.10760.

[161] Rafael Rafailov et al. (2024). Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. arXiv preprint arXiv:2406.02900. doi:10.48550/arXiv.2406.02900. https://arxiv.org/abs/2406.02900. arXiv:2406.02900.