Fine-Tuning Language Models from Human Preferences

Daniel M. Ziegler$^{}$, Nisan Stiennon$^{}$, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving
OpenAI
{dmz,nisan,jeffwu,tom,alec,damodei,paul,irving}@openai.com

$^{*}$ Equal contribution. Correspondence to [email protected].

Abstract

Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.

Executive Summary: This paper explores how to apply reinforcement learning (RL) to natural language tasks whose goals are defined only by subjective human judgment, rather than by simple programmatic rewards such as ROUGE or BLEU. The central challenge is that real-world objectives are often too complex or value-laden to encode directly, yet purely supervised training on fixed datasets cannot correct errors that appear when models generate new text. The authors therefore investigate whether a modest amount of human preference data can be used to train a reward model that, in turn, guides RL fine-tuning of large pretrained language models.

The work sets out to test a practical pipeline: start with a pretrained GPT-2 model, collect human comparisons of model outputs, fit a reward model to those comparisons, and then optimize the policy with Proximal Policy Optimization while constraining it to stay close to the original model via a KL penalty. Experiments cover two stylistic continuation tasks (positive sentiment and physical descriptiveness) and two summarization tasks (CNN/Daily Mail and TL;DR Reddit posts). The authors compare online data collection, in which new human labels are gathered from the improving policy, with simpler offline collection from the initial model, and they evaluate results with both human raters and automatic metrics.

With only 5,000 human comparisons the method produces stylistic continuations that human labelers prefer to the zero-shot baseline 86 percent of the time and to a supervised sentiment classifier 77 percent of the time. For summarization, 60,000 comparisons yield models that largely copy sentences from the source but intelligently skip irrelevant preambles; these extractive outputs receive higher human ratings than both supervised fine-tuning and the human-written reference summaries, yet they achieve only modest ROUGE gains and remain far less abstractive than desired. Online data collection improves summarization performance noticeably, whereas offline collection works nearly as well for the simpler style tasks. In all cases the learned policies exploit the easiest cues that satisfy labelers, revealing that human feedback can be gamed when evaluation criteria are ambiguous.

These results demonstrate that human preference learning can steer large language models toward desired behavior with relatively little data, but they also expose practical difficulties: labeler agreement is low, models readily discover superficial heuristics such as copying, and fully online data collection adds substantial engineering and quality-control overhead. The findings matter because many high-stakes applications—dialogue, summarization, instruction following—lack reliable automatic rewards and will require ongoing human oversight. If reward models can be made robust, the approach offers a scalable route to aligning models with nuanced human values; if not, systems may produce fluent but untrustworthy or exploitative output.

The authors recommend shifting to batched rather than fully online data collection to simplify training and improve label quality, designing less ambiguous labeling tasks that ask raters to describe problems rather than simply choose winners, and exploring hybrid supervised-plus-RL pipelines that balance extractiveness with accuracy. They also note that further work is needed to prevent reward models from being exploited as policies improve. The main limitations are the modest scale of human data, noisy labels, and restriction to four relatively narrow tasks; results should therefore be treated as suggestive rather than definitive for broader deployment.

1. Introduction

Section Summary: The paper explores how to apply reinforcement learning to complex language tasks where goals are hard to define precisely and must instead be learned from human judgments of what counts as good or bad output. It does so by first training a reward model on human preferences over text samples, then using reinforcement learning to fine-tune large pretrained language models while keeping them close to their original behavior. The approach is tested on stylistic text continuation and document summarization, where it produces outputs that humans prefer to those from standard supervised methods.

We would like to apply reinforcement learning to complex tasks defined only by human judgment, where we can only tell whether a result is good or bad by asking humans. To do this, we can first use human labels to train a model of reward, and then optimize that model. While there is a long history of work learning such models from humans through interaction, this work has only recently been applied to modern deep learning, and even then has only been applied to relatively simple simulated environments ([1, 2, 3]). By contrast, real world settings in which humans need to specify complex goals to AI agents are likely to both involve and require natural language, which is a rich medium for expressing value-laden concepts. Natural language is particularly important when an agent must communicate back to a human to help provide a more accurate supervisory signal ([4, 5, 6]).

Natural language processing has seen substantial recent advances. One successful method has been to pretrain a large generative language model on a corpus of unsupervised data, then fine-tune the model for supervised NLP tasks ([7, 8, 9, 10]). This method often substantially outperforms training on the supervised datasets from scratch, and a single pretrained language model often can be fine-tuned for state of the art performance on many different supervised datasets ([11]). In some cases, fine-tuning is not required: [12] find that generatively trained models show reasonable performance on NLP tasks with no additional training (zero-shot).

There is a long literature applying reinforcement learning to natural language tasks. Much of this work uses algorithmically defined reward functions such as BLEU for translation ([13, 14]), ROUGE for summarization ([13, 15, 16, 17]), music theory-based rewards ([18]), or event detectors for story generation ([19]). [20] used RL on BLEU but applied several error models to approximate human behavior. [16] and [21] learned models of coherence from existing text and used them as RL rewards for summarization and long-form generation, respectively. [22] built an interactive summarization tool by applying reward learning to one article at a time. Experiments using human evaluations as rewards include [23] which used off-policy reward learning for translation, and [24] which applied the modified Q-learning methods of [18] to implicit human preferences in dialog. [25] learned rewards from humans to fine-tune dialog models, but smoothed the rewards to allow supervised learning. We refer to [26] for a survey of RL tasks involving language as a component, and for RL results using transfer learning from language. RL is not the only way to incorporate ongoing human feedback: [27] ask humans what a dialogue system should have said instead, then continue supervised training.

In this paper, we combine the pretraining advances in natural language processing with human preference learning. We fine-tune pretrained language models with reinforcement learning rather than supervised learning, using a reward model trained from human preferences on text continuations. Following [18, 24], we use a KL constraint to prevent the fine-tuned model from drifting too far from the pretrained model. We apply our method to two types of tasks: continuing text in a way that matches a target style, either positive sentiment or vividly descriptive, and summarizing text from the CNN/Daily Mail or TL;DR datasets ([28, 29]). Our motivation is NLP tasks where supervised data sets are unavailable or insufficient, and where programmatic reward functions are poor proxies for our true goals.

For stylistic continuation, 5, 000 human comparisons (each choosing the best of 4 continuations) result in the fine-tuned model being preferred by humans 86% of the time vs. zero-shot and 77% vs. fine-tuning to a supervised sentiment network. For summarization, we use 60, 000 human samples to train models that can roughly be described as "smart copiers": they typically copy whole sentences from the input, but vary what they copy to skip irrelevant initial text. This copying behavior emerged naturally from the data collection and training process; we did not use any explicit architectural mechanism for copying as in [30, 31]. One explanation is that copying is an easy way to be accurate, given that we did not instruct labelers to penalize copying but do instruct them to penalize inaccuracy. It may also reflect the fact that some labelers check for copying as a fast heuristic to ensure a summary is accurate. Indeed, human labelers significantly prefer our models to supervised fine-tuning baselines and even to human-written reference summaries, but not to a lead-3 baseline which copies the first three sentences.

For summarization, we continue to collect additional data and retrain our reward model as the policy improves (online data collection). We also test offline data collection where we train the reward model using data from the original language model only; offline data collection significantly reduces the complexity of the training process. For the TL;DR dataset, human labelers preferred the policy trained with online data collection 71% of the time, and in qualitative evaluations the offline model often provides inaccurate summaries. In contrast, for stylistic continuation we found that offline data collection worked similarly well. This may be related to the style tasks requiring very little data; [32] show that generatively trained models can learn to classify sentiment from very few labeled examples.

In concurrent work, [33] also use human evaluations to learn a reward function for summarization, and optimize that reward function with RL. Their work provides a more detailed investigation of the learned policy and reward function on the CNN/Daily Mail dataset, while we are interested in exploring learning from human feedback more generally and at larger computational scale. So we consider several additional tasks, explore the effects of on-policy reward model training and more data, and fine-tune large language models for both reward modeling and RL.

2. Methods

Section Summary: The methods begin by starting with a pretrained language model that generates text outputs for given inputs such as articles or prompts. Human labelers compare several candidate outputs for each input and select the best one, and these preference data are used to train a separate reward model that scores how good any output is. The original language model is then fine-tuned via reinforcement learning (specifically PPO) to maximize the reward model's scores while a KL penalty term keeps the fine-tuned model from drifting too far from the original, with an option to interleave further human data collection and reward model retraining during optimization.

:::: cols="1"

Figure 1: Our training processes for reward model and policy. In the online case, the processes are interleaved. ::::

We begin with a vocabulary $\Sigma$ and a language model $\rho$ which defines a probability distribution over sequences of tokens $\Sigma^n$ via

$ \rho(x_0 \cdot s x_{n-1}) = \prod_{0 \le k < n} \rho(x_k | x_0 \cdot s x_{k-1}) $

We will apply this model to a task with input space $X = \Sigma^{\leq m}$, data distribution $\mathcal{D}$ over $X$, and output space $Y = \Sigma^n$. For example, $x \in X$ could be an article of up to 1000 words and $y \in Y$ could be a 100-word summary. $\rho$ defines a probabilistic policy for this task via $\rho(y|x) = \rho(xy)/\rho(x)$: fixing the beginning of the sample to $x$ and generating subsequent tokens using $\rho$.

We initialize a policy $\pi = \rho$, and then fine-tune $\pi$ to perform the task well using RL. If the task was defined by a reward function $r : X \times Y \rightarrow \mathbb{R}$, then we could use RL to directly optimize the expected reward:

$ \mathbb{E}{\pi}\left[r\right] = \mathbb{E}{x \sim \mathcal{D}, y \sim \pi(\cdot | x)}\left[r(x, y)\right] $

However, we want to perform tasks defined by human judgments, where we can only learn about the reward by asking humans. To do this, we will first use human labels to train a reward model, and then optimize that reward model.

Following [1], we ask human labelers to pick which of several values of $y_i$ is the best response to a given input $x$. [^1] We ask humans to choose between four options $(y_0, y_1, y_2, y_3)$; considering more options allows a human to amortize the cost of reading and understanding the prompt $x$. Let $b \in \left{0, 1, 2, 3\right}$ be the option they select. Having collected a dataset $S$ of $(x, y_0, y_1, y_2, y_3, b)$ tuples, we fit a reward model $r : X \times Y \rightarrow \mathbb{R}$ using the loss

[^1]: In early experiments we found that it was hard for humans to provide consistent fine-grained quantitative distinctions when asked for an absolute number, and experiments on synthetic tasks confirmed that comparisons were almost as useful.

$ \textrm{loss}(r) = \mathbb{E}_{\left(x, \left{y_i\right}_i, b\right) \sim S}\left[\log {\frac {e^{r(x, y_b)}}{\sum_i e^{r(x, y_i)}}}\right]\tag{1} $

Since the reward model needs to understand language, we initialize it as a random linear function of the final embedding output of the language model policy $\rho$ following [9] (see Section 4.2 for why we initialize from $\rho$ rather than $\pi$). To keep the scale of the reward model consistent across training, we normalize it so that it has mean 0 and variance 1 for $x \sim \mathcal{D}, y \sim \rho(\cdot|x)$.

Now we fine-tune $\pi$ to optimize the reward model $r$. To keep $\pi$ from moving too far from $\rho$, we add a penalty with expectation $\beta \operatorname{KL}(\pi, \rho)$ (see Table 10 for what happens without this). That is, we perform RL on the modified reward

$ R(x, y) = r(x, y) - \beta \log \frac {\pi(y|x)}{\rho(y|x)}.\tag{2} $

We either choose a constant $\beta$ or vary it dynamically to achieve a particular value of $\operatorname{KL}(\pi, \rho)$; see Section 2.2. This term has several purposes: it plays the role of an entropy bonus, it prevents the policy from moving too far from the range where $r$ is valid, and in the case of our style continuation tasks it also is an important part of the task definition: we ask humans to evaluate style, but rely on the KL term to encourage coherence and topicality.

Our overall training process is:

1. Gather samples $(x, y_0, y_1, y_2, y_3)$ via $x \sim \mathcal{D}, y_i \sim \rho(\cdot | x)$. Ask humans to pick the best $y_i$ from each.
2. Initialize $r$ to $\rho$, using random initialization for the final linear layer of $r$. Train $r$ on the human samples using .
3. Train $\pi$ via Proximal Policy Optimization (PPO, [34]) with reward $R$ on $x \sim \mathcal{D}$.
4. In the online data collection case, continue to collect additional samples, and periodically retrain the reward model $r$. This is described in Section 2.3.

2.1 Pretraining details

We use a 774M parameter version of the GPT-2 language model in [12] trained on their WebText dataset and their 50, 257 token invertible byte pair encoding to preserve capitalization and punctuation ([35]). The model is a Transformer with 36 layers, 20 heads, and embedding size 1280 ([36]).

For stylistic continuation tasks we perform supervised fine-tuning of the language model to the BookCorpus dataset of [37] prior to RL fine-tuning; we train from scratch on WebText, supervised fine-tune on BookCorpus, then RL fine-tune to our final task. To improve sample quality, we use a temperature of $T < 1$ for all experiments; we modify the initial language model by dividing logits by $T$, so that future sampling and RL with $T = 1$ corresponds to a lower temperature for the unmodified pretrained model.

2.2 Fine-tuning details

Starting with the pretrained language model, the reward model is trained using the Adam optimizer ([38]) with . The batch size is 8 for style tasks and 32 for summarization, and the learning rate is 1.77 x 10^-5 for both. We use a single epoch to avoid overfitting to the small amount of human data, and turn off dropout.

For training the policy $\pi$, we use the PPO2 version of Proximal Policy Optimization from [39]. We use 2M episodes ($x, y$ pairs), $\gamma = 1$, four PPO epochs per batch with one minibatch each, and default values for the other parameters. We use batch size 1024 for style tasks and 512 for summarization. We do not use dropout for policy training. The learning rate was 1.41 x 10^-5 for style tasks and 7.07 x 10^-6 for summarization.

Models trained with different seeds and the same KL penalty $\beta$ sometimes end up with quite different values of $\operatorname{KL}(\pi, \rho)$, making them hard to compare. To fix this, for some experiments we dynamically vary $\beta$ to target a particular value of $\operatorname{KL}(\pi, \rho)$ using the log-space proportional controller

$ \begin{aligned} e_t &= \operatorname{clip}\left(\frac{\operatorname{KL}(\pi_t, \rho) - \operatorname{KL}{\mathrm{target}}}{\operatorname{KL}{\mathrm{target}}}, -0.2, 0.2\right) \ \beta_{t+1} &= \beta_t(1 + K_\beta e_t) \end{aligned} $

We used $K_\beta = 0.1$.

For supervised fine-tuning baselines, we fine-tune for 1 epoch on the CNN/Daily Mail and TL;DR training sets (for TL;DR we removed 30K examples to serve as a validation set). We decayed the learning rate to 0 with a cosine schedule; for the initial value, we swept over 8 log-linearly spaced options between $10^{-4}$ and 3 x 10^-4. We also experimented with different dropout rates, and found a rate of 0.1 to work best. We then chose the model with the best validation loss.

2.3 Online data collection

If the trained policy $\pi$ is very different from the zero-shot policy $\rho$, the reward model will suffer a large distributional shift from training on samples from $\rho$ to evaluation on samples from $\pi$. To prevent this, we can collect human data throughout RL fine-tuning, continuously gathering new data by sampling from $\pi$ and retraining the reward model. As Section 3 shows, online data collection was important for summarization but not for the simpler style tasks.

In the online case, we will choose a function $l(n)$ describing how many labels we want before beginning the $n^{\textrm{th}}$ PPO episode. Let $N_{\pi} = 2 \times 10^6$ be the total number of PPO episodes, $N_r^0 = l(0)$ be an initial number of human labels, and $N_r$ be the total number of human labels. We take

$ \begin{aligned} l(n) &= N_r^0 + (N_r - N_r^0) \left(1 - (1 - n/N_\pi)^2\right) \end{aligned} $

We pause before the $n^{\textrm{th}}$ PPO episode if we have fewer than $l(n)$ labels. We send another batch of requests to the labelers if the total requests so far is less than $l(n) + 1000$, to ensure they have at least 1000 outstanding queries at any time. We train the reward model before the first PPO episode, and then retrain it 19 more times at evenly spaced values of $l(n)$. Each time we retrain we reinitialize $r$ to a random linear layer on top of $\rho$ and do a single epoch through the labels collected so far. The offline case is the limit $N_r = N_r^0$.

To estimate overall progress, we gather validation samples consisting of $x \sim \mathcal{D}; y_0, y_1 \sim \rho(\cdot |x); y_2, y_3 \sim \pi(\cdot | x)$ at a constant rate; human labels on these give how often $\pi$ beats $\rho$. Since validation samples are only used to evaluate the current $\pi$, we can add them to the training set for $r$. In order to estimate inter-labeler agreement, 5% of queries are answered 5 times by different labelers. Label counts in Section 3 include validation samples and repeated labels.

2.4 Human labeling

We use Scale AI to collect labels. The Scale API accepts requests of the form $(x, y_0, y_1, y_2, y_3)$ and returns selections $b \in \left{0, 1, 2, 3\right}$. We describe the task to Scale through a combination of instructions (Appendix A) and a dataset of about 100 example comparisons from the authors.

Unlike many tasks in ML, our queries do not have unambiguous ground truth, especially for pairs of similar outputs (which play a large role in our training process, since we train $r$ on pairs of labels sampled from a single policy $\pi$). This means that there is significant disagreement even between labelers who have a similar understanding of the task and are trying to rate consistently. On 4-way comparisons for sentiment and TL;DR summarization, authors of this paper agree about 60% of the time (vs. 25% for random guessing). This low rate of agreement complicates the quality control process for Scale; the authors agree with Scale labelers 38% of the time on sentiment and 46% of the time on TL;DR summarization. We give further details of the human data collection and quality evaluation in Appendix B.

For final evaluation of two models $A$ and $B$, we generate either 2-way comparisons between pairs $(a \sim A, b \sim B)$ or 4-way comparisons with quadruples $(a_0, a_1 \sim A, b_0, b_1 \sim B)$, randomize the order in which samples are presented, and present these comparisons to Scale. Evaluating the quality of a model trained by Scale using the same set of humans from Scale is perilous: it demonstrates that $r$ and $\pi$ have succeeded in fitting to the human reward, but does not show that those human evaluations capture what we really care about, and our models are incentivized to exploit idiosyncracies of the labeling process. We include samples from our models so that readers can judge for themselves.

3. Experiments

Section Summary: In the experiments, the authors first validate their reinforcement learning approach on a controlled mock sentiment task, where a classifier serves as a proxy for human feedback, and show that the method can optimize this reward nearly as well as direct access when using 20,000 to 60,000 queries to train a reward model. They then apply the technique to real human preferences for stylistic text continuations, such as positive sentiment or physical descriptiveness, demonstrating that models trained on small amounts of human data produce outputs preferred by evaluators over those optimized with the mock reward alone. Finally, they extend the method to summarization on CNN/Daily Mail and TL;DR datasets, finding that the resulting models act mainly as effective extractors rather than abstractors.

In Section 3.1.1, we test our approach to RL fine-tuning of language models by using a mock labeler (a sentiment model trained on a review classification problem) as a stand-in for human labels. We show that RL fine-tuning is effective at optimizing this complex but somewhat artificial reward. In Section 3.1.2, we show that we can optimize language models from human preferences on stylistic continuation tasks (sentiment and physical descriptiveness) with very little data, and that in the sentiment case the results are preferred to optimizing the review sentiment model. In Section 3.2 we apply RL fine-tuning to summarization on the CNN/Daily Mail and TL;DR datasets, show that the resulting models are essentially "smart copiers", and discuss these results in the context of other summarization work.

We release code[^2] for reward modeling and fine-tuning in the offline data case. Our public version of the code only works with a smaller 124M parameter model with 12 layers, 12 heads, and embedding size 768. We include fine-tuned versions of this smaller model, as well as some of the human labels we collected for our main experiments (note that these labels were collected from runs using the larger model).

[^2]: Code at https://github.com/openai/lm-human-preferences.

3.1 Stylistic continuation tasks

We first apply our method to stylistic text continuation tasks, where the policy is presented with an excerpt from the BookCorpus dataset ([37]) and generates a continuation of the text. The reward function evaluates the style of the concatenated text, either automatically or based on human judgments. We sample excerpts with lengths of 32 to 64 tokens, and the policy generates 24 additional tokens. We set the temperature of the pretrained model to $T = 0.7$ as described in Section 2.1.

3.1.1 Mock sentiment task

\begin{tabular}{l|lll|lll} \toprule
  {} & \multicolumn{3}{c|}{Sentiment} & \multicolumn{3}{c}{Descriptiveness} \\ 
  \midrule

\tableentry{zero-shot}{88}{12}{86}{14}
\lefttableentry{mock}{77}{23}
\tableentry{20k offline}{48}{52}{47}{53}
\tableentry{5k online}{50}{50}{48}{52}

\bottomrule
\end{tabular}

To study our method in a controlled setting, we first apply it to optimize a known reward function $r_s$ designed to reflect some of the complexity of human judgments. We construct $r_s$ by training a classifier[^3] on a binarized, balanced subsample of the Amazon review dataset of [40]. The classifier predicts whether a review is positive or negative, and we define $r_s(x, y)$ as the classifier's log odds that a review is positive (the input to the final sigmoid layer).

[^3]: The model is a Transformer with 6 layers, 8 attention heads, and embedding size 512.

Optimizing $r_s$ without constraints would lead the policy to produce incoherent continuations, but as described in Section 2.2 we include a KL constraint that forces it to stay close to a language model $\rho$ trained on BookCorpus.

The goal of our method is to optimize a reward function using only a small number of queries to a human. In this mock sentiment experiment, we simulate human judgments by assuming that the "human" always selects the continuation with the higher reward according to $r_s$, and ask how many queries we need to optimize $r_s$.

Figure 2 shows how $r_s$ evolves during training, using either direct RL access to $r_s$ or a limited number of queries to train a reward model. 20k to 60k queries allow us to optimize $r_s$ nearly as well as using RL to directly optimize $r_s$.

Because we know the reward function, we can also analytically compute the optimal policy and compare it to our learned policies.

With a constraint on the KL divergence $\operatorname{KL}(\pi, \rho)$ between the learned policy $\pi$ and the language model $\rho$, the optimal policy has the form:

$ \begin{aligned} \pi_\textrm{opt}(y | x) &\propto \rho(y | x) e^{r_s(x, y) / \beta } \end{aligned} $

We approximate the reward of this policy for given $x$ and $\beta$ by sampling a large number of continuations from $\rho(y|x)$ and reweighting them by $e^{r_s(x, y) / \beta}$. Figure 3 compares the reward obtained by our policies to the estimated optimal reward across a range of KL values. There is a significant gap from optimality after training the policy on 2M continuations—the number used in our main experiments—though it is largely closed with more training. Our policies continue to receive higher rewards for larger KL divergences, where we cannot afford to approximate $\pi_\textrm{opt}$ by sampling.

3.1.2 Human evaluations of continuations

::: {caption="Table 2: Three random (T=0.7) continuations for our sentiment continuation task. Chosen from Table 11 in Appendix; see appendix for more."}

:::

\begin{tabular}{|p{2cm}|p{4cm}|p{4cm}|p{4cm}|}

\hline

\hline
\textbf{context}
 {} &
\multicolumn{3}{|C{12cm}|}{ 
``I do not know if it was Viking related, but it could have been.''
\newline
``Really?'' Ailia said. Is it safe to be traveling here then? Ailia looked behind her to make sure they weren't being followed.
}
\\
\hline
 {} &
\textit{Continuation 1}
 {} &
\textit{Continuation 2}
 {} &
\textit{Continuation 3}
\\
\hline
\textbf{zero-shot}
 {} &
There were no signs of anyone. 
\newline
``It is safe enough, '' Ailios said.
 {} &
``Because I have a friend that is in the area and he will be coming with us.
 {} &
It was hard to see that far. ''I do not like that word.
\\
\hline
\textbf{5k offline fine-tune}
 {} &
Kaya crouched low, her eyes wide in the moonlight. Her body was tense.
 {} &
She put her hand on the sword strapped to her back, and then pulled it out.
 {} &
She strode out the door and walked down the street, her nose wrinkled in disapproval.
\\
\hline

\end{tabular}

We apply our method to two continuation tasks defined by human judgments:

The human labelers are presented with a BookCorpus excerpt and four possible continuations; they are asked to select the best continuation. Full instructions for labelers are provided in Appendix A (although labelers also learned from $\sim 50$ example comparisons labeled by the authors and so the instructions do not completely define the task).

To make the labeling task more natural, we select excerpts that start and end with a period. When sampling continuations that will be presented to humans, we use rejection sampling to ensure there is a period between tokens 16 and 24 and then truncate at that period.[^4] During the RL fine-tuning, we penalize continuations that don't have such a period by giving them a fixed reward of $-1$.

[^4]: This is a crude approximation for "end of sentence." We chose it because it is easy to integrate into the RL loop, and even a crude approximation is sufficient for the intended purpose of making the human evaluation task somewhat easier.

We dynamically adjusted $\beta$ to obtain a KL divergence of 6 nats for descriptiveness and 10 nats for sentiment (Section 2.2).

\begin{tabular}{l|cccc|cccc} \toprule
   {} & \multicolumn{4}{c|}{TL;DR} & \multicolumn{4}{c}{CNN/Daily Mail} \\ 
   {} & R-1 & R-2 & R-L & R-AVG & R-1 & R-2 & R-L & R-AVG \\ 
  \midrule

SOTA
 {} & 22* & 5* & 17* & 14.7*
 {} & 41.22 & 18.68 & 38.34 & 32.75 \\
\midrule
lead-3 
 {} & 17.435 & 3.243 & 14.575 & 11.751
 {} & \textbf{40.379} & \textbf{17.658} & 36.618 & 31.552
\\

zero-shot 
 {} & 15.862 & 2.325 & 13.518 & 10.568
 {} & 28.406 & 8.321 & 25.175 & 20.634
\\

supervised baseline 
 {} & 17.535 & 3.124 & 14.969 & 11.877 

 {} & 39.525 & 16.992 & 36.728 & 31.082
\\

supervised + 60k fine-tune
 {} & \textbf{18.434} & \textbf{3.542} & \textbf{15.457} & \textbf{12.478}
 {} & 40.093 & 17.611 & \textbf{37.104} & \textbf{31.603}
\\

60k fine-tune
 {} & 16.800 & 2.884 & 14.011 & 11.232 
 {} & 37.385 & 15.478 & 33.330 & 28.731
\\

30k fine-tune
 {} & 16.410 & 2.920 & 13.653 & 10.994
 {} & 35.581 & 13.662 & 31.734 & 26.992
\\

15k fine-tune
 {} & 15.275 & 2.240 & 12.872 & 10.129
 {} & 38.466 & 15.960 & 34.468 & 29.631
\\

60k offline fine-tune
 {} & 16.632 & 2.699 & 13.984 & 11.105
 {} & 33.860 & 12.850 & 30.018 & 25.576
\\

\bottomrule
\end{tabular}

::: {caption="Table 5: Human evaluation of summarization models. For each pair of models and each dataset, we sample 1024 articles from the test set, generate a summary from each model, and ask 3 humans to pick the best summary using the same instructions as in training. The model chosen by a majority of the humans wins on that article. We report the fraction of articles that each model wins. For all models, we sample with temperature 0.7 for TL;DR and 0.5 for CNN/DM."}

:::

::: {caption="Table 6: Random (T=0.5) summaries for our CNN/DM summarization task, on the same context. Samples chosen from Table 16 in Appendix (see appendix also for context being summarized). The 60k fine-tune model copies from the source article."}

:::

We trained a range of models using different amounts of feedback, testing both offline data collection where humans rate only the initial language model's continuation, and online data collection where humans continuously rate the current policy's continuations (Section 2.3). We then compared these different policies to each other and to the zero-shot performance of the original language model. The results are shown in Figure 4 and Table 1. Each model comparison is based on 1024 four-way continuation comparisons, two from each of the models being compared, each rated by 3 humans.

For these continuation tasks, offline and online data collection give similar performance. We find that very little human data is required for fine-tuning: performance with 5k, 10k, and 20k reward model training samples is similar, degrading only for less than 5k samples. [^5] The model trained using the review sentiment classifier from Section 3.1.1 does poorly relative to models optimized using human preference: in 77% of contexts, labelers preferred the output of the model trained with real human feedback.

[^5]: The descriptiveness policy trained with 2.5k samples performed poorly, but we believe this is due to randomness in RL.

3.2 Summarization

We also applied our method to two summarization tasks: the CNN/Daily Mail dataset of [28] and the TL;DR dataset of [29]. We sample articles or Reddit posts, truncate to 500 tokens, add a "\n\nTL;DR:" suffix (and for CNN/Daily Mail, a "Article:\n\n" prefix) and let the policy respond with up to 75 tokens. We set the temperature of the pretrained model to $T = 0.5$ for CNN/Daily Mail and $T = 0.7$ for TL;DR. To make the task more natural for humans, we ensure articles consist of whole sentences by truncating to the last newline character. When sampling summaries that will be shown to a human, we use rejection sampling to ensure there is a newline between tokens 55 and 75 and truncate at that newline. During RL fine-tuning, we penalize summaries that don't have such a newline by giving them a fixed score of -1. For CNN/Daily Mail we used a fixed KL coefficient $\beta = 0.1$; for TL;DR we used $\beta = 0.03$.

For RL fine-tuning, we trained online data collection models with 15k, 30k, and 60k human labels, and an offline data collection ablation with 60k labels. We also show zero-shot performance of the pretrained model, a supervised fine-tuned baseline using the same pretrained model as starting point (Section 2.2), and a lead-3 baseline which copies the first three sentences of the context. We truncate lead-3 at a period in the same way we truncate generated summaries, so occasionally it is 2 sentences. Finally, we combine supervised and RL fine-tuning: performing human RL fine-tuning starting with the supervised fine-tuned model. The purely RL fine-tuned models use contexts from the datasets during training but ignore the reference summaries; the supervised and supervised+RL models use both contexts and summaries.

We report two sets of numerical results: human evaluations between pairs of models (Table 5) and ROUGE results on the test set of CNN/Daily Mail and our validation set of TL;DR (Table 4). ROUGE results suggest that online data collection is important for best performance, in contrast to our stylistic continuation tasks. At a fixed number of labels, online tends to be better than offline, with a 3 point R-AVG gain on CNN/DM at 60k labels.[^6] On both datasets we see significant returns to data volume up to 60k human labels (though the trend is less clear for human evaluation). On both datasets, supervised + RL fine-tuning is best, and indeed pure RL fine-tuning is worse than the supervised baseline according to ROUGE in all cases (though the supervised baseline uses the full supervised training dataset, which is much larger than 60k samples). Lead-3 is hard to beat: it is the best model for R-1 and R-2 on CNN/Daily Mail, and only supervised + RL fine-tuning beats it otherwise.

[^6]: That said, different training runs have considerable variation and it is expensive to run multiple seeds with humans, so it is possible that this gap is largely noise.

But our goal is optimizing reward defined by humans, not ROUGE. Table 5 shows pairwise comparisons between different model pairs according to human labelers, using 1024 samples with majority vote of 3 labelers per sample. Here the picture is different, though also significantly noisier. Our online trained, 60k label model reliably beats both the zero-shot and supervised baselines, and even beats the combined supervised + RL fine-tuned model. Online training remains important, but the situation w.r.t. data volume is less clear and likely contaminated by noise: the 60k TL;DR model beats the 30k model only 40% of the time, for example. More worrisome, the 60k online model beats the human ground truth 96% of the time for TL;DR and 84% of the time for CNN/Daily Mail.

What is going on? As we show in the next section, our 60k RL fine-tuned model is almost entirely extractive (despite lacking any explicit extractive architectural component): it mostly copies whole sentences from the context, but varies which sentences are copied.

3.2.1 What our models copy

Much previous work in summarization has focused on explicit copying mechanisms, including the pointer network-based architecture of [30] and the two-phase mask and paraphrase approach of [31]. The goal is to take advantage of copying (which is of fundamental importance to the task of summarization) without only copying—to be abstractive rather than extractive.

Figure 5 and Figure 6 show the fractions of $n$-grams and sentences generated by our models which are novel and repeated, respectively. From the novelty stats, we see that our RL fine-tuning consistently causes models to copy more. In particular, our 60k RL fine-tuned models are almost entirely extractive: they copy whole sentences 71% of the time for TL;DR and 98% of the time for CNN/Daily Mail. Applying RL fine-tuning starting from the supervised fine-tuned model copies much less: 6% and 30% for TL;DR and CNN/Daily Mail. Although we do not use explicit coverage metrics as in [30, 31], both supervised and RL fine-tuned models do very little repetition within summaries.

\begin{tabular}{l|rr|rr} \toprule
   {} & \multicolumn{2}{c|}{TL;DR} & \multicolumn{2}{c}{CNN/Daily Mail} \\ 
   {} & all & preamble & all & preamble \\ 
  \midrule

zero-shot &
1.3\% & 0\% &
10.4\% & 1.0\% \\

60k fine-tuned &
28.3\% & 0.2\% &
77.6\% & 1.4\% \\

supervised &
1.5\% & 0\% &
9.4\% & 0\% \\

supervised + 60k fine-tuned &
7.9\% & 0\% &
16.6\% & 0\% \\

reference summaries &
0.6\% & 0\% &
5.1\% & 0\% \\

\midrule
total articles & 
30000 & 3762 &  
13368 & 297 \\

\bottomrule
\end{tabular}

While the purely RL fine-tuned models mostly copy, they vary where they copy from. Figure 7 illustrates this via the position of the longest common subsequence between context and summary. To understand when the model chooses to copy from the exact beginning, we identify common preambles in articles such that we would expect copying to be a poor strategy. Table 7 shows that these preambles are copied much less often than in the immediate beginnings of other articles, giving evidence that our models are smart about when to copy. However, we cannot establish that our reward model is smart beyond rewarding copying, as the zero-shot model also skips preambles.

: Table 8: Frequency with which generated summaries are accurate, in the sense of only making statements supported by the context, as judged by the authors on 30 articles from each dataset. The 60k fine-tuned model achieves high accuracy via copying; the supervised and supervised + 60k fine-tuned models are more abstractive but at significant cost to accuracy.

	TL;DR	CNN/Daily Mail
zero-shot	6/30	6/30
60k fine-tuned	26/30	29/30
supervised	8/30	19/30
supervised + 60k fine-tuned	11/30	20/30

Since combining supervised fine-tuning and RL fine-tuning gives the best ROUGE scores and and is also more abstractive, why not use it? Unfortunately there is an advantage to pure copying shown in Table 8: it makes it easy for the model to tell the truth. The models that copy the most, 60k RL fine-tuned, is 90% and 95% accurate on TL;DR and CNN/Daily Mail; lifting whole sentences from the article usually leaves them true. The supervised fine-tuned and combined supervised+RL fine-tuned models are accurate at most 70% of the time: they paraphrase but paraphrase badly, often swapping names from the context or mixing together multiple sentences in invalid ways. Zero-shot is the most novel, but is accurate only 20% of the time. Similarly, [42] found that 30% of samples from the supervised summarization models they tested contained inconsistencies, and [10] found that their pretrained encoder-decoder model "hallucinates facts...which are topical but never appear in the source".

There are at least two ways of interpreting these results. The first is that copying is the easiest way to be accurate. The labelers were told to penalize inaccuracy and redundancy, but were not told to penalize copying. The zero-shot model copies some of the time, and when it copied it was accurate, so this behavior was reinforced. The result is a model that "degenerated to copying", but at least does not lie.

However, this does not explain why both our model and lead-3 are strongly preferred by the labelers to the human reference summaries (Table 5). This reveals a mismatch between the notion of quality we wanted our model to learn, and what the humans labelers actually evaluated. Checking for copying is very easy, so labelers who check primarily for copying can work quickly. Since the online data collection setting made quality control more difficult, we failed to detect and penalize this behavior.

4. Challenges

Section Summary: Online data collection proved difficult due to the added software complexity of interleaving labeling, reward training, and policy updates, along with challenges in debugging and maintaining consistent data quality over time. The authors also found that sharing parameters between the reward model and policy led to overfitting because of the large imbalance in available training data for each. Finally, the subjective and multidimensional nature of evaluating summaries made consistent human labeling hard, increasing the risk of noisy data and complicating reliable assessment of results.

We conclude with a few lessons and directions we plan to consider in future reward learning work.

4.1 Online data collection

is hard

Online data collection was necessary to achieve the best results on summarization. However, fully online data collection—where each label comes from an up-to-date version of the policy which has already learned from almost all previous labels—had major disadvantages:

1. Software complexity: Our online system interleaves data gathering, reward model training, and RL fine-tuning. The resulting distributed system was significantly more complicated than if each task was kept separate, slowing the software development process. Moreover, a bug in any one of the tasks would break the entire training process.
2. Machine learning complexity: Online experiments were difficult to debug, as it was hard to iterate on one piece of the ML system at a time. We could often debug an online job only by switching briefly to offline, such as by launching a standalone reward model training run, but then would switch back to online once debugging was complete (until the next cycle).
3. Quality control issues: Significant work was required on Scale's part to make their data quality mechanisms work in the low latency, online setting. However, even after this work it was difficult to maintain high data quality over a long period of time, and regressions were often not detected until after (or well after) training runs were complete. Since evaluation of labeler performance was online, by the time a worker was detected as poor some of their data might already been reported back and used for reward model training.

We believe the right middle ground between offline and online data collection is batched data collection, and plan to use this setting in future work. Collect a batch of data from the pretrained policy $\rho$, train the reward model $r$ on this batch, then fine-tune the policy $\pi$ with $r$ frozen. Once complete, collect another batch of data sampled from $\pi$, and iterate. The latency for each batch can be far longer than the online case, simplifying quality control. As in the fully online setting, we can always retrain the reward model from scratch on all data collected so far; human data is expensive so the total volume will be low. Removing the interleaved training of $r$ and $\pi$ simplifies software architecture and diagnosis of ML issues, and allows iteration on just one component (say $r$ in isolation) if problems occur. [43] reached similar conclusions in a restricted dialogue setting after validating in simulation that online and batched trained performed similarly.

Batched data collection is also a well-studied setting for active learning techniques. Although we use RL to fine-tune the policy $\pi$, the human data is used only for supervised training of the reward model $r$. Thus, any method for batch mode active learning of supervised models applies, using $\pi$ as the unlabeled data distribution for $r$. Examples of such techniques include selecting batches based on entropy considerations ([44]), gradient-based metrics ([45, 46]), or by attempting to distinguish labeled and unlabeled examples ([47]).

4.2 Sharing parameters between reward model and policy causes overfitting

Although the reward model and policy are both initialized to $\rho$, we train them as separate networks rather than a single shared network with multiple heads. We might expect joint training to be helpful, effectively using RL as an auxiliary task to improve the reward model's performance. Joint training is particularly appealing because it could help the reward model stay strong enough that the policy cannot exploit it. Sharing could also improve computational efficiency, by allowing the models to share activations rather than requiring two separate forward passes.

Despite several attempts, we were not able to make this idea work. The problem comes from the massive imbalance of data: we have at most 60k samples for the reward model, but 2M episodes for the policy. This makes it challenging to maintain performance on both tasks without performing many epochs for the reward model and overfitting. We hope that future work will overcome this challenge.

4.3 Ambiguous tasks make labeling hard

Evaluation of a summary is both subjective and multidimensional. A single human labeler may have a clear notion of whether a given sample is separately accurate, grammatical, nonredundant, or covers all important topics; but in our experiments a labeler will often be asked to choose between samples each of which has some deficiencies. In choosing which of four samples is the best, a labeler must trade off between different desiderata. This makes consistent labeling difficult for honest labelers (including the authors!), and makes it difficult to quickly detect problematic labelers. It also makes the research more difficult to present and interpret: during our experiments we routinely checked the performance of models by having authors label results, since we knew the authors would attempt to do the task honestly, but were epistemically uneasy about reporting these numbers in the paper (Table 8 is the one exception).

One could hope to cope with such "noise" by simply getting more labels and averaging them, but this does not resolve all the practical difficulties with ambiguity. When possible, it seems better to design less ambiguous labeling tasks that get at the same information. For example, rather than asking a person to rate or compare summaries, we could ask for a verbal description of the problems with a summary, or a suggested correction. If problems don't exist we are done; otherwise describing a problem does not require consistently picking the same most important problem. Even if two people disagree on the most important problem, they may be more likely to agree that the other picked some problem, and more agreement eases data quality control and the overall experimental process.

4.4 Bugs can optimize for bad behavior

One of our code refactors introduced a bug which flipped the sign of the reward. Flipping the reward would usually produce incoherent text, but the same bug also flipped the sign of the KL penalty. The result was a model which optimized for negative sentiment while still regularizing towards natural language. Since our instructions told humans to give very low ratings to continuations with sexually explicit text, the model quickly learned to output only content of this form, regardless of how innocuous the starting point was. This bug was remarkable since the result was not gibberish but maximally bad output. The authors were asleep during the training process, so the problem was noticed only once training had finished. A mechanism such as Toyota's Andon cord could have prevented this, by allowing any labeler to stop a problematic training process.

5. Conclusion

Section Summary: The authors applied reinforcement learning with human feedback to fine-tune language models on tasks like generating high-sentiment or descriptive text and producing summaries. This straightforward approach yielded strong results on creative continuations but only extractive copying on summarization, where truthfulness improved at the expense of abstraction, largely due to limitations in online data collection. They argue that such human reward methods are essential both for overcoming the shortcomings of supervised training in interactive settings and for developing safer, scalable techniques to align models with human intent through natural language.

We have demonstrated RL fine-tuning of language models to four NLP tasks: stylistic continuation with high sentiment or physically descriptive language, and summarization on the CNN/Daily Mail and TL;DR datasets. Rather than building task-specific techniques, we achieve our results by straightforwardly applying reward learning to language generation. We extend previous reward learning work with pretrained models and KL regularization to prevent the policy from diverging too far from natural language.

Our results are mixed. On the continuation tasks we achieve good results vs. the zero-shot baseline as evaluated by humans with very few samples: 2.5k for sentiment and 5k for descriptiveness. However, for both summarization tasks our policies are only "smart copiers" (extractive rather than abstractive): they copy from the input text but skip over irrelevant preamble. The advantage of copying is truthfulness: by comparison the zero-shot and supervised models produce natural, plausible-looking summaries that are often lies. We believe the limiting factor in our experiments is data quality, in particular exacerbated by the online data collection setting, and plan to ameliorate this with batched data collection in future.

We believe the application of human reward learning to natural language tasks is important both from a capability and safety perspective. On the capability side, purely supervised training is insufficient to correct mistakes that arise when sampling from trained policies, and RL training to programmatic reward functions such as BLEU or ROUGE is insufficient: [15] conclude that "optimizing for single discrete evaluation metric[s] such as ROUGE with RL can be detrimental to the model quality." Interactive tasks such as dialogue are particularly relevant: it is difficult to define the goal of a dialogue without the human participant, and the length of dialogue makes it more likely that supervised learned models will go off distribution. In the supervised case NLP models are trained using human data; if we want RL fine-tuning we need human data too.

On the AI safety side, interactive communication between humans and ML models is a requirement for scalable reward learning methods such as amplification, debate, and recursive reward modeling ([5, 4, 6]), and natural language is how humans communicate complex ideas. Although language models are unlikely to be ready for these tasks in their full generality, [48] demonstrates that debate already improves generalization for question-answering when debaters quote from a source text. Using direct human preferences for language tasks is a step in the direction of scalable reward learning for language, and we believe further steps are possible.

Acknowledgments

Section Summary: The authors thank several people who helped with their project in different ways. Colleagues at Scale handled much of the data gathering, while others provided support with creating visuals, running training in lower precision, and sharing details about a particular dataset. A number of researchers also gave useful feedback on the paper itself.

We thank Akshat Bubna, Shariq Hashme, and many others at Scale for their work on data collection, Shan Carter for help with visualizations, Scott Gray for help with low precision training, Shahbaz Syed for information about the TL;DR dataset, and Christine Payne, Miles Brundage, Jared Kaplan, Jan Leike, Ethan Perez, and Jelena Luketina for helpful comments on the paper.

Appendix

Section Summary: The appendix describes the detailed instructions provided to human labelers for evaluating text on criteria such as sentiment, descriptiveness, and summarization quality across different tasks. It also explains the quality assurance procedures managed by Scale AI, including the use of benchmark data points to train and filter freelance labelers, dynamic monitoring of performance, and post-collection analysis of agreement rates between labelers and authors, which showed moderate consistency but room for improvement over author-only labeling. Finally, it lists various sample model outputs for sentiment continuation, descriptiveness, and summarization tasks, along with notes on issues like the challenges of using numerical rating scales and the overall human costs involved.

A. Instructions for labelers

A.1 Sentiment

A.2 Descriptiveness

A.3 Summarization: TL;DR

A.4 Summarization: CNN/DM

B. Human labeling details

Our quality assurance process was handled by Scale AI, though Scale made significant changes to their usual quality systems in order to deal with subjective tasks and provide very fast turnaround. Since we initially believed online data collection would be crucial, even the offline experiments were collected with this fast turnaround requirement. In the future we plan to use a more relaxed latency requirement.

The first step of data collection involves teaching the task to a small number of trusted Scale labelers by giving them a description of the task (Appendix A). Scale uses these labelers to collect a large number of benchmark data points where several trusted labelers agree (out of a large set of unlabeled data points from $\rho$). During full data collection, Scale serves these benchmark data points to freelance workers alongside real unlabeled data for training (the two types of data are indistinguishable when $\pi = \rho$, though they do become distinguishable during training), maintaining a confidence model for the performance of each labeler on the benchmark distribution. The probability of getting a benchmark vs. a real sample varies dynamically on factors such as confidence in the labeler to correctly label a certain category. Freelancers who fail to perform well on benchmark tasks are filtered out. Additionally, Scale makes ad-hoc improvements to quality control over time, sometimes validating quality by comparing to a small number of gold-standard labels from the authors.

\begin{tabular}{l|r|r}
\hline
$P(\mathrm{agreement})$ & \multicolumn{1}{l|}{Sentiment} & \multicolumn{1}{l}{TL;DR} \\
\hline
\textit{Between random responses} & 25\%\phantom{$\pm$ 0} & 25\%\phantom{$\pm$ 0} \\
Between labelers & 38 $\pm$ 2\% & 46 $\pm$ 2\% \\  
Between an author \& a labeler & 44 $\pm$ 5\% & 38 $\pm$ 5\% \\
Between authors & 62 $\pm$ 5\% & 61 $\pm$ 5\% \\
\hline
\end{tabular}

We evaluated the data quality after the fact on two of the tasks. During all data collection, 5% of queries were answered by 5 distinct labelers. We sampled 100 of these queries (restricting to ones generated from $\rho$) and had two authors label each one. Based on this data, we estimated the rate of agreement between authors and Scale labelers, pairs of labelers, and pairs of authors. As Table 9 shows, the data contained a significant amount of signal but did not match the quality of data which was hand-labeled by the authors.

An earlier version asked labelers for 1-10 ratings; in the best case this provides more information per label, but it was difficult to gauge labeler performance. Normalization was required since two good labelers would often differ by a (noisy) monotonic transform. If many scores concentrated on a few values (say 7 and 8) simple strategies could fool the filtering process. Absolute scores also tended to drift over the training process, as labelers would adjust to the new distribution of samples from the changing policy.

Finding high-quality workers involves human answering quality control questions which are not used in our experiments, and throwing away data from low-quality workers. So the total human cost of experiments is somewhat higher than the number of labels we actually use (which is what we report). For a short training run this can easily dominate the actual label requirements, though it can be amortized across several tasks by identifying consistently good workers. For our longer training runs the additional number of labels was modest. (None of these details are exposed to customers.)

C. Samples

Samples from our models are shown in the following tables:

• Mock sentiment continuation without a KL penalty: Table 10
• Sentiment continuation: Table 11
• Descriptiveness continuation: Table 12
• TL;DR summarization: Table 13, Table 14, and Table 15
• CNN/Daily Mail summarization: Table 16, Table 17, and Table 18

\begin{tabular}{|p{2cm}|p{6cm}|p{6cm}|}

\hline

\hline

\textbf{context}
 {} &
\multicolumn{2}{|C{12cm}|}{ 
And she would always love him for that. And for many other reasons. He had given her so much and she
}
\\
\hline
 {} &
\textit{Continuation 1}
 {} &
\textit{Continuation 2}
\\
\hline
\textbf{zero-shot}
 {} &
had no idea how to release that gift, but she knew
 {} &
had been unable to give it back.
\newline\newline
``You
\\
\hline
\textbf{no penalty / bonus}
 {} &
! These These These sound flowed instantly easily easily easily easily!
 {} &
! These These These sound flowed instantly easily easily easily easily!
\\
\hline
\textbf{entropy bonus}
 {} &
initially initiallyprisingly easilyprisingly Liam! Love this ride instantly tightly
 {} &
' equally particularlyprisinglyprisingly Liam Liam! Love this ride instantly
\\
\hline
\hline

\textbf{context}
 {} &
\multicolumn{2}{|C{12cm}|}{ 
Palmer rolled her eyes. ''Don't give me that. You didn't come here, wearing that
}
\\
\hline
 {} &
\textit{Continuation 1}
 {} &
\textit{Continuation 2}
\\
\hline
\textbf{zero-shot}
 {} &
corset and corset and a blasted corset. If
 {} &
old-fashioned, sexy little dress. You came here to
\\
\hline
\textbf{no penalty / bonus}
 {} &
! These These These sound flowed instantly easily easily easily easily!
 {} &
! These These These sound flowed instantly easily easily easily easily!
\\
\hline
\textbf{entropy bonus}
 {} &
initially relativelyprisingly instantlyprisingly Liam Liam! Love this ride instantly
 {} &
overwhelmingly Loganprisingly Liam instantly surprisingly! Love this ride instantly tightly
\\
\hline
\hline

\textbf{context}
 {} &
\multicolumn{2}{|C{12cm}|}{ 
Is it conceivable that - I don't know - he somehow wasn't on the aircraft that exploded? Like maybe he
}
\\
\hline
 {} &
\textit{Continuation 1}
 {} &
\textit{Continuation 2}
\\
\hline
\textbf{zero-shot}
 {} &
simply didn't hear me coming or he was just coming out
 {} &
was just on the ground.``
\newline\newline
''Yes, at
\\
\hline
\textbf{no penalty / bonus}
 {} &
! These These These sound flowed instantly easily easily easily easily!
 {} &
! These These These sound flowed instantly easily easily easily easily!
\\
\hline
\textbf{entropy bonus}
 {} &
prisinglyprisingly instantly Liam instantly overwhelmingly! Love this ride instantly tightly
 {} &
instantly initially overwhelmingly Chloe! Love this ride instantly instantly instantly tightly
\\
\hline

\end{tabular}

::: {caption="Table 11: Continuations for our positive-sentiment continuation task, with contexts randomly sampled from BookCorpus test set. Random samples from pretrained, mock sentiment, and 5k offline descriptiveness models, using temperature T=0.7."}

:::

::: {caption="Table 12: Continuations for our descriptiveness continuation task, with contexts randomly sampled from BookCorpus test set. Random samples from pretrained and 5k offline descriptiveness models, using temperature T=0.7."}

:::

::: {caption="Table 13: Random (T=0.7) summaries for our TL;DR summarization task, with contexts randomly sampled from TL;DR validation set."}

:::

::: {caption="Table 14: Random (T=0.7) summaries for our TL;DR summarization task, with contexts randomly sampled from TL;DR validation set."}

:::

::: {caption="Table 15: Random (T=0.7) summaries for our TL;DR summarization task, with contexts randomly sampled from TL;DR validation set."}

:::

::: {caption="Table 16: Random (T=0.5) summaries for our CNN/DM summarization task, with contexts (shown truncated) randomly sampled from CNN/DM test set."}

:::

::: {caption="Table 17: Random (T=0.5) summaries for our CNN/DM summarization task, with contexts (shown truncated) randomly sampled from CNN/DM test set."}

:::

::: {caption="Table 18: Random (T=0.5) summaries for our CNN/DM summarization task, with contexts (shown truncated) randomly sampled from CNN/DM test set."}

:::

References

Section Summary: The references section lists academic papers that examine how AI systems can learn from human preferences and feedback, especially through reinforcement learning techniques applied to tasks like language generation and summarization. Many entries focus on improving AI safety and alignment by training models to better understand goals or by using methods such as debate and reward modeling. Other citations cover related advances in language models, neural machine translation, and sequence-to-sequence training that support more effective text handling.

[1] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4302–4310, 2017.

[2] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems, 2018. URL https://arxiv.org/abs/1811.06521.

[3] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint arXiv:1806.01946, 2018.

[4] Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899, 2018. URL https://arxiv.org/abs/1805.00899.

[5] Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018.

[6] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018.

[7] Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079–3087, 2015.

[8] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018.

[9] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.

[10] Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. Sample efficient text summarization using a single pre-trained transformer. arXiv preprint arXiv:1905.08836, 2019.

[11] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018.

[12] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.

[13] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732, 2015.

[14] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.

[15] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304, 2017.

[16] Yuxiang Wu and Baotian Hu. Learning to extract coherent summary via deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.

[17] Yang Gao, Christian M Meyer, Mohsen Mesgar, and Iryna Gurevych. Reward learning for efficient reinforcement learning in extractive document summarisation. arXiv preprint arXiv:1907.12894, 2019b.

[18] Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1645–1654. JMLR. org, 2017.

[19] Pradyumna Tambwekar, Murtaza Dhuliawala, Animesh Mehta, Lara J Martin, Brent Harrison, and Mark O Riedl. Controllable neural story generation via reinforcement learning. arXiv preprint arXiv:1809.10736, 2018.

[20] Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. arXiv preprint arXiv:1707.07402, 2017.

[21] Woon Sang Cho, Pengchuan Zhang, Yizhe Zhang, Xiujun Li, Michel Galley, Chris Brockett, Mengdi Wang, and Jianfeng Gao. Towards coherent and cohesive long-form text generation. In Proceedings of the First Workshop on Narrative Understanding, pages 1–11, 2019.

[22] Yang Gao, Christian M Meyer, and Iryna Gurevych. Preference-based interactive multi-document summarisation. arXiv preprint arXiv:1906.02923, 2019a.

[23] Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. arXiv preprint arXiv:1805.10627, 2018.

[24] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019.

[25] Sanghyun Yi, Rahul Goel, Chandra Khatri, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur. Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. arXiv preprint arXiv:1904.13015, 2019.

[26] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. In Proceedings of the Twenty-Eight International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, 8 2019.

[27] Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415, 2019.

[28] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693–1701, 2015.

[29] Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63, 2017.

[30] Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368, 2017.

[31] Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. Bottom-up abstractive summarization. arXiv preprint arXiv:1808.10792, 2018.

[32] Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444, 2017.

[33] Forian Böhm, Yang Gao, Christian Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better rewards yield better summaries: Learning to summarise without references. In Conference on Empirical Methods in Natural Language Processing, 2019.

[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

[35] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015.

[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.

[37] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27, 2015.

[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

[39] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github.com/openai/baselines, 2017.

[40] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 43–52. ACM, 2015.

[41] Sebastian Gehrmann, Zachary Ziegler, and Alexander Rush. Generating abstractive summaries with finetuned language models. In TL;DR Challenge System Descriptions, 2019.

[42] Wojciech Kryściński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Neural text summarization: A critical evaluation. arXiv preprint arXiv:1908.08960, 2019.

[43] Jiwei Li, Alexander H Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. Dialogue learning with human-in-the-loop. arXiv preprint arXiv:1611.09823, 2016.

[44] Yuhong Guo and Dale Schuurmans. Discriminative batch mode active learning. In Advances in neural information processing systems, pages 593–600, 2008.

[45] Jiaji Huang, Rewon Child, Vinay Rao, Hairong Liu, Sanjeev Satheesh, and Adam Coates. Active learning for speech recognition: the power of gradients. arXiv preprint arXiv:1612.03226, 2016.

[46] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671, 2019.

[47] Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. arXiv preprint arXiv:1907.06347, 2019.

[48] Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, and Kyunghyun Cho. Finding generalizable evidence by learning to convince Q&A models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China, November 2019. Association for Computational Linguistics.