Self-Rag: Self-reflective Retrieval augmented Generation

Akari Asai$^{\dagger}$, Zeqiu Wu$^{\dagger}$, Yizhong Wang$^{\dagger,\S}$, Avirup Sil$^{\ddagger}$, Hannaneh Hajishirzi$^{\dagger,\S}$

$^{\dagger}$University of Washington $^{\S}$Allen Institute for AI $^{\ddagger}$IBM Research AI

{akari,zeqiuwu,yizhongw,hannaneh} @cs.washington.edu, [email protected]

Abstract

Despite their remarkable capabilities, large language models (LLMs) often produce responses containing factual inaccuracies due to their sole reliance on the parametric knowledge they encapsulate. Retrieval-Augmented Generation (RAG), an ad hoc approach that augments LMs with retrieval of relevant knowledge, decreases such issues. However, indiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary, or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation. We introduce a new framework called Self-Reflective Retrieval-Augmented Generation ($\textsc{Self-Rag}$) that enhances an LM's quality and factuality through retrieval and self-reflection. Our framework trains a single arbitrary LM that adaptively retrieves passages on-demand, and generates and reflects on retrieved passages and its own generations using special tokens, called reflection tokens. Generating reflection tokens makes the LM controllable during the inference phase, enabling it to tailor its behavior to diverse task requirements. Experiments show that $\textsc{Self-Rag}$ (7B and 13B parameters) significantly outperforms state-of-the-art LLMs and retrieval-augmented models on a diverse set of tasks. Specifically, $\textsc{Self-Rag}$ outperforms ChatGPT and retrieval-augmented Llama2-chat on Open-domain QA, reasoning and fact verification tasks, and it shows significant gains in improving factuality and citation accuracy for long-form generations relative to these models.[^1]

[^1]: Our code and trained models are available at https://selfrag.github.io/.

Executive Summary: Self-RAG addresses a persistent weakness of large language models: their tendency to produce plausible but factually incorrect answers when they rely solely on knowledge stored in their parameters. Standard retrieval-augmented generation (RAG) mitigates some errors by adding external passages, yet it always retrieves a fixed number of documents regardless of need and provides no guarantee that the final output is supported by those documents. These shortcomings reduce versatility on tasks that do not require external facts and make outputs harder to verify.

The work introduces a single training framework, Self-Reflective Retrieval-Augmented Generation (Self-RAG), that teaches any language model to decide when retrieval is useful, to critique the relevance and support of retrieved passages, and to evaluate the overall quality of its own segments. The authors first train a critic model on GPT-4-generated labels to insert special “reflection tokens” into training examples offline. They then fine-tune Llama-2 models (7 B and 13 B parameters) on 150 k instruction-following examples augmented with these tokens. At inference the model predicts a retrieval token on demand, processes multiple passages in parallel, and uses the resulting critique scores to guide a segment-level beam search. Weights on the critique dimensions can be adjusted at test time to favor factuality, fluency, or other priorities without further training.

Experiments on six tasks show that both Self-RAG sizes outperform ChatGPT, Llama-2-chat, and conventional RAG baselines on open-domain question answering, fact verification, scientific reasoning, and long-form generation. Gains are largest on factuality (FactScore) and citation accuracy; on biographies and the ASQA long-form QA task, Self-RAG produces outputs that are more fully supported by cited evidence while remaining competitive in fluency. Ablations confirm that on-demand retrieval, relevance and support filtering, and critique-guided selection each contribute measurably to the improvements.

These results indicate that explicit self-reflection can deliver more reliable and verifiable generations without sacrificing the model’s original versatility or requiring proprietary data. Organizations that need trustworthy answers on knowledge-intensive tasks can obtain higher citation precision and fewer unsupported claims from models that are an order of magnitude smaller than the strongest closed systems. At the same time, the framework still occasionally produces partially unsupported statements, its performance depends on retriever quality, and results are reported on a fixed set of evaluation datasets. Scaling the training corpus, testing on additional domains, and integrating stronger retrievers are therefore the most immediate next steps before broad deployment.

1. Introduction

Section Summary: Large language models often produce factual mistakes even after scaling up in size, and standard retrieval methods that add outside passages to their inputs can sometimes add irrelevant details or fail to ensure the generated text stays consistent with those passages. Self-RAG addresses this by training a model to generate special reflection tokens that let it decide on its own when retrieval would help, judge the relevance and quality of retrieved passages, critique its own output segments for factuality, and provide citations. The result is more accurate and verifiable text across tasks, while remaining adaptable to different user needs through adjustable decoding.

State-of-the-art LLMs continue to struggle with factual errors ([1, 2]) despite their increased model and data scale ([3]). Retrieval-Augmented Generation (RAG) methods (Figure 1 left; [4, 5]) augment the input of LLMs with relevant retrieved passages, reducing factual errors in knowledge-intensive tasks ([6, 7]). However, these methods may hinder the versatility of LLMs or introduce unnecessary or off-topic passages that lead to low-quality generations ([8]) since they retrieve passages indiscriminately regardless of whether the factual grounding is helpful. Moreover, the output is not guaranteed to be consistent with retrieved relevant passages ([9]) since the models are not explicitly trained to leverage and follow facts from provided passages. This work introduces Self-Reflective Retrieval-augmented Generation ($\textsc{Self-Rag}$) to improve an LLM's generation quality, including its factual accuracy without hurting its versatility, via on-demand retrieval and self-reflection. We train an arbitrary LM in an end-to-end manner to learn to reflect on its own generation process given a task input by generating both task output and intermittent special tokens (i.e., reflection tokens). Reflection tokens are categorized into retrieval and critique tokens to indicate the need for retrieval and its generation quality respectively (Figure 1 right). In particular, given an input prompt and preceding generations, $\textsc{Self-Rag}$ first determines if augmenting the continued generation with retrieved passages would be helpful. If so, it outputs a retrieval token that calls a retriever model on demand (Step 1). Subsequently, $\textsc{Self-Rag}$ concurrently processes multiple retrieved passages, evaluating their relevance and then generating corresponding task outputs (Step 2). It then generates critique tokens to criticize its own output and choose best one (Step 3) in terms of factuality and overall quality. This process differs from conventional RAG (Figure 1 left), which consistently retrieves a fixed number of documents for generation regardless of the retrieval necessity (e.g., the bottom figure example does not require factual knowledge) and never second visits the generation quality. Moreover, $\textsc{Self-Rag}$ provides citations for each segment with its self-assessment of whether the output is supported by the passage, leading to easier fact verification.

$\textsc{Self-Rag}$ trains an arbitrary LM to generate text with reflection tokens by unifying them as the next token prediction from the expanded model vocabulary. We train our generator LM on a diverse collection of text interleaved with reflection tokens and retrieved passages. Reflection tokens, inspired by reward models used in reinforcement learning ([10, 3]), are inserted offline into the original corpus by a trained critic model. This eliminates the need to host a critic model during training, reducing overhead. The critic model, in part, is supervised on a dataset of input, output, and corresponding reflection tokens collected by prompting a propriety LM (i.e., GPT-4; [11]). While we draw inspiration from studies that use control tokens to start and guide text generation ([12, 13]), our trained LM uses critique tokens to assess its own predictions after each generated segment as an integral part of the generation output.

$\textsc{Self-Rag}$ further enables a customizable decoding algorithm to satisfy hard or soft constraints, which are defined by reflection token predictions. In particular, our inference-time algorithm enables us to (1) flexibly adjust retrieval frequency for different downstream applications and (2) customize models' behaviors to user preferences by leveraging reflection tokens through segment-level beam search using the weighted linear sum of the reflection token probabilities as segment score.

Empirical results on six tasks, including reasoning and long-form generation, demonstrate that $\textsc{Self-Rag}$ significantly outperforms pre-trained and instruction-tuned LLMs that have more parameters and widely adopted RAG approaches with higher citation accuracy. In particular, $\textsc{Self-Rag}$ outperforms retrieval-augmented ChatGPT on four tasks, Llama2-chat ([14]) and Alpaca ([15]) on all tasks. Our analysis demonstrates the effectiveness of training and inference with reflection tokens for overall performance improvements as well as test-time model customizations (e.g., balancing the trade-off between citation previsions and completeness).

2. Related Work

Section Summary: This section reviews prior work on retrieval-augmented generation, where language models are enhanced by adding external text passages to improve performance on knowledge-heavy tasks, though often at the cost of speed, reliability when passages are off-topic, and clear sourcing. It notes several recent papers that fine-tune retrievers or use separate models to filter or select passages, and contrasts them with the Self-RAG approach of training a single model to decide when retrieval is needed and to reflect on output quality. The section also compares the method to reinforcement learning from human feedback and other critique-guided systems, highlighting that Self-RAG uses offline reflection tokens for lower-cost training and more flexible control during generation.

Retrieval-Augmented Generation. Retrieval-Augmented Generation (RAG) augments the input space of LMs with retrieved text passages ([5, 4]), leading to large improvements in knowledge-intensive tasks after fine-tuning or used with off-the-shelf LMs ([6]). A more recent work ([16]) instruction-tunes an LM with a fixed number of retrieved passages prepended to input, or pre-train a retriever and LM jointly, followed by few-shot fine-tuning on task datasets ([17]). While prior work often retrieves only once at the beginning, [18] propose to adaptively retrieve passages for generation on top of a proprietary LLM or [19] train an LM to generate API calls for named entities. Yet, the improved task performance of such approaches often comes at the expense of runtime efficiency ([1]), robustness to irrelevant context ([8]), and lack of attributions ([20, 9]). We introduce a method to train an arbitrary LM to learn to use retrieval on-demand for diverse instruction-following queries and introduce controlled generation guided by reflections tokens to further improve generation quality and attributions.

Concurrent RAG work. A few concurrent works[^2] on RAG propose new training or prompting strategies to improve widely-adopted RAG approaches. [21] fine-tune both the retriever and LM on instruction-tuning datasets in two steps. While we also train our model on diverse instruction-following datasets, $\textsc{Self-Rag}$ enables retrieval on demand and selection of the best possible model output via fine-grained self-reflection, making it widely applicable and more robust and controllable. [22] use a natural language inference model and [23] use a summarization model to filter out or compress retrieved passages before using them to prompt the LM to generate the output. $\textsc{Self-Rag}$ processes passages in parallel and filters out irrelevant ones through self-reflection, without relying on external models at inference. Moreover, our self-reflection mechanism also evaluates other aspects of the model output quality including factuality. LATS ([24]) prompt off-the-shelf LMs to search for relevant information for question answering tasks and to generate with tree search, guided by LM-generated value scores. While their value function simply indicates an overall score of each generation, $\textsc{Self-Rag}$ trains to an arbitrary LM to learn to generate fine-grained self-reflection and customizable inference.

[^2]: All work is arXived within a week of this preprint.

Training and generating with critics. Training LLMs with reinforcement learning (e.g., Proximal Policy Optimization or PPO; [25]) from human feedback (RLHF) has proven effective in aligning LLMs with human preferences ([3]). [26] introduce fine-grained RLHF with multiple reward models. Though our work also studies fine-grained critique on retrieval and generation, we train our target LM on task examples augmented with reflection tokens from a critic model offline, with a far lower training cost compared to RLHF. In addition, reflection tokens in $\textsc{Self-Rag}$ enable controllable generation at inference, while RLHF focuses on human preference alignment during training. Other works use general control tokens to guide LM generation ([12, 27]), while $\textsc{Self-Rag}$ uses reflection tokens to decide the need for retrieval and to self-evaluate generation quality. [28] propose a self-evaluation-guided decoding framework, but they focus only on reasoning tasks with one evaluation dimension (reasoning path consistency) and without retrieval. Recent work on LLM refinement ([29, 30, 31]) prompts a model to generate task output, natural language feedback and refined task output iteratively, but at the cost of inference efficiency.

3. $\textsc{Self-Rag}$: Learning to Retrieve, Generate and Critique

Section Summary: Self-Rag is a training framework that lets a language model decide when to pull in outside information and then critique its own draft outputs for accuracy and usefulness. The model learns to insert special reflection tokens that flag whether retrieval is needed, whether a passage is relevant, and whether the generated text is fully supported, turning those signals into ordinary next-token predictions during both training and inference. This approach avoids the indiscriminate retrieval common in other systems while preserving the model's flexibility and avoiding reliance on external critics at runtime.

We introduce Self-Reflective Retrieval-Augmented Generation ($\textsc{Self-Rag}$), shown in Figure 1. $\textsc{Self-Rag}$ is a framework that enhances the quality and factuality of an LLM through retrieval and self-reflection, without sacrificing LLM's original creativity and versatility. Our end-to-end training lets an LM $\mathcal{M}$ generate text informed by retrieved passages, if needed, and criticize the output by learning to generate special tokens. These reflection tokens (Table 1) signal the need for retrieval or confirm the output's relevance, support, or completeness. In contrast, common RAG approaches retrieve passages indiscriminately, without ensuring complete support from cited sources.

3.1 Problem Formalization and Overview

Formally, given input $x$, we train $\mathcal{M}$ to sequentially generate textual outputs $y$ consisting of multiple segments $y=[y_1, \dots, y_T]$, where $y_t$ indicates a sequence of tokens for the $t$-th segment.[^3] Generated tokens in $y_t$ include text from the original vocabulary as well as the reflection tokens (Table 1).

[^3]: In this paper, we treat one sentence as a segment in our experiments, but our framework is applicable to any segment unit (i.e., sub-sentence).

::: {caption="Table 1: Four types of reflection tokens used in Self-Rag. Each type uses several tokens to represent its output values. The bottom three rows are three types of fboxCritique tokens, and the bold text indicates the most desirable critique tokens. x, y, d indicate input, output, and a relevant passage, respectively."}

:::

**Require:** Generator LM M, Retriever R, Large-scale passage collections (d₁, …, d(N))
Input: input prompt x and preceding generation y(<t), Output: next output segment y(t)
M predicts Retrieve given (x,y(<t))
**if** Retrieve == `Yes` **then**
  Retrieve relevant text passages D using R given (x,y(t-1)) // Retrieve
  M predicts
  IsRel
  given x, d and y(t) given x,d, y(<t) for each d ∈ D // Generate
  M predicts
  IsSup
  and
  IsUse
  given x, y(t), d for each d ∈ D // Critique
  Rank y(t) based on
  IsRel
  ,
  IsSup
  ,
  IsUse
  //
**else if** Retrieve == `No` **then**
  M(gen) predicts y(t) given x // Generate
  M(gen) predicts
  IsUse
  given x,y(t) // Critique
**end if**

Inference overview. Figure 1 and Algorithm 1 present an overview of $\textsc{Self-Rag}$ at inference. For every $x$ and preceding generation $y_{<t}$, the model decodes a retrieval token to evaluate the utility of retrieval. If retrieval is not required, the model predicts the next output segment, as it does in a standard LM. If retrieval is needed, the model generates: a critique token to evaluate the retrieved passage's relevance, the next response segment, and a critique token to evaluate if the information in the response segment is supported by the passage. Finally, a new critique token evaluates the overall utility of the response.[^4] To generate each segment, $\textsc{Self-Rag}$ processes multiple passages in parallel and uses its own generated reflection tokens to enforce soft constraints (Section 3.3) or hard control (Algorithm 1) over the generated task output. For instance, in Figure 1 (right), the retrieved passages $d_1$ is selected at the first time step since $d_2$ does not provide direct evidence ($\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ is Irrelevant) and $d_3$ output is only partially supported while $d_1$ are fully supported.

[^4]: We follow [20] in using a "perceived" utility value that is independent of retrieved passages.

Training overview. $\textsc{Self-Rag}$ enables an arbitrary LM to generate text with reflection tokens by unifying them as next token predictions from the expanded model vocabulary (i.e., the original vocabulary plus reflection tokens). Specifically, we train the generator model $\mathcal{M}$ on a curated corpus with interleaving passages retrieved by a retriever $\mathcal{R}$ and reflection tokens predicted by a critic model $\mathcal{C}$ (summarized in Algorithm 2 in Appendix). We train $\mathcal{C}$ to generate reflection tokens for evaluating retrieved passages and the quality of a given task output (Section 3.2.1). Using the critic model, we update the training corpus by inserting reflection tokens into task outputs offline. Subsequently, we train the final generator model ($\mathcal{M}$) using the conventional LM objective (Section 3.2.2) to enable $\mathcal{M}$ to generate reflection tokens by itself without relying on the critic at inference time.

3.2 $\textsc{Self-Rag}$ Training

Here, we describe the supervised data collection and training of two models, the critic $\mathcal{C}$ (Section 3.2.1) and the generator $\mathcal{M}$ (Section 3.2.2).

3.2.1 Training the Critic Model

Data collection for critic model. Manual annotation of reflection tokens for each segment is expensive ([26]). A state-of-the-art LLM like GPT-4 ([11]) can be effectively used to generate such feedback ([32]). However, depending on such proprietary LMs can raise API costs and diminish reproducibility ([33]). We create supervised data by prompting GPT-4 to generate reflection tokens and then distill their knowledge into an in-house $\mathcal{C}$. For each group of reflection tokens, we randomly sample instances from the original training data: ${X^{sample}, Y^{sample}} \sim {X, Y}$. As different reflection token groups have their own definitions and input, as shown in Table 1, we use different instruction prompts for them. Here, we use $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$ as an example. We prompt GPT-4 with a type-specific instruction ("Given an instruction, make a judgment on whether finding some external documents from the web helps to generate a better response.") followed by few-shot demonstrations $I$ the original task input $x$ and output ${y}$ to predict an appropriate reflection token as text: $p(r|I, x, y)$. Manual assessment reveals that GPT-4 reflection token predictions show high agreement with human evaluations. We collect 4k-20k supervised training data for each type and combine them to form training data for $\mathcal{C}$. Appendix D shows the full list of instructions, and Appendix A.1 contains more details and our analysis.

Critic learning.

After we collect training data $\mathcal{D}{critic}$, we initialize $\mathcal{C}$ with a pre-trained LM and train it on $\mathcal{D}{critic}$ using a standard conditional language modeling objective, maximizing likelihood:

$ \max_{\mathcal{C}}\mathbb{E}{((x, y), r) \sim \mathcal{D}{critic}} \log p_{\mathcal{C}} (r| x, y), \text{ }r\text{ for reflection tokens. }\tag{1} $

Though the initial model can be any pre-trained LM, we use the same one as the generator LM (i.e., Llama 2-7B; [14]) for $\mathcal{C}$ initialization. The critic achieves a higher than 90% agreement with GPT-4-based predictions on most reflection token categories (Figure 5 in Appendix).

3.2.2 Training the Generator Model

Data collection for generator.

Given an input-output pair $(x, y)$, we augment the original output $y$ using the retrieval and critic models to create supervised data that precisely mimics the $\textsc{Self-Rag}$ inference-time process (Section 3.1). For each segment $y_t \in y$, we run $\mathcal{C}$ to assess whether additional passages could help to enhance generation. If retrieval is required, the retrieval special token $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$ =Yes is added, and $\mathcal{R}$ retrieves the top $K$ passages, $\mathbf{D}$. For each passage, $\mathcal{C}$ further evaluates whether the passage is relevant and predicts $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ . If a passage is relevant, $\mathcal{C}$ further evaluates whether the passage supports the model generation and predicts $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ . Critique tokens $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ and $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ are appended after the retrieved passage or generations. At the end of the output, $y$ (or $y_T$), $\mathcal{C}$ predicts the overall utility token $\textcolor{#334C99}{\fbox{\textsc{IsUse}}}$

, and an augmented output with reflection tokens and the original input pair is added to $\mathcal{D}_{gen}$. See the example training data in Figure 2.

Generator learning. We train the generator model $\mathcal{M}$ by training on the curated corpus augmented with reflection tokens $\mathcal{D}_{gen}$ using the standard next token objective:

$ \max_{\mathcal{M}}\mathbb{E}{(x, y, r) \sim \mathcal{D}{gen}} \log p_{\mathcal{M}} (y, r| x).\tag{2} $

Unlike $\mathcal{C}$ training Equation (1), $\mathcal{M}$ learns to predict the target output as well as the reflection tokens. During training, we mask out the retrieved text chunks (surrounded by <p> and </p> in Figure 2) for loss calculation and expand the original vocabulary $\mathcal{V}$ with a set of reflection tokens ${\textcolor{#334C99}{\fbox{\textsc{Critique}}}, \textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}}$.

Connections to prior work on learning with critique.

Recent work incorporates additional critique (feedback) during training, e.g., RLHF ([3]) via PPO. While PPO relies on separate reward models during training, we compute critique offline and directly insert them into the training corpus, where the generator LM is trained with a standard LM objective. This significantly reduces training costs compared to PPO. Our work also relates to prior work that incorporates special tokens to control generation ([13, 12, 27]). Our $\textsc{Self-Rag}$ learns to generate special tokens to evaluate its own prediction after each generated segment, enabling the use of a soft re-ranking mechanism or hard constraints at inference (discussed next).

3.3 $\textsc{Self-Rag}$ Inference

Generating reflection tokens to self-evaluate its own output makes $\textsc{Self-Rag}$ controllable during the inference phase, enabling it to tailor its behavior to diverse task requirements. For tasks demanding factual accuracy ([2]), we aim for the model to retrieve passages more frequently to ensure that the output aligns closely with the available evidence. Conversely, in more open-ended tasks, like composing a personal experience essay, the emphasis shifts towards retrieving less and prioritizing the overall creativity or utility score. In this section, we describe approaches to enforce control to meet these distinct objectives during the inference process.

Adaptive retrieval with threshold. $\textsc{Self-Rag}$ dynamically decides when to retrieve text passages by predicting $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$. Alternatively, our framework allows a threshold to be set. Specifically, if the probability of generating the $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$ =Yes token normalized over all output tokens in $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$ surpasses a designated threshold, we trigger retrieval (details in Appendix A.3).

Tree-decoding with critique tokens. At each segment step $t$, when retrieval is required, based either on hard or soft conditions, $\mathcal{R}$ retrieves $K$ passages, and the generator $\mathcal{M}$ processes each passage in parallel and outputs $K$ different continuation candidates. We conduct a segment-level beam search (with the beam size= $B$) to obtain the top- $B$ segment continuations at each timestamp $t$, and return the best sequence at the end of generation. The score of each segment $y_t$ with respect to passage $d$ is updated with a critic score $\mathcal{S}$ that is the linear weighted sum of the normalized probability of each $\textcolor{#334C99}{\fbox{\textsc{Critique}}}$ token type. For each critique token group $G$ (e.g., $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$), we denote its score at timestamp $t$ as $s_t^G$, and we compute a segment score as follows:

$ f(y_t, d, \textcolor{#334C99}{\fbox{\textsc{Critique}}}) = p(y_{t} | x, d, y_{<t})) + \mathcal{S}(\textcolor{#334C99}{\fbox{\textsc{Critique}}})\text{, where}\tag{3} $

$ \mathcal{S}(\textcolor{#334C99}{\fbox{\textsc{Critique}}}) = \sum_{G \in \mathcal{G}} w^G s_t^G \text{ for } \mathcal{G} = { \textcolor{#334C99}{\fbox{\textsc{IsRel}}}, \textcolor{#334C99}{\fbox{\textsc{IsSup}}}, \textcolor{#334C99}{\fbox{\textsc{IsUse}}}},\tag{4} $

where $s_t^G = \frac{p_t(\hat{r})}{\sum_{i=1}^{N^G} p_t(r_i)}$ stands for the generation probability of the most desirable reflection token $\hat{r}$ (e.g., $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ =Relevant) for the critique token type $G$ with $N^G$ distinct tokens (that represent different possible values for $G$). The weights $w^G$ in Equation 4 are hyperparameters that can be adjusted at inference time to enable customized behaviors at test time. For instance, to ensure that result $y$ is mostly supported by evidence, we can set a weight term for the $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ score higher, while relatively lowering weights for other aspects. Alternatively, we could further enforce hard constraints during decoding using $\textcolor{#334C99}{\fbox{\textsc{Critique}}}$. Instead of using a soft reward function in Equation 4, we could explicitly filter out a segment continuation when the model generates an undesirable $\textcolor{#334C99}{\fbox{\textsc{Critique}}}$ token (e.g., $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ =No support) . Balancing the trade-off between multiple preferences has been studied in RLHF ([14, 26]), which often requires training to change models' behaviors. $\textsc{Self-Rag}$ tailors an LM with no additional training.

4. Experiments

Section Summary: In the experiments, researchers tested Self-Rag against a variety of existing language models on tasks ranging from verifying facts and answering multiple-choice science questions to generating short answers on obscure topics and writing longer biographies or responses. They compared versions of the system to models like Llama2 and Alpaca, as well as to retrieval-augmented approaches and commercial systems such as ChatGPT, using metrics focused on accuracy, factual support, and fluent output. The models were trained on a collection of 150,000 instruction examples and run with specific settings for retrieving documents and deciding when to use reflection signals during generation.

4.1 Tasks and Datasets

We conduct evaluations of our $\textsc{Self-Rag}$ and diverse baselines on a range of downstream tasks, holistically evaluating outputs with metrics designed to assess overall correctness, factuality, and fluency. Throughout these experiments, we conduct zero-shot evaluations, where we provide instructions describing tasks without few-shot demonstrations ([34, 35]). Details of our experiments' settings, including test-time instructions, are available in the Appendix B.1.

Closed-set tasks include two datasets, i.e., a fact verification dataset about public health (PubHealth; [36]) and a multiple-choice reasoning dataset created from scientific exams (ARC-Challenge; [37]). We use accuracy as an evaluation metric and report on the test set. We aggregate the answer probabilities of target classes for both of these datasets (Appendix B.2).

Short-form generations tasks include two open-domain question answering (QA) datasets, PopQA ([1]) and TriviaQA-unfiltered ([38]), where systems need to answer arbitrary questions about factual knowledge. For PopQA, we use the long-tail subset, consisting of 1, 399 rare entity queries whose monthly Wikipedia page views are less than 100. As the TriviaQA-unfiltered (open) test set is not publicly available, we follow prior work's validation and test split ([39, 5]), using 11, 313 test queries for evaluation. We evaluate performance based on whether gold answers are included in the model generations instead of strictly requiring exact matching, following [1, 19].

Long-form generation tasks include a biography generation task ([2]) and a long-form QA task ALCE-ASQA [9, 40]. We use FactScore ([2]) to evaluate biographies, and we use official metrics of correctness (str-em), fluency based on MAUVE ([41]), and citation precision and recall ([9]) for ASQA. ^5

4.2 Baselines

Baselines without retrievals. We evaluate strong publicly available pre-trained LLMs, Llama2 ${\textsc{7b}, \textsc{13b}}$ ([14]), instruction-tuned models, Alpaca ${\textsc{7b}, \textsc{13b}}$ ([15]) (our replication based on Llama2); and models trained and reinforced using private data, ChatGPT ([3]) and Llama2-chat ${\textsc{13b}}$. For instruction-tuned LMs, we use the official system prompt or instruction format used during training if publicly available. We also compare our method to concurrent work, CoVE $\textsc{65b}$ ([29]), which introduces iterative prompt engineering to improve the factuality of LLM generations.

Baselines with retrievals. We evaluate models augmented with retrieval at test time or during training. The first category includes standard RAG baselines, where an LM (Llama2, Alpaca) generates output given the query prepended with the top retrieved documents using the same retriever as in our system. It also includes Llama2-FT, where Llama2 is fine-tuned on all training data we use without the reflection tokens or retrieved passages. We also report the result of retrieval-augmented baselines with LMs trained with private data: Ret-ChatGPT and Ret-Llama2-chat, which deploy the same augmentation technique above, as well as perplexity.ai, an InstructGPT-based production search system. The second category includes concurrent methods that are trained with retrieved text passages, i.e., SAIL ([16]) to instruction-tune an LM on the Alpaca instruction-tuning data with top retrieved documents inserted before instructions, and Toolformer ([19]) to pre-train an LM with API calls (e.g., Wikipedia APIs).[^6]

[^6]: We report numbers using the results reported in the paper as the implementations are not available.

4.3 Experimental settings

Training data and settings. Our training data consists of diverse instruction-following input-output pairs. In particular, we sample instances from Open-Instruct processed data ([42]) and knowledge-intensive datasets ([43, 40, 44]). In total, we use 150k instruction-output pairs. We use Llama2 7B and 13B ([14]) as our generator base LM, and we use Llama2 7B as our base critic LM. For the retriever model $\mathcal{R}$, we use off-the-shelf Contriever-MS MARCO ([45]) by default and retrieve up to ten documents for each input. More training details are in the Appendix B.1.

Inference settings. As a default configuration, we assign the weight terms $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ , $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ , $\textcolor{#334C99}{\fbox{\textsc{IsUse}}}$ values of 1.0, 1.0 and 0.5, respectively. To encourage frequent retrieval, we set the retrieval threshold to 0.2 for most tasks and to 0 for ALCE ([9]) due to citation requirements. We speed up inference using vllm ([46]). At each segment level, we adopt a beam width of 2. For a token-level generation, we use greedy decoding. By default, we use the top five documents from Contriever-MS MARCO ([45]); for biographies and open-domain QA, we use additional top five documents retrieved by a web search engine, following [16]; for ASQA, we use the author-provided top 5 documents by GTR-XXL ([47]) across all baselines for a fair comparison.

5. Results and Analysis

Section Summary: Self-Rag substantially outperforms both standard fine-tuned language models and existing retrieval-augmented systems across six tasks measuring factuality, correctness, fluency, and citation quality, sometimes surpassing even ChatGPT. Its gains come from training the model to decide when to retrieve passages and to critique its own outputs with special reflection tokens, rather than simply from the training data or retrieval alone. Ablation experiments confirm that removing the critic tokens, adaptive retrieval decisions, or support-based scoring during generation leads to clear drops in performance, validating the contribution of each element.

::: {caption="Table 2: Overall experiment results on six tasks. Bold numbers indicate the best performance among non-proprietary models, and gray-colored bold text indicates the best proprietary model when they outperforms all non-proprietary models. ^* indicates concurrent or recent results reported by concurrent work. – indicates numbers that are not reported by the original papers or are not applicable. Models are sorted based on scale. FS, em, rg, mau, prec, rec denote FactScore (factuality); str-em, rouge (correctness); MAUVE (fluency); citation precision and recall, respectively."}

:::

5.1 Main Results

Comparison against baselines without retrieval. Table 2 (top) presents the baselines without retrieval. Our $\textsc{Self-Rag}$ (bottom two rows) demonstrates a substantial performance advantage over supervised fine-tuned LLMs in all tasks and even outperforms ChatGPT in PubHealth, PopQA, biography generations, and ASQA (Rouge and MAUVE). Our approach also significantly outperforms a concurrent method that employs sophisticated prompt engineering; specifically, on the bio generation task, our 7B and 13B models outperform the concurrent CoVE ([29]), which iteratively prompts Llama2 $_\textsc{65b}$ to refine output.

Comparison against baselines with retrieval. As shown in Table 2 (bottom), our $\textsc{Self-Rag}$ also outperforms existing RAG in many tasks, obtaining the best performance among non-proprietary LM-based models on all tasks. While our method outperforms other baselines, on PopQA or Bio, powerful instruction-tuned LMs with retrieval (e.g., LLama2-chat, Alpaca) show large gains from their non-retrieval baselines. However, we found that these baselines provide limited solutions for tasks where we cannot simply copy or extract sub-strings of retrieved passages. On PubHealth and ARC-Challenge, baselines with retrieval do not improve performance notably from their no-retrieval counterparts. We also observe that most baselines with retrieval struggle to improve citation accuracy. On ASQA, our model shows significantly higher citation precision and recall than all models except ChatGPT. [9] found that ChatGPT consistently exhibits superior efficacy in this particular task, surpassing smaller LMs. Our $\textsc{Self-Rag}$ bridges this performance gap, even outperforming ChatGPT in citation precision, which measures whether the model-generated claim is fully supported by cited evidence. We also found that on the metrics for factual precision, $\textsc{Self-Rag}$ 7B occasionally outperforms our 13B due to the tendency of smaller $\textsc{Self-Rag}$ to often generate precisely grounded yet shorter outputs. Llama2-FT $_{\textsc{7b}}$, which is the baseline LM trained on the same instruction-output pairs as $\textsc{Self-Rag}$ without retrieval or self-reflection and is retrieval-augmented at test time only, lags behind $\textsc{Self-Rag}$. This result indicates $\textsc{Self-Rag}$ gains are not solely from training data and demonstrate the effectiveness of $\textsc{Self-Rag}$ framework.

5.2 Analysis

Ablation studies.

We conduct a set of ablations of our framework to identify which factors play key roles. We evaluate two model variants trained differently than our model: No Retriever trains an LM using the standard instruction-following method given instruction-output pairs, without retrieved passages; No Critic trains an LM trained with input-output pairs that are always augmented with the top one retrieved document without reflection tokens. This is similar to SAIL ([16]), and we use our instruction-output data instead of using the Alpaca dataset ([15]), as in SAIL. We also conduct ablation on our inference-time algorithm, including No retrieval disables retrieval during inference; Hard constraints indicates the model performance that retrieves when $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$ =Yes instead of using the adaptive threshold; Retrieve top 1 always retrieves and uses the top one document only, similar to standard RAG approaches; Remove $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ indicates the model performance that removes $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ score only during critique-guided beam search in Equation 4. In this ablation experiment, we use a training instance size of 50k for a more efficient exploration of training variations. Later in this section, we conduct an analysis of the effect of training data size. We conduct the ablation studies on three datasets, PopQA, PubHealth, and ASQA. On ASQA, we evaluate models on sampled 150 instances and exclude ablations involving adaptive or no retrieval processes.

We show in Figure 3a the ablation results. The top part of the table shows results for training ablations, and the bottom part is for inference ablations. We see that all components play important roles. We also observe a large performance gap between $\textsc{Self-Rag}$ and No Retriever or Critic baselines across tasks, indicating that training an LM with those models largely contributes to the performance gain of $\textsc{Self-Rag}$. Using the top passages regardless of their relevance (Retrieve top 1) as in conventional RAG approaches causes a large drop in PopQA and ASQA, and removing $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ during the beam search results hurts performance on ASQA. This demonstrates the effectiveness of $\textsc{Self-Rag}$ 's capabilities of carefully selecting generations based fine-grained multiple criterion, instead of naively using all of the top passages from the retrieval model or solely depending on relevance scores.

Effects of inference-time customization. One key benefit of our proposed framework is that it enables us to control how much each critique type affects the final generation sampling. We analyze the effects of different parameter weights on the top of our 7B model during inference time on ASQA, where multiple evaluation aspects are considered. Figure 3b shows the effects of changing the weighting term for $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ , which criticizes how supported the output is by the text passage. As the figure shows, increasing the weight leads to positive effects on the models' citation precision since this puts more emphasis on whether model generation is supported by the evidence. On the contrary, a larger weight results in lower MAUVE scores: when generation gets longer and more fluent, there are often more claims that are not fully supported by citations, consistent with findings by [20]. Our framework lets practitioners choose and customize models' behaviors at test time by adjusting such parameters without requiring additional training.

Efficiency and accuracy trade-off. Using our framework, practitioners can adjust how often retrieval occurs using the token probability of reward tokens. We evaluate how this adaptive threshold affects overall accuracy and frequency of retrieval, and we evaluate the performance with varying numbers of threshold $\delta$ (larger $\delta$ results in less retrieval) on PubHealth and PopQA. Figure 3c shows that the model's retrieval frequencies dramatically change on both datasets. as $\delta$ varies. On one hand, performance deterioration by retrieving less is smaller on PubHealth but larger in PopQA.

Effects of training data size. We conduct an analysis of how the data scale affects the model's performance. In particular, we randomly sample 5k, 10k, 20k, and 50k instances from our original 150k training instances, and fine-tune four $\textsc{Self-Rag}$ ${\textsc{7b}}$ variants on those subsets. Then, we compare the model performance on PopQA, PubHealth, and ASQA (citation precision) with our final $\textsc{Self-Rag}$ trained on the full 150k instances. We also evaluate Figure 4a, Figure 4b and Figure 4c shows the models' performance trained on different amount of data. Across all datasets, increasing data size often shows upward trajectories and the improvements are significantly larger in PopQA and ASQA, while we do not observed such significant improvements on Llama2-FT ${\textsc{7b}}$ when increasing the training data from 50k to 150k. These results also indicate that further expanding the training data of $\textsc{Self-Rag}$ may lead to further improvements, although in this work we limit our training data size to 150k.

Human evaluations. We conduct small human evaluations on $\textsc{Self-Rag}$ outputs, as well as the reliability of predicted reflection tokens. In particular, we sampled 50 samples from PopQA and Bio results. Following [48], human annotators evaluate S&P, which indicates whether the model output is plausible (i.e., the output is a reasonable and on-topic response to the question as if it were occurring in a conversation) and supported (i.e., the provided evidence is sufficient to verify the validity of the answer). For S&P, we do not consider the instances where $\textsc{Self-Rag}$ predicts irrelevant or no support. We then ask our annotators whether the model-predicted reflection tokens about $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ and $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ match their inspections (e.g., whether the fully supported output is supported by the cited evidence). Human annotators find $\textsc{Self-Rag}$ answers are often plausible and supported by relevant passages with higher S&P scores on short-form PopQA, which is consistent with [48]. Human annotators also find $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ and $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ reflection token predictions are mostly aligned with their assessments. Table 6 in Appendix shows several annotated examples and explanations on assessments.

6. Conclusion

Section Summary: This paper presents Self-Rag, a framework that improves the accuracy and reliability of large language models by letting them fetch external information only when needed and then critique their own outputs. It does so by training the model to generate both normal text and special markers, called reflection tokens, that guide retrieval, fact-checking, and response adjustments. Experiments on six tasks show the approach outperforms larger models and conventional retrieval methods.

This work introduces $\textsc{Self-Rag}$, a new framework to enhance the quality and factuality of LLMs through retrieval on demand and self-reflection. $\textsc{Self-Rag}$ trains an LM to learn to retrieve, generate, and critique text passages and its own generation by predicting the next tokens from its original vocabulary as well as newly added special tokens, called reflection tokens. $\textsc{Self-Rag}$ further enables the tailoring of LM behaviors at test time by leveraging reflection tokens. Our holistic evaluations on six tasks using multiple metrics demonstrate that $\textsc{Self-Rag}$ significantly outperforms LLMs with more parameters or with conventional retrieval-augmented generation approaches.

Ethical Concerns

Section Summary: This research seeks to make large language models produce more accurate information and avoid real harms such as spreading misinformation or giving dangerous advice. The new method improves overall performance and sourcing, yet it can still generate statements that are not fully supported by the cited material. The authors suggest that built-in self-checking and precise source links may help users more easily catch and correct remaining errors.

This work aims to improve the factuality of LLM outputs, the lack of which continues to cause numerous real-world problems (e.g., spread of misinformation and provision of incorrect and dangerous advice). While our method shows significant improvements in terms of performance, factuality, and citation accuracy, it can still generate outputs that are not fully supported by the citations. We hope that explicit self-reflection and fine-grained attribution may help users verify factual errors in the model outputs.

Acknowledgments

We thank Sewon Min, Scott Wen-tau Yih, Sean Welleck, and Kawin Ethayarajh for fruitful discussions in the early stages of this work. We thank Sewon Min, Joongwon (Daniel) Kim, and Sandy Kaplan for valuable feedback on the paper, and Tianyu Gao and Weijia Shi for their help on evaluations. Akari Asai is supported by the IBM Fellowship. We thank Stability AI for providing computing to train and evaluate the LMs in this work, and Microsoft Accelerate Foundation Models Research Program for the access to OpenAI APIs. This work was funded in part by the DARPA MCS program through NIWC Pacific (N66001-19-2-4031), NSF IIS-2044660, and gifts from AI2.

Appendix

Section Summary: The appendix details the reflection tokens used in Self-Rag, which assess whether retrieval is needed, how relevant and supportive retrieved evidence is for the output, and how useful the response is overall. It explains the process of gathering labeled examples via GPT-4 prompts on diverse instruction and question-answering datasets, followed by training a critic model to predict these tokens and then using its outputs to fine-tune the main generator model on over 145,000 instances. Manual checks and accuracy tests confirm that the critic aligns well with GPT-4 judgments.

A. $\textsc{Self-Rag}$ Details

A.1 Reflection Tokens.

Definitions of reflection tokens.

Below, we provide a detailed definition of reflection type and output tokens. The first three aspects will be provided at each segment level, while the final aspect is only given at each output level.

• Retrieval-on-demand ($\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$): Given an input and previous-step generation (if applicable), an LM determines whether the continuation requires factual grounding. No indicates retrieval is unnecessary as the sequence does not require factual grounding or may not be enhanced by knowledge retrieval, Yes indicates retrieval is necessary. We additionally have continue to use evidence, which indicates that a model can continue to use the evidence retrieved previously. For instance, a passage may contain rich factual information, and thus $\textsc{Self-Rag}$ generates multiple segments based on the passage.
• Relevant ($\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$): Retrieved knowledge may not be always relevant to the input. This aspect indicates whether the evidence provides useful information (Relevant) or not (Irrelevant).
• Supported ($\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$): Attribution is the concept of whether the output is fully supported by certain evidence ([48, 49]). This aspect judges how much information in the output is entailed by the evidence. We evaluate attributions in three scale, Fully supported, Partially supported, and No support / Contradictory, following [50, 51].
• Useful ($\textcolor{#334C99}{\fbox{\textsc{IsUse}}}$): Following the definitions from [20], we define the perceived utility as whether the response is a helpful and informative answer to the query, independently from whether it is in fact factual or not. This can be also viewed as plausibility in [48]. For usefulness, we use a five-scale evaluation (1 is the lowest and 5 is the highest).

Details of GPT-4-based data collections.

We use the instruction and demonstration pairs to prompt GPT-4, listed in Appendix D. Following an official recommendation, we separate instructions and outputs with "##". We use the temperature 1 and set the maximum output token counts to be 200. We discard instances where GPT-4 does not follow the designated output formats or output sequences that do not match our expected category names. As a result, we collected 1, 2594 for $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$, 11, 181 for $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ , 19, 317 for relevance, 3, 831 for utility.

Manual analysis of the GPT-4 predictions.

The authors of this paper manually assess randomly sampled 20 instances for each aspect and check if GPT-4 predictions match their assessments given the same instruction, demonstrations, and test instances. We found our assessments show high agreement with GPT-4 predictions, especially for relevance (95%), retrieval necessity (95%), and the degree of support (90%). Agreement was slightly lower in usefulness (80%), mostly due to the disagreement between 1 and 2 or 4 and 5.

A.2 $\textsc{Self-Rag}$ Training

Overview of training.

Algorithm 2 provides a high-level overview of our training.

Input input-output data D = (X, Y), generator M, C θ
Initialize C with a pre-trained LM
Sample data (X(sample), Y(sample)) ∼ (X, Y) // Training Critic LM (Section 3.2.1)
**for** (x, y) ∈ (X(sample), Y(sample)) // Data collections for C **do**
  Prompt GPT-4 to collect a reflection token r for (x,y)
  Add ((x, y, r)) to D(critic)
**end for**
Update C with next token prediction loss // Critic learning; Eq. 1
Initialize M with a pre-trained LM // Training Generator LM (Section 3.2.2)
**for** (x, y) ∈ (X, Y) // Data collection for M with D(critic) **do**
  Run C to predict r given (x, y)
  Add (x, y, r) to D(gen)
**end for**
Update M on D(gen) with next token prediction loss // Generator LM learning; Eq. 2

Full list of seed datasets.

To sample diverse input-output pairs, we sample instances of the Open-Instruct ([42]) dataset. In particular, we use their ShareGPT, GPT-4 Alpaca, Alpaca, OpenAssistant, and FLAN subsets subsets. We also sample instances from a couple of knowledge-intensive datasets, Natural Questions ([52]), Wizard of Wikipedia ([53]) and FEVER ([54]) from the KILT benchmark ([43]), ASQA ([40]) and multiple QA datasets including ARC-Easy and OpenBookQA ([44]). Table 3 shows the full list of training instances, and in total, we use 145, 619 instances.

: Table 3: The generator LM $\mathcal{M}$ training data statistics.

Dataset name	category	Data source	the number of instances
GPT-4 Alpaca	Instruction-following	Open-Instruct	26, 168
Stanford Alpaca	Instruction-following	Open-Instruct	25, 153
FLAN-V2	Instruction-following	Open-Instruct	17, 817
ShareGPT	Instruction-following	Open-Instruct	13, 406
Open Assistant 1	Instruction-following	Open-Instruct	9, 464
Wizard of Wikipedia	Knowledge-intensive	KILT	17, 367
Natural Questions	Knowledge-intensive	KILT	15, 535
FEVER	Knowledge-intensive	KILT	9, 966
OpenBoookQA	Knowledge-intensive	HF Dataset	4, 699
Arc-Easy	Knowledge-intensive	HF Dataset	2, 147
ASQA	Knowledge-intensive	ASQA	3, 897

Performance of the Critic $\mathcal{C}$.

We evaluate the accuracy of reward predictions by splitting GPT-4 generated feedback into training, development, and test sets. The accuracy of the reward model is as follows. Figure 5 shows the model performance of predicting GPT-4 judgments. As you can see, overall our fine-tuned reward model shows high prediction matching with GPT-4 predicted feedback. While our final model uses Llama2-7B as a base LM, we also train and compare FLAN-3B ([34]) model on the same data, to investigate the effectiveness of different data sizes affect final reward predictions. In most aspects, our reward model shows higher than 80% accuracy, indicating the powerful ability of fine-tuned specialized LMs to evaluate text. While both models show relatively lower performance on $\textcolor{#334C99}{\fbox{\textsc{IsUse}}}$ , this is because both models often confuse between the two highest cases (5 and 4), where human annotators can also disagree.

Details of $\mathcal{M}$ data creation.

Here, we provide detailed data creation procedures. Algorithm 3 summarizes the process. Here we set $y_t$ to $y$ for simplification. Once we train the critic model, we first run it on input data from the aforementioned datasets, to predict whether retrieval is needed or not. For the instances where the critic predicts $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$ =No, we only predict the $\textcolor{#334C99}{\fbox{\textsc{IsUse}}}$ given input and output. For the instances where the critic predicts $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$ =Yes, we first retrieve passages using the input and the entire output as queries, to find passages that are relevant to the entire output. We then split output sentences using Spacy.^7 For each sentence, we run $\mathcal{C}$ to predict whether the retrieval is necessary or not, given the input, preceding segments, and the initial retrieved passage. If $\mathcal{C}$ predicts $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$ =No, then do not insert any paragraph at the $t$ th segment. If $\mathcal{C}$ predicts $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$ =Yes, then we use the original input and the $t$ th segment as a retrieval query to find relevant passages for the $t$-th segment. For each retrieved passage, we predict $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ and $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ . If there is any passage and continuation with $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ =Relevant and $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ =Fully Supported / $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ =Partially Supported, then we sample it as the continuation. If there is more than one passage satisfying this criterion, we use the one with the highest retrieval score. If there are only $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ =Irrelevant or $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ =No Support passages, we randomly sample one passage.

Input Input-output data D = X, Y
**for** (x, y) ∈ (X, Y) **do**
  Given (x,y) C predicts Retrieve
  **if** Retrieve is predicted **then**
    Retrieve relevant passages D using R given (x,y)
    //
    **for** d ∈ D **do**
      C predicts
      IsRel
      for each d // Predict relevance of passages
      C predicts
      IsSup
      for each (y, d) // Predict supports of outputs
    **end for**
    C predicts
    IsUse
    for each d // Predict overall utility (t=T only)
    Sample d
  **else if** Retrieve is not predicted **then**
    C predicts
    IsUse
    given x, y
  **end if**
  Add augmented (x, y, d, r) to D(gen)
**end for**

Training examples.

Table 4 show several training examples used for $\mathcal{M}$ training.

::: caption="Table 4: List of the training examples."

:::

A.3 $\textsc{Self-Rag}$ Inference

Details of beam-search score calculations.

We first compute scores for each critique type by taking the normalized probabilities of desirable tokens. For $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ , we compute the score as follows:

$ s(\textcolor{#334C99}{\fbox{\textsc{IsRel}}}) = \frac{p(\textcolor{#334C99}{\fbox{\textsc{IsRel}}} = \textsc{Relevant})}{p(\textcolor{#334C99}{\fbox{\textsc{IsRel}}} = \textsc{Relevant}) + p(\textcolor{#334C99}{\fbox{\textsc{IsRel}}} = \textsc{Irrelevant})}. $

For $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ , we compute the score as follows:

$ s(\textcolor{#334C99}{\fbox{\textsc{IsRel}}}) = \frac{p(\textcolor{#334C99}{\fbox{\textsc{IsSup}}} = \textsc{Fully})}{S} + 0.5 \times \frac{p(\textcolor{#334C99}{\fbox{\textsc{IsSup}}} = \textsc{Partially})}{S}, $

where $S = \sum_{t \in {\textsc{Fully}, \textsc{Partially}, \textsc{No}}}p(\textcolor{#334C99}{\fbox{\textsc{IsSup}}} =t)$. For $\textcolor{#334C99}{\fbox{\textsc{IsUse}}}$ where we have a five-scale score, we compute the weighted sum of the scores. We assigns weighted scores of $w ={-1, -0.5, 0, 0.5, 1}$ to the tokens $\textcolor{#334C99}{\fbox{\textsc{IsUse}}}$ = ${1, 2, 3, 4, 5}$, and compute the final scores as follows:

$ s(\textcolor{#334C99}{\fbox{\textsc{IsUse}}}) = \sum_{i}^5 w_i\frac{p(\textcolor{#334C99}{\fbox{\textsc{IsUse}}} = i)}{S}, $

where $S =\sum_{t \in {1, 2, 3, 4, 5}}p(\textcolor{#334C99}{\fbox{\textsc{IsUse}}} =t)$.

Details of adaptive retrieval.

For retrieval based on soft constraints, we trigger retrieval if the following condition is satisfied:

$ \frac{p(\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}} = \textsc{Yes})}{p(\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}} = \textsc{Yes})+ p(p(\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}} = \textsc{No})} > \delta. $

B. Experimental Details

B.1 More Details of Training

More details of training and computations.

We use 4 Nvidia A100 with 80GB memory to train our models. All models are trained for 3 epochs with a batch size of 128, a peak learning rate of 2e-5 with 3% warmup steps, and linear decay afterward. We set the maximum token length to be 2, 048 for the 7B model, and 1, 524 for the 13B model due to the memory constraint. We use Deepspeed stage 3 ([55]) to conduct multi-GPU distributed training, with training precision Bfloat16 enabled. FlashAttention ([56]) is used to make the long-context training more efficient. We run inference of our trained models using 1-2 Quadro RTX 6000 GPUs with 24GB memory.

B.2 More Details of Evaluations

Retrieval setup details.

By default, we use Contriever-MS MARCO to retrieve the top five documents from Wikipedia{, and use official Wikipedia embeddings based on 2018 English Wikipedia. On PopQA, where question and answer pairs are created based on WikiData in 2022, we found that the 2018 Wikipedia sometimes lacks articles about some entities that have been more recently added to Wikipedia. Therefore, for PopQA, we used the December 2020 preprocessed Wikipedia corpus provided by [17] and generated document embeddings.^8 The issues of performance variance from different Wikipedia dumps have been reported by prior work ([57, 17]). Yet, we observe limited effectiveness of such off-the-shelf retrieval models trained primarily on knowledge-intensive tasks for open-ended generation (e.g., instruction following). Recent or concurrent work studies instruction-tuning of retrieval systems ([58]) or joint training of retrieval and LM components ([21]), while we leave exploring the effectivess of such appraoches for future work. For bio generation and open-domain QA tasks, we additionally retrieve five documents using Google Programmable Search^9 and search documents from English Wikipedia. As this API only provides snippets, we retrieve Wikipedia introductory paragraphs for the corresponding entities.

Detailed experimental settings for individual datasets.

For OpenQA datasets, we set the maximum new token number to 100 tokens. For closed-set tasks (PubHealth and ARC-C), we set the maximum new token length to 50 for all baselines. For $\textsc{Self-Rag}$ inference on PubHealth and ARC-C, instead of determining the output with the highest score Equation 4 as in other tasks, we aggregate the scores for each option and select the answer option with the highest score. We found in zero-shot settings of fact checking, some LLMs can generate capitalized class labels (e.g., True) while our gold labels are lower-cased. Therefore, across different LMs, for fact checking, we lowercase the predictions. In multiple choice tasks, we found some models generate answers in slightly different ways (e.g., (A) instead of A). We slightly modify instructions for each LLM to avoid such format violations, and further conduct string matching between each candidate and model predictions if format violations still remain. After that processing, in closed set tasks, model predictions match one of the gold classes in almost all cases. For ALCE, we found that Llama2-chat tend to generate significantly lower outputs than other models (e.g., on average, their output is nearly 100 token, while ChatGPT generates 40 tokens on average), resulting in inflated str-em scores. We limit the maximum generation length to 100 tokens for all baselines to avoid this issue, rather than the original 300 tokens in the ALCE paper. Consequently, all of the baseline output length is within 30-60 tokens. For FactScore, we set the maximum new token length to 500 for baselines and 200 for $\textsc{Self-Rag}$ at each segment level.

Task-specific instructions.

Table 5 shows the list of the instructions used during evaluations. For Open-domain QA, we do not provide explicit instructions.

: Table 5: Full list of instructions used during zero-shot evaluations. For open-domain QA, we don't use any task specific instruction and simply use the original questions as input query.

Dataset	Instruction
ARC-C	Given four answer candidates, A, B, C and D, choose the best answer choice. Please answer with the capitalized alphabet only, without adding any extra phrase or period.
PubHealth	Is the following statement correct or not? Say true if it's correct; otherwise, say false. Don't capitalize or add periods, just say "true" or "false".
Bio Generation	Tell me a bio about [Person Name]
ASQA (baseline)	Instruction: Write an accurate, engaging, and concise answer for the given question using only the provided search results (some of which might be irrelevant) and cite them properly. Use an unbiased and journalistic tone. Always cite for any factual claim. When citing several search results, use [1][2][3]. Cite at least one document and at most three documents in each sentence. If multiple documents support the sentence, only cite a minimum sufficient subset of the documents.
ASQA (ours)	Answer the following question. The question may be ambiguous and have multiple correct answers, and in that case, you have to provide a long-form answer including all correct answers.

C. Results

C.1 Analysis

Reliance on parametric- and non-parametric memories.

We conduct analysis on how frequently model answers come from retrieved passages (non-parametric memories) or their own parametric memories. On two open-domain QA datasets, TriviaQA and PopQA, we conduct the following analysis: 1) sample query models successfully answer correctly, 2) for each query in this group, check whether the matched ground-truth answer is a sub-string of the retrieved passage or not. We evaluate $\textsc{Self-Rag}$ 7B, Alpaca 7B, Alpaca 13B, and Llama2-Chat-13B. We found that $\textsc{Self-Rag}$ significantly less frequently generates answers that are not included in the provided evidence; in particular, in Alpaca 30B, 20% of the correct predictions are not included in the provided passages, followed by Llama2-chat 13B (18%) and Alpaca (15%), while it is only 2% in $\textsc{Self-Rag}$. When retrieved passages are not relevant, $\textsc{Self-Rag}$ generates $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ =Irrelevant, indicating that the following answers may not be factually grounded, while those instruction-tuned models continue to generate plausible answers.

C.2 Human Evaluation Examples

Table 6 shows examples with human evaluations on S&P and correctness of $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ and $\textcolor{#334C99}{\fbox{\textsc{IsSup}}}$ reflection tokens.

::: caption="Table 6: Examples of human evaluation results."

:::

C.3 Qualitative Examples

Table 7 shows several examples predicted by our $\textsc{Self-Rag}$ (13B). The first example is the model output to an ASQA question. The first reference states that Emperor Constantine made Sunday a day of rest from labor, and further the second citation supports the fact that the official adoption of Sunday as a day of rest by Constantine in AD 321. In the second example, the model predicts Contradictory to the first output as the output says the person has served as the CEO since 2010, while the passage says he stepped down as CEO in 2015. Indicating those factual contradictions as reflection tokens enables to enforcement of hard control and also verification of model outputs easily. In the third example, while the generation is mostly correct, $\textsc{Self-Rag}$ predicts Partially Support to the statement listing the name of the songs, as they were not explicitly mentioned.

::: caption="Table 7: Examples of outputs."

:::

D. Full List of Instructions and Demonstrations for GPT-4

Here, we show the instructions and demonstrations used to prompt GPT-4 to collect reflection tokens. Table 8 shows the instructions and demonstrations for the initial retrieval token. Table 9 shows the instruction and demonstrations used to collect the three-way output tokens for $\textcolor{#B24C00}{\fbox{\textsc{Retrieve}}}$ given instruction, preceding sentences, and previously retrieved passages. Due to the longer demonstration and test input, we only use a single demonstration. Table 10 shows an instruction and demonstrations used to collect the three-way output tokens for $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ . Table 11 shows an instruction and demonstrations used to collect the three-way output tokens for $\textcolor{#334C99}{\fbox{\textsc{IsRel}}}$ . Table 12 shows an instruction and demonstrations used to collect the five-way output tokens for $\textcolor{#334C99}{\fbox{\textsc{IsUse}}}$ .

::: caption="Table 8: Instructions and demonstrations for fboxRetrieve aspect given the input only."

:::

::: {caption="Table 9: Instructions and demonstrations for fboxRetrieve aspect given the input, preceding generations, and retrieved passages."}

:::

::: caption="Table 10: Instructions and demonstrations for fboxIsRel aspect given the input only."

:::

::: caption="Table 11: Instructions and demonstrations for fboxIsSup tokens."

:::

::: caption="Table 12: Instructions and demonstrations for fboxIsUse tokens."

:::

References

Section Summary: This section lists academic citations for a paper on language models, primarily drawing from recent conference proceedings and arXiv preprints. The referenced works address topics such as evaluating factual accuracy in model outputs, combining parametric knowledge with external retrieval, training via human feedback, and challenges like distraction by irrelevant context. They include contributions from researchers at institutions like OpenAI, Meta, and various universities, spanning 2019 to 2023.

[1] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023. URL https://aclanthology.org/2023.acl-long.546.

[2] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023. URL https://arxiv.org/abs/2305.14251.

[3] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=TG8KACxEON.

[4] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, 2020. URL https://proceedings.neurips.cc/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf.

[5] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International Conference on Machine Learning, 2020. URL https://dl.acm.org/doi/pdf/10.5555/3524938.3525306.

[6] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2302.00083.

[7] Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Tutorial), 2023a. URL https://aclanthology.org/2023.acl-tutorials.6.

[8] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning, 2023. URL https://proceedings.mlr.press/v202/shi23a.html.

[9] Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627, 2023. URL https://arxiv.org/abs/2305.14627.

[10] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. URL https://arxiv.org/abs/1909.08593.

[11] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. URL https://arxiv.org/abs/2303.08774.

[12] Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. QUARK: Controllable text generation with reinforced unlearning. In Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=5HaIds3ux5O.

[13] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019. URL https://arxiv.org/abs/1909.05858.

[14] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. URL https://arxiv.org/abs/2307.09288.

[15] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023. URL https://arxiv.org/abs/2305.14387.

[16] Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Sail: Search-augmented instruction learning. arXiv preprint arXiv:2305.15225, 2023. URL https://arxiv.org/abs/2305.15225.

[17] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022b. URL https://arxiv.org/abs/2208.03299.

[18] Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983, 2023. URL https://arxiv.org/abs/2305.06983.

[19] Timo Schick, Jane Dwivedi-Yu, Roberto Dess`ı, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. URL https://arxiv.org/abs/2302.04761.

[20] Nelson F Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. arXiv preprint arXiv:2304.09848, 2023a. URL https://arxiv.org/abs/2304.09848.

[21] Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Scott Yih. Ra-dit: Retrieval-augmented dual instruction tuning, 2023. URL https://arxiv.org/abs/2310.01352.

[22] Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context, 2023. URL https://arxiv.org/abs/2310.01558.

[23] Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented lms with compression and selective augmentation, 2023. URL https://arxiv.org/abs/2310.04408.

[24] Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2023. URL https://arxiv.org/abs/2310.04406.

[25] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347.

[26] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693, 2023. URL https://arxiv.org/abs/2306.01693.

[27] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, 2023. URL https://openreview.net/forum?id=AT8Iw8KOeC.

[28] Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. Decomposition enhances reasoning via self-evaluation guided decoding. arXiv preprint arXiv:2305.00633, 2023. URL https://arxiv.org/abs/2305.00633.

[29] Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495, 2023. URL https://arxiv.org/abs/2309.11495.

[30] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. URL https://arxiv.org/abs/2303.17651.

[31] Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904, 2023. URL https://arxiv.org/abs/2304.01904.

[32] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023b. URL https://arxiv.org/abs/2303.16634.

[33] Lingjiao Chen, Matei Zaharia, and James Zou. How is chatgpt's behavior changing over time? arXiv preprint arXiv:2307.09009, 2023. URL https://arxiv.org/abs/2307.09009.

[34] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=gEZrGCozdqR.

[35] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4.

[36] Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Luc Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Interpretable unified language checking. arXiv preprint arXiv:2304.03728, 2023. URL https://arxiv.org/abs/2304.03728.

[37] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. URL https://arxiv.org/abs/1803.05457.

[38] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017. URL https://aclanthology.org/P17-1147.

[39] Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. A discrete hard EM approach for weakly supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019. URL https://aclanthology.org/D19-1284.

[40] Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. ASQA: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022. URL https://aclanthology.org/2022.emnlp-main.566.

[41] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=Tqx7nJp7PR.

[42] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751, 2023. URL https://arxiv.org/abs/2306.04751.

[43] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021. URL https://aclanthology.org/2021.naacl-main.200.

[44] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. URL https://aclanthology.org/D18-1260.

[45] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022a. URL https://openreview.net/forum?id=jKN1pXi7b0.

[46] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. URL https://arxiv.org/abs/2309.06180.

[47] Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022. URL https://aclanthology.org/2022.emnlp-main.669.

[48] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022. URL https://arxiv.org/abs/2203.11147.

[49] Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037, 2022. URL https://arxiv.org/abs/2212.08037.

[50] Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311, 2023. URL https://arxiv.org/abs/2305.06311.

[51] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. URL https://arxiv.org/abs/2112.09332.

[52] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 2019. URL https://aclanthology.org/Q19-1026.

[53] Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=r1l73iRqKm.

[54] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018. URL https://aclanthology.org/N18-1074.

[55] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2020. URL https://dl.acm.org/doi/10.5555/3433701.3433727.

[56] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=H4DqfPSibmx.

[57] Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. Learning to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJgVHkrYDH.

[58] Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. In Findings of the Association for Computational Linguistics, 2023b. URL https://aclanthology.org/2023.findings-acl.225.