Parameter-Efficient Transfer Learning for NLP

Neil Houlsby $^{1}$
Google Research

Andrei Giurgiu $^{1}$ $^{*}$
Google Research

Stanisław Jastrzębski $^{2}$ $^{*}$
Jagiellonian University

Bruna Morrone $^{1}$
Google Research

Quentin de Laroussilhe $^{1}$
Google Research

Andrea Gesmundo $^{1}$
Google Research

Mona Attariyan $^{1}$
Google Research

Sylvain Gelly $^{1}$
Google Research

$^{*}$ Equal contribution
$^{1}$ Google Research
$^{2}$ Jagiellonian University.
Correspondence to: Neil Houlsby [email protected].
$^{1}$ Code at https://github.com/google-research/adapter-bert

Abstract

Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.$^{1}$

Executive Summary: Large pre-trained language models such as BERT deliver strong results on NLP tasks through fine-tuning, yet fine-tuning becomes impractical when many downstream tasks must be supported. Each new task normally requires a separate full copy of the model, driving up storage and compute costs and complicating deployment in cloud services where tasks arrive sequentially. The paper therefore evaluates an alternative transfer approach based on adapter modules that keeps the original model fixed and adds only a small number of task-specific parameters.

The central objective is to show that adapters can match the accuracy of full fine-tuning while training roughly two orders of magnitude fewer parameters per task and allowing new tasks to be added without revisiting or degrading earlier ones. The method inserts lightweight bottleneck layers (down-project, nonlinearity, up-project) after the attention and feed-forward sublayers of each Transformer block; only these adapters and the associated layer-normalization parameters are trained. Experiments were run on the nine-task GLUE benchmark with BERTLARGE, on 17 additional public classification datasets with BERTBASE, and on SQuAD extractive question answering, with hyper-parameter search over adapter size, learning rate, and training length.

On GLUE the adapters reached a mean score of 80.0 compared with 80.4 for full fine-tuning, using 3.6 % of the trainable parameters per task and a total model size of only 1.3× the base model versus 9× for fine-tuning. Across the 17 extra classification tasks the gap remained 0.4 % while the total parameter count stayed near 1.2×. On SQuAD, adapters of size 2 % of the base model produced an F1 of 90.4 versus 90.7 for full fine-tuning. Performance proved stable across adapter sizes from roughly 0.5 % to 8 % of the original network, and analysis showed that adapters placed in higher layers contribute most, automatically focusing capacity where task-specific features are learned.

These results imply that organizations can host dozens of NLP tasks inside a single shared model footprint, materially lowering serving cost and latency while eliminating catastrophic forgetting. The approach is therefore attractive for any setting that must support a growing stream of classification or question-answering workloads.

The authors recommend adopting adapter tuning as the default transfer strategy whenever multiple tasks must be served from one base model. Immediate next steps are to test adapters on generation and structured-prediction tasks, to combine them with existing model-compression techniques, and to validate scaling behavior beyond the current suite of 27 tasks. The principal limitations are that the study covers only Transformer architectures and that adapter size and learning-rate still require modest per-task tuning; however, the performance margin relative to full fine-tuning is small and consistent across the evaluated workloads, giving high confidence in the core efficiency claim.

1. Introduction

Section Summary: Pre-trained models such as BERT deliver strong results on NLP tasks, but standard transfer methods require training a full new set of parameters for each task, which becomes inefficient when tasks arrive sequentially. The paper proposes inserting small adapter modules between the layers of a frozen pre-trained network so that only a tiny number of new parameters are learned per task. This produces compact, extensible models that match the accuracy of full fine-tuning on benchmarks like GLUE while using roughly 3 percent as many task-specific parameters and without overwriting earlier tasks.

Transfer from pre-trained models yields strong performance on many NLP tasks ([1, 2, 3]). BERT, a Transformer network trained on large text corpora with an unsupervised loss, attained state-of-the-art performance on text classification and extractive question answering ([4]).

In this paper we address the online setting, where tasks arrive in a stream. The goal is to build a system that performs well on all of them, but without training an entire new model for every new task. A high degree of sharing between tasks is particularly useful for applications such as cloud services, where models need to be trained to solve many tasks that arrive from customers in sequence. For this, we propose a transfer learning strategy that yields compact and extensible downstream models. Compact models are those that solve many tasks using a small number of additional parameters per task. Extensible models can be trained incrementally to solve new tasks, without forgetting previous ones. Our method yields a such models without sacrificing performance.

The two most common transfer learning techniques in NLP are feature-based transfer and fine-tuning. Instead, we present an alternative transfer method based on adapter modules ([5]). Features-based transfer involves pre-training real-valued embeddings vectors. These embeddings may be at the word ([6]), sentence ([7]), or paragraph level ([8]). The embeddings are then fed to custom downstream models. Fine-tuning involves copying the weights from a pre-trained network and tuning them on the downstream task. Recent work shows that fine-tuning often enjoys better performance than feature-based transfer ([2]).

Both feature-based transfer and fine-tuning require a new set of weights for each task. Fine-tuning is more parameter efficient if the lower layers of a network are shared between tasks. However, our proposed adapter tuning method is even more parameter efficient. Figure 1 demonstrates this trade-off. The x-axis shows the number of parameters trained per task; this corresponds to the marginal increase in the model size required to solve each additional task. Adapter-based tuning requires training two orders of magnitude fewer parameters to fine-tuning, while attaining similar performance.

Adapters are new modules added between layers of a pre-trained network. Adapter-based tuning differs from feature-based transfer and fine-tuning in the following way. Consider a function (neural network) with parameters $\bm w$: $\phi_{\bm w}(\bm x)$. Feature-based transfer composes $\phi_{\bm w}$ with a new function, $\chi_{\bm v}$, to yield $\chi_{\bm v}(\phi_{\bm w}(\bm x))$. Only the new, task-specific, parameters, $\bm v$, are then trained. Fine-tuning involves adjusting the original parameters, $\bm w$, for each new task, limiting compactness. For adapter tuning, a new function, $\psi_{\bm w, \bm v}(\bm x)$, is defined, where parameters $\bm w$ are copied over from pre-training. The initial parameters $\bm v_0$ are set such that the new function resembles the original: $\psi_{\bm w, \bm v_0}(\bm x) \approx \phi_{\bm w}(\bm x)$. During training, only $\bm v$ are tuned. For deep networks, defining $\psi_{\bm w, \bm v}$ typically involves adding new layers to the original network, $\phi_{\bm w}$. If one chooses $|\bm v|\ll|\bm w|$, the resulting model requires $\sim|\bm w|$ parameters for many tasks. Since $\bm w$ is fixed, the model can be extended to new tasks without affecting previous ones.

Adapter-based tuning relates to multi-task and continual learning. Multi-task learning also results in compact models. However, multi-task learning requires simultaneous access to all tasks, which adapter-based tuning does not require. Continual learning systems aim to learn from an endless stream of tasks. This paradigm is challenging because networks forget previous tasks after re-training ([9, 10]). Adapters differ in that the tasks do not interact and the shared parameters are frozen. This means that the model has perfect memory of previous tasks using a small number of task-specific parameters.

We demonstrate on a large and diverse set of text classification tasks that adapters yield parameter-efficient tuning for NLP. The key innovation is to design an effective adapter module and its integration with the base model. We propose a simple yet effective, bottleneck architecture. On the GLUE benchmark, our strategy almost matches the performance of the fully fine-tuned BERT, but uses only 3% task-specific parameters, while fine-tuning uses 100% task-specific parameters. We observe similar results on a further $17$ public text datasets, and SQuAD extractive question answering. In summary, adapter-based tuning yields a single, extensible, model that attains near state-of-the-art performance in text classification.

2. Adapter tuning for NLP

Section Summary: Adapter tuning allows large pre-trained language models to be adapted to new tasks by inserting small trainable modules into the existing network instead of updating all parameters. These adapter modules use a simple bottleneck design that adds only a few percent more parameters per task, and they are placed after the attention and feed-forward sublayers within each Transformer block while the original weights stay frozen. The method supports sequential training across tasks, keeps the shared base model intact for reuse, and relies on near-identity initialization to maintain stable performance.

We present a strategy for tuning a large text model on several downstream tasks. Our strategy has three key properties: (i) it attains good performance, (ii) it permits training on tasks sequentially, that is, it does not require simultaneous access to all datasets, and (iii) it adds only a small number of additional parameters per task. These properties are especially useful in the context of cloud services, where many models need to be trained on a series of downstream tasks, so a high degree of sharing is desirable.

To achieve these properties, we propose a new bottleneck adapter module. Tuning with adapter modules involves adding a small number of new parameters to a model, which are trained on the downstream task ([5]). When performing vanilla fine-tuning of deep networks, a modification is made to the top layer of the network. This is required because the label spaces and losses for the upstream and downstream tasks differ. Adapter modules perform more general architectural modifications to re-purpose a pre-trained network for a downstream task. In particular, the adapter tuning strategy involves injecting new layers into the original network. The weights of the original network are untouched, whilst the new adapter layers are initialized at random. In standard fine-tuning, the new top-layer and the original weights are co-trained. In contrast, in adapter-tuning, the parameters of the original network are frozen and therefore may be shared by many tasks.

Adapter modules have two main features: a small number of parameters, and a near-identity initialization. The adapter modules need to be small compared to the layers of the original network. This means that the total model size grows relatively slowly when more tasks are added. A near-identity initialization is required for stable training of the adapted model; we investigate this empirically in Section 3.6. By initializing the adapters to a near-identity function, original network is unaffected when training starts. During training, the adapters may then be activated to change the distribution of activations throughout the network. The adapter modules may also be ignored if not required; in Section 3.6 we observe that some adapters have more influence on the network than others. We also observe that if the initialization deviates too far from the identity function, the model may fail to train.

2.1 Instantiation for Transformer Networks

We instantiate adapter-based tuning for text Transformers. These models attain state-of-the-art performance in many NLP tasks, including translation, extractive QA, and text classification problems ([11, 3, 4]). We consider the standard Transformer architecture, as proposed in [11].

Adapter modules present many architectural choices. We provide a simple design that attains good performance. We experimented with a number of more complex designs, see Section 3.6, but we found the following strategy performed as well as any other that we tested, across many datasets.

Figure 2 shows our adapter architecture, and its application it to the Transformer. Each layer of the Transformer contains two primary sub-layers: an attention layer and a feedforward layer. Both layers are followed immediately by a projection that maps the features size back to the size of layer's input. A skip-connection is applied across each of the sub-layers. The output of each sub-layer is fed into layer normalization. We insert two serial adapters after each of these sub-layers. The adapter is always applied directly to the output of the sub-layer, after the projection back to the input size, but before adding the skip connection back. The output of the adapter is then passed directly into the following layer normalization.

To limit the number of parameters, we propose a bottleneck architecture. The adapters first project the original $d$-dimensional features into a smaller dimension, $m$, apply a nonlinearity, then project back to $d$ dimensions. The total number of parameters added per layer, including biases, is $2md+d+m$. By setting $m\ll d$, we limit the number of parameters added per task; in practice, we use around $0.5-8%$ of the parameters of the original model. The bottleneck dimension, $m$, provides a simple means to trade-off performance with parameter efficiency. The adapter module itself has a skip-connection internally. With the skip-connection, if the parameters of the projection layers are initialized to near-zero, the module is initialized to an approximate identity function.

Alongside the layers in the adapter module, we also train new layer normalization parameters per task. This technique, similar to conditional batch normalization ([12]), FiLM ([13]), and self-modulation ([14]), also yields parameter-efficient adaptation of a network; with only $2d$ parameters per layer. However, training the layer normalization parameters alone is insufficient for good performance, see Section 3.4.

3. Experiments

Section Summary: Experiments show that adding small adapter modules to a pre-trained BERT model lets it reach performance nearly identical to standard full fine-tuning on text tasks while training far fewer parameters. On the GLUE benchmark the adapters come within 0.4 points of fine-tuning yet require only about 3 percent as many trainable weights per task, keeping the total model size to roughly 1.3 times the original network rather than nine times. The same pattern holds across seventeen extra classification datasets and SQuAD question answering, with the adapters automatically concentrating changes in the higher layers of the network.

We show that adapters achieve parameter efficient transfer for text tasks. On the GLUE benchmark ([15]), adapter tuning is within $0.4%$ of full fine-tuning of BERT, but it adds only $3%$ of the number of parameters trained by fine-tuning. We confirm this result on a further $17$ public classification tasks and SQuAD question answering. Analysis shows that adapter-based tuning automatically focuses on the higher layers of the network.

3.1 Experimental Settings

We use the public, pre-trained BERT Transformer network as our base model. To perform classification with BERT, we follow the approach in [4]. The first token in each sequence is a special "classification token". We attach a linear layer to the embedding of this token to predict the class label.

Our training procedure also follows [4]. We optimize using Adam ([16]), whose learning rate is increased linearly over the first $10%$ of the steps, and then decayed linearly to zero. All runs are trained on $4$ Google Cloud TPUs with a batch size of $32$. For each dataset and algorithm, we run a hyperparameter sweep and select the best model according to accuracy on the validation set. For the GLUE tasks, we report the test metrics provided by the submission website^2. For the other classification tasks we report test-set accuracy.

We compare to fine-tuning, the current standard for transfer of large pre-trained models, and the strategy successfully used by BERT. For $N$ tasks, full fine-tuning requires $N{\times}$ the number of parameters of the pre-trained model. Our goal is to attain performance equal to fine-tuning, but with fewer total parameters, ideally near to $1{\times}$.

3.2 GLUE benchmark

\begin{tabular}{l|ll|rrrrrrrrr|r}
\toprule
 & \pbox{3cm}{Total num\\ params} & \pbox{3cm}{Trained \\ params / task} & CoLA & SST & MRPC & STS-B & QQP & MNLI\textsubscript{m} & MNLI\textsubscript{mm} & QNLI & RTE & Total \\
\midrule
BERT\textsubscript{LARGE} & $9.0\times$ & $100\%$ & $60.5$ & $94.9$ & $89.3$ & $87.6$ & $72.1$ & $86.7$ & $85.9$ & $91.1$ & $70.1$ & $80.4$ \\
Adapters ($8$-$256$) & $1.3\times$ & $3.6\%$ & $59.5$ & $94.0$ & $89.5$ & $86.9$ & $71.8$ & $84.9$ & $85.1$ & $90.7$ & $71.5$ & $80.0$ \\
Adapters ($64$) & $1.2\times$ & $2.1\%$ & $56.9$ & $94.2$ & $89.6$ & $87.3$ & $71.8$ & $85.3$ & $84.6$ & $91.4$ & $68.8$ & $79.6$ \\
\bottomrule
\end{tabular}

We first evaluate on GLUE.[^3] For these datasets, we transfer from the pre-trained BERT $_\text{LARGE}$ model, which contains $24$ layers, and a total of $330$ M parameters, see [4] for details. We perform a small hyperparameter sweep for adapter tuning: We sweep learning rates in ${3 \cdot 10^{-5}, 3 \cdot 10^{-4}, 3 \cdot 10^{-3}}$, and number of epochs in ${3, 20}$. We test both using a fixed adapter size (number of units in the bottleneck), and selecting the best size per task from ${8, 64, 256}$. The adapter size is the only adapter-specific hyperparameter that we tune. Finally, due to training instability, we re-run $5$ times with different random seeds and select the best model on the validation set.

[^3]: We omit WNLI as in [4] because the no current algorithm beats the baseline of predicting the majority class.

Table 1 summarizes the results. Adapters achieve a mean GLUE score of $80.0$, compared to $80.4$ achieved by full fine-tuning. The optimal adapter size varies per dataset. For example, $256$ is chosen for MNLI, whereas for the smallest dataset, RTE, $8$ is chosen. Restricting always to size $64$, leads to a small decrease in average accuracy to $79.6$. To solve all of the datasets in Table 1, fine-tuning requires $9\times$ the total number of BERT parameters.[^4] In contrast, adapters require only $1.3\times$ parameters.

[^4]: We treat MNLI $\text{m}$ and MNLI $\text{mm}$ as separate tasks with individually tuned hyperparameters. However, they could be combined into one model, leaving $8\times$ overall.

3.3 Additional Classification Tasks

To further validate that adapters yields compact, performant, models, we test on additional, publicly available, text classification tasks. This suite contains a diverse set of tasks: The number of training examples ranges from $900$ to $330$ k, the number of classes ranges from $2$ to $157$, and the average text length ranging from $57$ to $1.9$ k characters. Statistics and references for all of the datasets are in the appendix.

For these datasets, we use a batch size of $32$. The datasets are diverse, so we sweep a wide range of learning rates: ${1 \cdot 10^{-5}, 3 \cdot 10^{-5}, 1 \cdot 10^{-4}, 3 \cdot 10^{-3}}$. Due to the large number of datasets, we select the number of training epochs from the set ${20, 50, 100}$ manually, from inspection of the validation set learning curves. We select the optimal values for both fine-tuning and adapters; the exact values are in the appendix.

We test adapters sizes in ${2, 4, 8, 16, 32, 64}$. Since some of the datasets are small, fine-tuning the entire network may be sub-optimal. Therefore, we run an additional baseline: variable fine-tuning. For this, we fine-tune only the top $n$ layers, and freeze the remainder. We sweep $n\in{1, 2, 3, 5, 7, 9, 11, 12}$. In these experiments, we use the BERT $_\text{BASE}$ model with $12$ layers, therefore, variable fine-tuning subsumes full fine-tuning when $n=12$.

Unlike the GLUE tasks, there is no comprehensive set of state-of-the-art numbers for this suite of tasks. Therefore, to confirm that our BERT-based models are competitive, we collect our own benchmark performances. For this, we run a large-scale hyperparameter search over standard network topologies. Specifically, we run the single-task Neural AutoML algorithm, similar to [17, 18]. This algorithm searches over a space of feedforward and convolutional networks, stacked on pre-trained text embeddings modules publicly available via TensorFlow Hub^5. The embeddings coming from the TensorFlow Hub modules may be frozen or fine-tuned. The full search space is described in the appendix. For each task, we run AutoML for one week on CPUs, using $30$ machines. In this time the algorithm explores over $10$ k models on average per task. We select the best final model for each task according to validation set accuracy.

The results for the AutoML benchmark ("no BERT baseline"), fine-tuning, variable fine-tuning, and adapter-tuning are reported in Table 2. The AutoML baseline demonstrates that the BERT models are competitive. This baseline explores thousands of models, yet the BERT models perform better on average. We see similar pattern of results to GLUE. The performance of adapter-tuning is close to full fine-tuning ($0.4%$ behind). Fine-tuning requires $17\times$ the number of parameters to BERT $_\text{BASE}$ to solve all tasks. Variable fine-tuning performs slightly better than fine-tuning, whilst training fewer layers. The optimal setting of variable fine-tuning results in training $52%$ of the network on average per task, reducing the total to $9.9\times$ parameters. Adapters, however, offer a much more compact model. They introduce $1.14%$ new parameters per task, resulting in $1.19\times$ parameters for all $17$ tasks.

3.4 Parameter/Performance trade-off

The adapter size controls the parameter efficiency, smaller adapters introduce fewer parameters, at a possible cost to performance. To explore this trade-off, we consider different adapter sizes, and compare to two baselines: (i) Fine-tuning of only the top $k$ layers of BERT $_\text{BASE}$. (ii) Tuning only the layer normalization parameters. The learning rate is tuned using the range presented in Section 3.2.

Figure 3 shows the parameter/performance trade-off aggregated over all classification tasks in each suite (GLUE and "additional"). On GLUE, performance decreases dramatically when fewer layers are fine-tuned. Some of the additional tasks benefit from training fewer layers, so performance of fine-tuning decays much less. In both cases, adapters yield good performance across a range of sizes two orders of magnitude fewer than fine-tuning.

Figure 4 shows more details for two GLUE tasks: MNLI $\text{m}$ and CoLA. Tuning the top layers trains more task-specific parameters for all $k>2$. When fine-tuning using a comparable number of task-specific parameters, the performance decreases substantially compared to adapters. For instance, fine-tuning just the top layer yields approximately $9$ M trainable parameters and $77.8 % \pm 0.1 %$ validation accuracy on MNLI $\text{m}$. In contrast, adapter tuning with size $64$ yields approximately $2$ M trainable parameters and $83.7% \pm 0.1 %$ validation accuracy. For comparison, full fine-tuning attains $84.4 % \pm 0.02 %$ on MNLI $_\text{m}$. We observe a similar trend on CoLA.

As a further comparison, we tune the parameters of layer normalization alone. These layers only contain point-wise additions and multiplications, so introduce very few trainable parameters: $40$ k for BERT $_\text{BASE}$. However this strategy performs poorly: performance decreases by approximately $3.5%$ on CoLA and $4%$ on MNLI.

To summarize, adapter tuning is highly parameter-efficient, and produces a compact model with a strong performance, comparable to full fine-tuning. Training adapters with sizes $0.5-5%$ of the original model, performance is within $1%$ of the competitive published results on BERT $_\text{LARGE}$.

3.5 SQuAD Extractive Question Answering

Finally, we confirm that adapters work on tasks other than classification by running on SQuAD v1.1 ([19]). Given a question and Wikipedia paragraph, this task requires selecting the answer span to the question from the paragraph. Figure 5 displays the parameter/performance trade-off of fine-tuning and adapters on the SQuAD validation set. For fine-tuning, we sweep the number of trained layers, learning rate in ${3\cdot 10^{-5}, 5\cdot 10^{-5}, 1\cdot 10^{-4}}$, and number of epochs in ${2, 3, 5}$. For adapters, we sweep the adapter size, learning rate in ${3\cdot 10^{-5}, 1\cdot 10^{-4}, 3\cdot 10^{-4}, 1\cdot 10^{-3}}$, and number of epochs in ${3, 10, 20}$. As for classification, adapters attain performance comparable to full fine-tuning, while training many fewer parameters. Adapters of size $64$ ($2%$ parameters) attain a best F1 of $90.4$ %, while fine-tuning attains $90.7$. SQuAD performs well even with very small adapters, those of size $2$ ($0.1%$ parameters) attain an F1 of $89.9$.

3.6 Analysis and Discussion

We perform an ablation to determine which adapters are influential. For this, we remove some trained adapters and re-evaluate the model (without re-training) on the validation set. Figure 6 shows the change in performance when removing adapters from all continuous layer spans. The experiment is performed on BERT $_\text{BASE}$ with adapter size $64$ on MNLI and CoLA.

First, we observe that removing any single layer's adapters has only a small impact on performance. The elements on the heatmaps' diagonals show the performances of removing adapters from single layers, where largest performance drop is $2%$. In contrast, when all of the adapters are removed from the network, the performance drops substantially: to $37%$ on MNLI and $69%$ on CoLA – scores attained by predicting the majority class. This indicates that although each adapter has a small influence on the overall network, the overall effect is large.

Second, Figure 6 suggests that adapters on the lower layers have a smaller impact than the higher-layers. Removing the adapters from the layers $0-4$ on MNLI barely affects performance. This indicates that adapters perform well because they automatically prioritize higher layers. Indeed, focusing on the upper layers is a popular strategy in fine-tuning ([2]). One intuition is that the lower layers extract lower-level features that are shared among tasks, while the higher layers build features that are unique to different tasks. This relates to our observation that for some tasks, fine-tuning only the top layers outperforms full fine-tuning, see Table 2.

Next, we investigate the robustness of the adapter modules to the number of neurons and initialization scale. In our main experiments the weights in the adapter module were drawn from a zero-mean Gaussian with standard deviation $10^{-2}$, truncated to two standard deviations. To analyze the impact of the initialization scale on the performance, we test standard deviations in the interval $[10^{-7}, 1]$. Figure 6 summarizes the results. We observe that on both datasets, the performance of adapters is robust for standard deviations below $10^{-2}$. However, when the initialization is too large, performance degrades, more substantially on CoLA.

To investigate robustness of adapters to the number of neurons, we re-examine the experimental data from Section 3.2. We find that the quality of the model across adapter sizes is stable, and a fixed adapter size across all the tasks could be used with small detriment to performance. For each adapter size we calculate the mean validation accuracy across the eight classification tasks by selecting the optimal learning rate and number of epochs[^6]. For adapter sizes $8$, $64$, and $256$, the mean validation accuracies are $86.2%$, $85.8%$ and $85.7%$, respectively. This message is further corroborated by Figure 4 and Figure 5, which show a stable performance across a few orders of magnitude.

[^6]: We treat here MNLI $\text{m}$ and MNLI $\text{mm}$ as separate tasks. For consistency, for all datasets we use accuracy metric and exclude the regression STS-B task.

Finally, we tried a number of extensions to the adapter's architecture that did not yield a significant boost in performance. We document them here for completeness. We experimented with (i) adding a batch/layer normalization to the adapter, (ii) increasing the number of layers per adapter, (iii) different activation functions, such as tanh, (iv) inserting adapters only inside the attention layer, (v) adding adapters in parallel to the main layers, and possibly with a multiplicative interaction. In all cases we observed the resulting performance to be similar to the bottleneck proposed in Section 2.1. Therefore, due to its simplicity and strong performance, we recommend the original adapter architecture.

4. Related Work

Section Summary: The related work section reviews prior techniques for reusing pre-trained language models on new tasks, including feature extraction from embeddings such as ELMo and full-model fine-tuning of networks like BERT. It contrasts these with multi-task learning, which requires simultaneous training on all tasks, and continual-learning approaches that suffer from catastrophic forgetting or rapid growth in model size. The section also notes similar adapter-style modules developed for computer vision and concurrent work on BERT that differs in architecture and training procedure.

Pre-trained text representations

Pre-trained textual representations are widely used to improve performance on NLP tasks. These representations are trained on large corpora (usually unsupervised), and fed as features to downstream models. In deep networks, these features may also be fine-tuned on the downstream task. Brown clusters, trained on distributional information, are a classic example of pre-trained representations ([20]). [21] show that pre-trained embeddings of words outperform those trained from scratch. Since deep-learning became popular, word embeddings have been widely used, and many training strategies have arisen ([6, 22, 23]). Embeddings of longer texts, sentences and paragraphs, have also been developed ([8, 24, 25, 7]).

To encode context in these representations, features are extracted from internal representations of sequence models, such as MT systems ([26]), and BiLSTM language models, as used in ELMo ([27]). As with adapters, ELMo exploits the layers other than the top layer of a pre-trained network. However, this strategy only reads from the inner layers. In contrast, adapters write to the inner layers, re-configuring the processing of features through the entire network.

Fine-tuning

Fine-tuning an entire pre-trained model has become a popular alternative to features ([1, 2, 3]) In NLP, the upstream model is usually a neural language model ([28]). Recent state-of-the-art results on question answering ([29]) and text classification ([15]) have been attained by fine-tuning a Transformer network ([11]) with a Masked Language Model loss ([4]). Performance aside, an advantage of fine-tuning is that it does not require task-specific model design, unlike representation-based transfer. However, vanilla fine-tuning does require a new set of network weights for every new task.

Multi-task Learning

Multi-task learning (MTL) involves training on tasks simultaneously. Early work shows that sharing network parameters across tasks exploits task regularities, yielding improved performance ([30]). The authors share weights in lower layers of a network, and use specialized higher layers. Many NLP systems have exploited MTL. Some examples include: text processing systems (part of speech, chunking, named entity recognition, etc.) ([31]), multilingual models ([32]), semantic parsing ([33]), machine translation ([34]), and question answering ([35]). MTL yields a single model to solve all problems. However, unlike our adapters, MTL requires simultaneous access to the tasks during training.

Continual Learning

As an alternative to simultaneous training, continual, or lifelong, learning aims to learn from a sequence of tasks ([36]). However, when re-trained, deep networks tend to forget how to perform previous tasks; a challenge termed catastrophic forgetting ([9, 10]). Techniques have been proposed to mitigate forgetting ([37, 38]), however, unlike for adapters, the memory is imperfect. Progressive Networks avoid forgetting by instantiating a new network "column" for each task ([39]). However, the number of parameters grows linearly with the number of tasks, since adapters are very small, our models scale much more favorably.

Transfer Learning in Vision

Fine-tuning models pre-trained on ImageNet ([40]) is ubiquitous when building image recognition models ([41, 42]). This technique attains state-of-the-art performance on many vision tasks, including classification ([43]), fine-grained classifcation ([44]), segmentation ([45]), and detection ([46]). In vision, convolutional adapter modules have been studied ([5, 47, 48]). These works perform incremental learning in multiple domains by adding small convolutional layers to a ResNet ([49]) or VGG net ([50]). Adapter size is limited using $1\times 1$ convolutions, whilst the original networks typically use $3\times 3$. This yields $11%$ increase in overall model size per task. Since the kernel size cannot be further reduced other weight compression techniques must be used to attain further savings. Our bottleneck adapters can be much smaller, and still perform well.

Concurrent work explores similar ideas for BERT ([51]). The authors introduce Projected Attention Layers (PALs), small layers with a similar role to our adapters. The main differences are i) [51] use a different architecture, and ii) they perform multitask training, jointly fine-tuning BERT on all GLUE tasks. [52] perform an emprical comparison of our bottleneck Adpaters and PALs on SQuAD v2.0 ([19]).

Acknowledgments

We would like to thank Andrey Khorlin, Lucas Beyer, Noé Lutz, and Jeremiah Harmsen for useful comments and discussions.

Appendix

Section Summary: The appendix supplies supporting details for a set of extra text-classification experiments. It first lists the sizes, class counts, text lengths, and sources of seventeen datasets ranging from news articles to customer complaints. It then reports the training epochs used for each dataset under two modeling approaches, followed by the full hyper-parameter ranges explored for baseline models and the specific settings ultimately chosen by the search procedure.

: Table 3: Statistics and references for the additional text classification tasks.

Dataset	Train examples	Validation examples	Test examples	Classes	Avg text length	Reference
20 newsgroups	$15076$	$1885$	$1885$	$20$	$1903$	([53])
Crowdflower airline	$11712$	$1464$	$1464$	$3$	$104$	crowdflower.com
Crowdflower corporate messaging	$2494$	$312$	$312$	$4$	$121$	crowdflower.com
Crowdflower disasters	$8688$	$1086$	$1086$	$2$	$101$	crowdflower.com
Crowdflower economic news relevance	$6392$	$799$	$800$	$2$	$1400$	crowdflower.com
Crowdflower emotion	$32000$	$4000$	$4000$	$13$	$73$	crowdflower.com
Crowdflower global warming	$3380$	$422$	$423$	$2$	$112$	crowdflower.com
Crowdflower political audience	$4000$	$500$	$500$	$2$	$205$	crowdflower.com
Crowdflower political bias	$4000$	$500$	$500$	$2$	$205$	crowdflower.com
Crowdflower political message	$4000$	$500$	$500$	$9$	$205$	crowdflower.com
Crowdflower primary emotions	$2019$	$252$	$253$	$18$	$87$	crowdflower.com
Crowdflower progressive opinion	$927$	$116$	$116$	$3$	$102$	crowdflower.com
Crowdflower progressive stance	$927$	$116$	$116$	$4$	$102$	crowdflower.com
Crowdflower US economic performance	$3961$	$495$	$496$	$2$	$305$	crowdflower.com
Customer complaint database	$146667$	$18333$	$18334$	$157$	$1046$	catalog.data.gov
News aggregator dataset	$338349$	$42294$	$42294$	$4$	$57$	([54])
SMS spam collection	$4459$	$557$	$558$	$2$	$81$	([55])

: Table 4: Number of training epochs selected for the additional classification tasks.

Dataset	Epochs (Fine-tune)	Epochs (Adapters)
20 newsgroups	$50$	$50$
Crowdflower airline	$50$	$20$
Crowdflower corporate messaging	$100$	$50$
Crowdflower disasters	$50$	$50$
Crowdflower economic news relevance	$20$	$20$
Crowdflower emotion	$20$	$20$
Crowdflower global warming	$100$	$50$
Crowdflower political audience	$50$	$20$
Crowdflower political bias	$50$	$50$
Crowdflower political message	$50$	$50$
Crowdflower primary emotions	$100$	$100$
Crowdflower progressive opinion	$100$	$100$
Crowdflower progressive stance	$100$	$100$
Crowdflower US economic performance	$100$	$20$
Customer complaint database	$20$	$20$
News aggregator dataset	$20$	$20$
SMS spam collection	$50$	$20$

: Table 5: The search space of baseline models for the additional text classification tasks.

Parameter	Search Space
1) Input embedding modules	Refer to Table 6
2) Fine-tune input embedding module	True, False
3) Lowercase text	True, False
4) Remove non alphanumeric text	True, False
5) Use convolution	True, False
6) Convolution activation	relu, relu6, leaky relu, swish, sigmoid, tanh
7) Convolution batch norm	True, False
8) Convolution max ngram length	2, 3
9) Convolution dropout rate	[0.0, 0.4]
10) Convolution number of filters	[50, 200]
11) Convolution embedding dropout rate	[0.0, 0.4]
12) Number of hidden layers	0, 1, 2, 3, 5
13) Hidden layers size	64, 128, 256
14) Hidden layers activation	relu, relu6, leaky relu, swish, sigmoid, tanh
15) Hidden layers normalization	none, batch norm, layer norm
16) Hidden layers dropout rate	0.0, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5
17) Deep tower learning rate	0.001, 0.005, 0.01, 0.05, 0.1, 0.5
18) Deep tower regularization weight	0.0, 0.0001, 0.001, 0.01
19) Wide tower learning rate	0.001, 0.005, 0.01, 0.05, 0.1, 0.5
20) Wide tower regularization weight	0.0, 0.0001, 0.001, 0.01
21) Number of training samples	1e5, 2e5, 5e5, 1e6, 2e6

: Table 6: Options for text input embedding modules. These are pre-trained text embedding tables. We provide the handle for the modules that are publicly distributed via the TensorFlow Hub service (https://www.tensorflow.org/hub).

ID	Dataset	Embed	Vocab.	Training	TensorFlow Hub Handles
	size	dim.	size	algorithm	Prefix: https://tfhub.dev/google/
	(tokens)
English-small	7B	50	982k	Lang. model	nnlm-en-dim50-with-normalization/1
English-big	200B	128	999k	Lang. model	nnlm-en-dim128-with-normalization/1
English-wiki-small	4B	250	1M	Skipgram	Wiki-words-250-with-normalization/1
English-wiki-big	4B	500	1M	Skipgram	Wiki-words-500-with-normalization/1
Universal-sentence-encoder	-	512	-	([56])	universal-sentence-encoder/2

: Table 7: Search space parameters (see Table 5) for the AutoML baseline models that were selected.

Dataset	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21
20 newsgroups	Universal-sentence-encoder	False	True	True	False	relu6	False	2	0.37	94	0.38	1	128	leaky relu	batch norm	0.5	0.5	0	0.05	0.0001	1000000
Crowdflower airline	English-big	False	False	False	True	leaky relu	False	3	0.36	200	0.07	0	128	tanh	layer norm	0.4	0.1	0.001	0.05	0.001	200000
Crowdflower corporate messaging	English-big	False	False	True	True	tanh	True	3	0.40	56	0.40	1	64	tanh	batch norm	0.5	0.5	0.001	0.01	0	200000
Crowdflower disasters	Universal-sentence-encoder	True	True	False	True	swish	True	3	0.27	52	0.22	0	64	relu	none	0.2	0.005	0.0001	0.005	0.01	500000
Crowdflower economic news relevance	Universal-sentence-encoder	True	True	False	False	leaky relu	False	2	0.27	63	0.04	3	128	swish	layer norm	0.2	0.01	0.01	0.001	0	100000
Crowdflower emotion	Universal-sentence-encoder	False	True	False	False	relu6	False	3	0.35	132	0.34	1	64	tanh	none	0.05	0.05	0	0.05	0	200000
Crowdflower global warming	Universal-sentence-encoder	False	True	True	False	swish	False	3	0.39	200	0.36	1	128	leaky relu	batch norm	0.4	0.05	0	0.001	0.001	1000000
Crowdflower political audience	English-small	True	False	True	True	relu	False	3	0.11	98	0.07	0	64	relu	none	0.5	0.05	0.001	0.001	0	100000
Crowdflower political bias	English-big	False	True	True	False	swish	False	3	0.12	81	0.30	0	64	relu6	none	0	0.01	0	0.005	0.01	200000
Crowdflower political message	Universal-sentence-encoder	False	False	True	False	swish	True	2	0.36	57	0.35	0	64	tanh	none	0.5	0.01	0.001	0.005	0	200000
Crowdflower primary emotions	English-big	False	True	True	True	swish	False	3	0.40	191	0.03	0	256	relu6	none	0.5	0.1	0.001	0.05	0	200000
Crowdflower progressive opinion	English-big	True	False	True	True	relu6	False	3	0.40	199	0.28	0	128	relu	batch norm	0.3	0.1	0.01	0.005	0.001	200000
Crowdflower progressive stance	Universal-sentence-encoder	True	False	True	False	relu	True	3	0.01	195	0.00	2	256	tanh	layer norm	0.4	0.005	0	0.005	0.0001	500000
Crowdflower us economic performance	English-big	True	False	True	True	tanh	True	2	0.31	53	0.24	1	256	leaky relu	batch norm	0.3	0.05	0.0001	0.001	0.0001	100000
Customer complaint database	English-big	True	False	False	False	tanh	False	2	0.03	69	0.10	1	256	leaky relu	layer norm	0.1	0.05	0.0001	0.05	0.001	1000000
News aggregator dataset	Universal-sentence-encoder	False	True	True	False	sigmoid	True	2	0.00	156	0.29	3	256	relu	batch norm	0.05	0.05	0	0.5	0.0001	1000000
Sms spam collection	English-wiki-small	True	True	True	True	leaky relu	False	3	0.20	54	0.00	1	128	leaky relu	batch norm	0	0.1	0	0.05	0.01	1000000

A. Learning Rate Robustness

We test the robustness of adapters and fine-tuning to the learning rate. We ran experiments with learning rates in the range $[2\cdot 10^{-5}, 10^{-3}]$, and selected the best hyperparameters for each method at each learning rate. Figure 7 shows the results.

References

Section Summary: This section consists of a numbered bibliography listing dozens of academic papers primarily from machine learning and natural language processing conferences and journals. The cited works address topics such as language model pre-training, word and sentence embeddings, transfer learning, attention mechanisms, and challenges like catastrophic forgetting, with many references from venues including NIPS, ACL, ICLR, and EMNLP between 2003 and 2019. These sources appear to support research on models like BERT and related benchmarks.

[1] Dai, A. M. and Le, Q. V. Semi-supervised sequence learning. In NIPS. 2015.

[2] Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. In ACL, 2018.

[3] Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/language-unsupervised/language_ understanding_paper. pdf, 2018.

[4] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

[5] Rebuffi, S.-A., Bilen, H., and Vedaldi, A. Learning multiple visual domains with residual adapters. In NIPS. 2017.

[6] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. In NIPS. 2013.

[7] Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Strope, B., and Kurzweil, R. Universal sentence encoder for english. In EMNLP, 2019.

[8] Le, Q. and Mikolov, T. Distributed representations of sentences and documents. In ICML, 2014.

[9] McCloskey, M. and Cohen, N. J. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. 1989.

[10] French, R. M. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1999.

[11] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In NIPS. 2017.

[12] De Vries, H., Strub, F., Mary, J., Larochelle, H., Pietquin, O., and Courville, A. C. Modulating early visual processing by language. In NIPS, 2017.

[13] Perez, E., Strub, F., de Vries, H., Dumoulin, V., and Courville, A. C. Film: Visual reasoning with a general conditioning layer. AAAI, 2018.

[14] Chen, T., Lucic, M., Houlsby, N., and Gelly, S. On self modulation for generative adversarial networks. ICLR, 2019.

[15] Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analysis platform for natural language understanding. ICLR, 2018.

[16] Kingma, D. and Ba, J. Adam: A method for stochastic optimization. ICLR, 2014.

[17] Zoph, B. and Le, Q. V. Neural architecture search with reinforcement learning. In ICLR, 2017.

[18] Wong, C., Houlsby, N., Lu, Y., and Gesmundo, A. Transfer learning with neural automl. In NeurIPS. 2018.

[19] Rajpurkar, P., Jia, R., and Liang, P. Know what you don't know: Unanswerable questions for squad. In ACL, 2018.

[20] Brown, P. F., deSouza, P. V., Mercer, R. L., Pietra, V. J. D., and Lai, J. C. Class-based n-gram models of natural language. Computational Linguistics, 1992.

[21] Turian, J., Ratinov, L., and Bengio, Y. Word representations: A simple and general method for semi-supervised learning. In ACL, 2010.

[22] Pennington, J., Socher, R., and Manning, C. Glove: Global vectors for word representation. In EMNLP, 2014.

[23] Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. Enriching word vectors with subword information. ACL, 2017.

[24] Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. Skip-thought vectors. In NIPS. 2015.

[25] Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. Supervised learning of universal sentence representations from natural language inference data. In EMNLP, 2017.

[26] McCann, B., Bradbury, J., Xiong, C., and Socher, R. Learned in translation: Contextualized word vectors. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), NIPS. 2017.

[27] Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In NAACL, 2018.

[28] Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. A neural probabilistic language model. Journal of Machine Learning Research, 2003.

[29] Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.

[30] Caruana, R. Multitask learning. Machine Learning, 1997.

[31] Collobert, R. and Weston, J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, 2008.

[32] Huang, J.-T., Li, J., Yu, D., Deng, L., and Gong, Y. Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In ICASSP, 2013.

[33] Peng, H., Thomson, S., and Smith, N. A. Deep multitask learning for semantic dependency parsing. In ACL, 2017.

[34] Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., Hughes, M., and Dean, J. Google's multilingual neural machine translation system: Enabling zero-shot translation. ACL, 2017.

[35] Choi, E., Hewlett, D., Uszkoreit, J., Polosukhin, I., Lacoste, A., and Berant, J. Coarse-to-fine question answering for long documents. In ACL, 2017.

[36] Thrun, S. Learning to learn. chapter Lifelong Learning Algorithms. 1998.

[37] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.

[38] Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. ICML, 2017.

[39] Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.

[40] Deng, J., Dong, W., Socher, R., jia Li, L., Li, K., and Fei-fei, L. Imagenet: A large-scale hierarchical image database. In In CVPR, 2009.

[41] Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks? In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D., and Weinberger, K. Q. (eds.), NIPS. 2014.

[42] Huh, M., Agrawal, P., and Efros, A. A. What makes imagenet good for transfer learning? arXiv preprint arXiv:1608.08614, 2016.

[43] Kornblith, S., Shlens, J., and Le, Q. V. Do better imagenet models transfer better? arXiv preprint arXiv:1805.08974, 2018.

[44] Hermans, A., Beyer, L., and Leibe, B. In Defense of the Triplet Loss for Person Re-Identification. arXiv preprint arXiv:1703.07737, 2017.

[45] Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In CVPR, 2015.

[46] Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.

[47] Rebuffi, S., Vedaldi, A., and Bilen, H. Efficient parametrization of multi-domain deep neural networks. In CVPR, 2018.

[48] Rosenfeld, A. and Tsotsos, J. K. Incremental learning through deep adaptation. IEEE transactions on pattern analysis and machine intelligence, 2018.

[49] He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.

[50] Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. ICLR, 2014.

[51] Stickland, A. C. and Murray, I. BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning. arXiv preprint arXiv:1902.02671, 2019.

[52] Sina Semnani, Kaushik Sadagopan, F. T. BERT-A: Fine-tuning BERT with Adapters and Data Augmentation. http://web.stanford.edu/class/cs224n/reports/default/15848417.pdf, 2019.

[53] Lang, K. Newsweeder: Learning to filter netnews. In ICML, 1995.

[54] Lichman, M. UCI machine learning repository, 2013.

[55] Almeida, T. A., Hidalgo, J. M. G., and Yamakami, A. Contributions to the Study of SMS Spam Filtering: New Collection and Results. In Proceedings of the 11th ACM Symposium on Document Engineering. ACM, 2011.

[56] Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. Universal sentence encoder. arXiv preprint arXiv:1803.11175, 2018.