BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Elad Ben-Zaken$^1$ Shauli Ravfogel$^{1,2}$ Yoav Goldberg$^{1,2}$
$^1$ Computer Science Department, Bar Ilan University
$^2$ Allen Institute for Artificial Intelligence
{benzakenelad, shauli.ravfogel, yoav.goldberg}@gmail.com

Abstract

We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

Executive Summary: BitFit is a fine-tuning method for large pre-trained language models such as BERT that updates only the bias terms instead of all parameters. The work addresses the high cost of deploying separate full models for each downstream task and the open question of whether fine-tuning mainly learns new knowledge or mainly exposes capabilities already present after pre-training.

The authors set out to test whether modifying a tiny, fixed subset of parameters—the bias terms—could match the accuracy of updating the entire network on standard NLP benchmarks. They ran controlled experiments on the GLUE suite with BERT_BASE, BERT_LARGE, and RoBERTa_BASE, plus checks on SQuAD and part-of-speech tagging. Results were compared against full fine-tuning and against two leading sparse-adaptation methods (Adapters and Diff-Pruning). Training-set size was varied systematically to map performance across data regimes.

BitFit reaches or exceeds full fine-tuning accuracy on most GLUE tasks while changing only 0.09 % of the parameters; a further-reduced variant that touches just two bias groups (query and middle MLP) uses 0.04 % and loses only about one point on average. The method outperforms or matches the prior sparse baselines while satisfying the additional constraint that the same parameter locations are updated for every task. Gains are largest in the low- and medium-data regimes; the advantage reverses once training data becomes abundant. Randomly chosen parameters of equal count produce markedly worse results, indicating that bias terms are not interchangeable with other weights. The generalization gap between train and test error is also smaller than under full fine-tuning.

These outcomes matter for organizations that must serve many tasks under tight memory or latency budgets. A single shared base model plus small per-task bias vectors can replace dozens of complete copies, lowering storage, simplifying version control, and enabling hardware designs in which most weights remain fixed. The findings also lend support to the view that fine-tuning primarily surfaces knowledge already induced by language-model pre-training rather than injecting large amounts of task-specific linguistic information.

For immediate use, BitFit is recommended whenever labeled data per task is modest and memory efficiency is a priority. When data volumes are large, standard full fine-tuning or hybrid schedules should be retained. Further work is needed to confirm behavior on additional model families, token-level structured-prediction tasks, and production-scale data volumes before broad policy or hardware commitments are made.

1. Introduction

Section Summary: Large pre-trained language models such as BERT deliver strong results on many language tasks but are costly to train and deploy because of their size. Researchers have therefore explored methods that adapt these models to new tasks by updating only a tiny subset of parameters while leaving the rest fixed. The authors find that adjusting just the bias terms—sometimes only two specific bias components—matches or exceeds the accuracy of full fine-tuning while changing as little as 0.04 percent of all parameters, which simplifies multi-task deployment in memory-limited settings.

Large pre-trained transformer based language models, and in particular bidirectional masked language models from the BERT family [1, 2, 3], are responsible for significant gains in many NLP tasks. Under the common paradigm, the model is pre-trained on large, annotated corpora with the LM objective, and then finetuned on task-specific supervised data. The large size of these models make them expensive to train and, more importantly, expensive to deploy. This, along with theoretical questions on the extent to which finetuning must change the original model, has led researchers to consider fine-tuning variants where one identifies a small subset of the model parameters which need to be changed for good performance in end-tasks, while keeping all others intact (§Section 2).

We present a simple and effective approach to fine tuning (§ 3), which has the following benefits:

1. Changing very few parameters per fine-tuned task.
2. Changing the same set of parameters for every tasks (task-invariance).
3. The changed parameters are both isolated and localized across the entire parameter space.
4. For small to medium training data, changing only these parameters reaches the same task accuracy as full fine-tuning, and sometimes even improves results.

Specifically, we show that freezing most of the network and fine-tuning only the bias-terms is surprisingly effective. Moreover, if we allow the tasks to suffer a small degradation in performance, we can fine-tune only two bias components (the "query" and "middle-of-MLP" bias terms), amounting to half of the bias parameters in the model, and only 0.04% of all model parameters.

This result has a large practical utility in deploying multi-task fine-tuned models in memory-constrained environments, as well as opens the way to trainable hardware implementations in which most of the parameters are fixed. Additionally, it opens up a set of research directions regarding the role of bias terms in pre-trained networks, and the dynamics of the fine-tuning process.

2. Background: fine-tuning and parameter-efficient fine-tuning

Section Summary: Fine-tuning adapts a pre-trained model to a new task by adding a task-specific layer on top and training the full network end-to-end, but this produces a large unique model per task that is costly to store and deploy. To address this, recent methods aim to match full fine-tuning performance while updating only a tiny fraction of parameters, supporting sequential task arrivals and hardware-friendly consistency in which weights change. Two such approaches include inserting small adapter modules between layers or adding sparse task-specific difference vectors, both of which largely keep the original parameters fixed and shared.

In transfer-learning via model fine-tuning, a pre-trained encoder network takes the input and produces contextualized representations. Then, a task-specific classification layer (here we consider linear classifiers) is added on top of the encoder, and the entire network (encoder+task specific classifiers) is trained end-to-end to minimize the task loss.

Desired properties. While fine-tuning per-task is very effective, it also results in a unique, large model for each pre-trained task, making it hard to reason about what was changed in the fine-tuning process, as well as hard to deploy, especially as the number of tasks increases. Ideally, one would want a fine-tuning method that:

(i) matches the results of a fully fine-tuned model;

(ii) changes only a small portion of the model's parameters; and (iii) enables tasks to arrive in a stream, instead of requiring simultaneous access to all datasets. For efficient hardware based deployments, it is further preferred that (iv): the set of parameters that change values is consistent across different tasks.

Learning vs. Exposing. The feasibility of fulfilling the above requirements depends on a fundamental question regarding the nature of the fine-tuning process of large pre-trained LMs: to what extent does the fine-tuning process induces the learning of new capabilities, vs. the exposing of existing capabilities, which were learned during the pre-training process.

Existing approaches. Two recent works have demonstrated that adaptation to various end-tasks can in fact be achieved by changing only a small subset of parameters.

The first work, by [4] ("Adapters"), achieves this goal by injecting small, trainable task-specific "adapter" modules between the layers of the pre-trained model, where the original parameters are shared between tasks.

The second work, by [5] ("Diff-Pruning"), achieves the same goal by adding a sparse, task-specific difference-vector to the original parameters, which remain fixed and are shared between tasks. The difference-vector is regularized to be sparse.

Both methods allow adding only a small number of trainable parameters per-task (criteria ii), and each task can be added without revisiting previous ones (criteria iii).

They also partially fulfill criteria (i), suffering only a small drop in performance compared to full fine-tuning.

The Adapter method, but not the Diff-Pruning method, also supports criteria (iv). However, Diff-Pruning is more parameter efficient than the Adapter method (in particular, it adds no new parameters), and also achieves better task scores.

We compare against Diff-Pruning and Adapters in the experiments section, and show that we perform favorably on many tasks while also satisfying criteria (iv).

3. Bias-terms Fine-tuning (BitFit)

Section Summary: BitFit is a lightweight fine-tuning technique that freezes nearly all parameters in a transformer model such as BERT and updates only the bias vectors plus the final task-specific classification layer. The resulting updates match or approach the accuracy of full-model fine-tuning yet require storing fewer than 0.1 percent of the original parameters per task, allowing models to be adapted sequentially as new datasets arrive. Even restricting training to the bias terms of the query projections and the second MLP layer is often sufficient to retain strong performance.

We propose a method we call BitFit[^1] (BIas-Term FIne-Tuning), in which we freeze most of the transformer-encoder parameters, and train only the bias-terms and the task-specific classification layer. BitFit has three key properties: (i) match the results of fully fine-tuned model. (ii) enable tasks to arrive in a stream, this way it does not require simultaneous access to all datasets. (iii) fine-tune only a small portion of the model's parameters.

[^1]: Our code is publicly available at www.github.com/benzakenelad/BitFit

The approach is parameter-efficient: each new task requires storing only the bias terms parameter vectors (which amount to less than 0.1% of the total number of parameters), and the task-specific final linear classifier layer.

Concretely, the BERT encoder is composed of $L$ layers, where each layer $\ell$ starts with $M$ self-attention heads, where a self attention head $(m, \ell)$ has key, query and value encoders, each taking the form of a linear layer:

$ \begin{aligned} \mathbf{Q}^{m, \ell}(\mathbf{x}) &= \mathbf{W}{q}^{m, \ell}\mathbf{x} + \mathbf{b}{q}^{m, \ell} \ \mathbf{K}^{m, \ell}(\mathbf{x}) &= \mathbf{W}{k}^{m, \ell}\mathbf{x} + \mathbf{b}{k}^{m, \ell} \ \mathbf{V}^{m, \ell}(\mathbf{x}) &= \mathbf{W}{v}^{m, \ell}\mathbf{x} + \mathbf{b}{v}^{m, \ell} \end{aligned} $

Where $\mathbf{x}$ is the output of the former encoder layer (for the first encoder layer $\mathbf{x}$ is the output of the embedding layer). These are then combined using an attention mechanism that does not involve new parameters:

$ \mathbf{h}_1^{\ell} = att \big(\mathbf{Q}^{1, \ell}, \mathbf{K}^{1, \ell}, \mathbf{V}^{1, \ell}, .., \mathbf{Q}^{m, \ell}, \mathbf{K}^{m, \ell}, \mathbf{V}^{m, l}\big) $

and then fed to an MLP with layer-norm (LN):

$ \begin{aligned} \mathbf{h}{2}^{\ell} &= \text{Dropout}\big(\mathbf{W}{m_1}^{\ell} \cdot \mathbf{h}{1}^{\ell};;+;;\mathbf{b}{m_1}^{\ell} \big)\ \mathbf{h}{3}^{\ell} &= \mathbf{g}{LN_1}^{\ell}\odot\frac{(\mathbf{h}{2}^{\ell} + \mathbf{x}) - \mu}{\sigma} + \mathbf{b}{LN_1}^{\ell}\ \mathbf{h}{4}^{\ell} &= ;;\text{GELU}\big(\mathbf{W}{m_2}^{\ell} \cdot \mathbf{h}{3}^{\ell} ;;;+;;\mathbf{b}{m_2}^{\ell}\big)\ \mathbf{h}{5}^{\ell} &= \text{Dropout}\big(\mathbf{W}{m_3}^{\ell} \cdot \mathbf{h}{4}^{\ell};;+;;;\mathbf{b}{m_3}^{\ell}\big)\ \mathbf{;;;\text{out}}^{\ell} &= \mathbf{g}{LN_2}^{\ell}\odot\frac{(\mathbf{h}{5}^{\ell} + \text{h}{3}^{\ell}) - \mu}{\sigma} + \mathbf{b}{LN_2}^{\ell} \end{aligned} $

The collection of all matrices $\mathbf{W}{(\cdot)}^{\ell, (\cdot)}$ and vectors $\mathbf{g}{(\cdot)}^{\ell}$, $\mathbf{b}{(\cdot)}^{\ell, (\cdot)}$, indicated in blue and purple are the network's parameters $\Theta$, where the subset of purple vectors $\mathbf{b}{(\cdot)}^{\ell, (\cdot)}$ are the bias terms[^2].

[^2]: In Appendix § A.1 we relate this notation with parameter names in HuggingFace implementation.

::: {caption="Table 1: BERT _LARGE model performance on the GLUE benchmark validation set (V) and test set (T). Lines with † and ‡ indicate results taken from [5] and [4] (respectively)."}

:::

The bias terms are additive, and correspond to a very small fraction of the network, in BERT $\text{BASE}$ and BERT $\text{LARGE}$ bias parameters make up 0.09% and 0.08% of the total number of parameters in each model, respectively.

We show that by freezing all the parameters $\mathbf{W}^{(\cdot)}$ and $\mathbf{g}^{(\cdot)}$ and fine-tuning only the additive bias terms $\mathbf{b}^{(\cdot)}$, we achieve transfer learning performance which is comparable (and sometimes better!) than fine-tuning of the entire network,

We also show that we can fine-tune only a subset of the bias parameters, namely those associated with the query and the second MLP layer (only $\mathbf{b}{q}^{(\cdot)}$ and $\mathbf{b}{m_2}^{(\cdot)}$), and still achieve accuracies that rival full-model fine-tuning.

4. Experiments and Results

Section Summary: BitFit was tested on the GLUE benchmark using BERT and RoBERTa models, where it matched or exceeded the performance of full fine-tuning, diff-pruning, and adapter methods on most tasks while updating only about 0.1% of the parameters. Experiments showed that certain bias terms, especially those in the query projections and feed-forward layers, drive most of the gains, and randomly chosen parameters of similar size performed noticeably worse. Results held across model sizes, with BitFit also producing a smaller gap between training and test performance.

Datasets. We evaluate BitFit on the GLUE[^3] benchmark [6]. Consistent with previous work [4, 5] we exclude the WNLI task, on which BERT models do not outperform the majority baseline.

[^3]: Appendix § A.3 lists the tasks and evaluation metrics.

Models and Optimization. We use the publicly available pre-trained BERT $\text{BASE}$, BERT $\text{LARGE}$ [1] and RoBERTa $_\text{BASE}$ [2] models, using the HuggingFace [7] interface and implementation. Appendix § A.2 lists optimization details.

Comparison to Diff-Pruning and Adapters (Table 1) In the first experiment, we compare BitFit to Diff-Pruning method and Adapters method, when using a fewer number of parameters. Table 1 reports the dev-set and test-set performance compared to the Diff-Pruning and Adapters numbers reported by [5] and [4] (respectively). This experiment used the BERT $_\text{LARGE}$ model.

On validation set, BitFit outperforms Diff-Pruning on 4 out of 9 tasks, while using 6x fewer trainable parameters[^4]. As for test-set results, two clear wins compared to Diff-Pruning and 4 clear wins compared to Adapters while using 45x fewer trainable parameters.

[^4]: QNLI results are not directly comparable, as the GLUE benchmark updated the test set since then.

Different Base-models (Table 2) We repeat the BERT $\text{LARGE}$ results on different base-models (the smaller BERT $\text{BASE}$ and the better performing RoBERTa $_\text{BASE}$). The results in Table 2 show that the trends remain consistent.

::: caption="Table 2: Dev-set results for different base models. BB: BERT _BASE. BL: BERT _LARGE. Ro: RoBERTa _BASE."

:::

\begin{tabular}{lccccccccccc}
\hline \textbf{} & \textbf{\% Param} & \textbf{QNLI} & \textbf{SST-2} & \textbf{MNLI\textsubscript{m}} & \textbf{MNLI\textsubscript{mm}} & \textbf{CoLA} & \textbf{MRPC} & \textbf{STS-B} & \textbf{RTE} & \textbf{QQP} & \textbf{Avg.}\\ \hline
 Full-FT & 100\% & 90.7 $\pm$ 0.2 & 92.0 $\pm$ 0.4 & 83.5 $\pm$ 0.1 & 83.7 $\pm$ 0.3 & 56.4 $\pm$ 0.9 & 89.0 $\pm$ 1.0 & 88.9 $\pm$ 0.7 & 70.5 $\pm$ 0.6 & 87.1 $\pm$ 0.1  & 82.3\\
BitFit & 0.09\% & 90.2 $\pm$ 0.2 & 92.1 $\pm$ 0.3 & 81.4 $\pm$ 0.2 & 82.2 $\pm$ 0.2 & 58.8 $\pm$ 0.5 & 90.4 $\pm$ 0.5 & 89.2 $\pm$ 0.2 & 72.3 $\pm$ 0.9 & 84.0 $\pm$ 0.2 & \textbf{82.4}\\
$\mathbf{b}_{m2}, \mathbf{b}_q$ & 0.04\% & 89.4 $\pm$ 0.1 & 91.2 $\pm$ 0.2 & 80.4 $\pm$ 0.2 & 81.5 $\pm$ 0.2 & 57.4 $\pm$ 0.8 & 89.0 $\pm$ 0.2 & 88.4 $\pm$ 0.1 & 68.6 $\pm$ 0.6 & 83.7 $\pm$ 0.2 & 81.1\\
$\mathbf{b}_{m2}$ & 0.03\% & 88.9 $\pm$ 0.1 & 91.1 $\pm$ 0.3 & 79.9 $\pm$ 0.3 & 80.7 $\pm$ 0.2 & 54.9 $\pm$ 0.9 & 87.9 $\pm$ 0.6 & 88.2 $\pm$ 0.1 & 66.8 $\pm$ 0.6 & 82.1 $\pm$ 0.4 & 80.0 \\
$\mathbf{b}_{q}$ & 0.01\% & 86.8 $\pm$ 0.1 & 89.6 $\pm$ 0.2 & 74.4 $\pm$ 0.3 & 75.7 $\pm$ 0.2 & 49.1 $\pm$ 1.5 & 84.4 $\pm$ 0.2 & 85.6 $\pm$ 0.1 & 61.4 $\pm$ 1.1 & 80.6 $\pm$ 0.4 & 76.6 \\
Frozen & 0.0\% & 68.7 $\pm$ 0.3 & 81.7 $\pm$ 0.1 & 42.4 $\pm$ 0.1 & 43.8 $\pm$ 0.1 & 31.9 $\pm$ 1.1 & 81.1 $\pm$ 0.1 & 71.4 $\pm$ 0.1 & 56.9 $\pm$ 0.4 & 62.4 $\pm$ 0.2 & 62.1 \\
rand uniform & 0.09\% & 87.8 $\pm$ 0.3 & 90.5 $\pm$ 0.3 & 78.3 $\pm$ 0.3 & 78.8 $\pm$ 0.2 & 54.1 $\pm$ 1.0 & 84.3 $\pm$ 0.3 & 87.2 $\pm$ 0.4 & 62.9 $\pm$ 0.9 & 82.4 $\pm$ 0.3 & 78.5 \\
rand row/col & 0.09\% & 88.4 $\pm$ 0.2 & 91.0 $\pm$ 0.3 & 79.4 $\pm$ 0.3 & 80.1 $\pm$ 0.3 & 53.4 $\pm$ 0.6 & 88.0 $\pm$ 0.7 & 87.9 $\pm$ 0.2 & 65.1 $\pm$ 0.7 & 82.3 $\pm$ 0.2 & 79.5 \\
\hline

\end{tabular}

Are bias parameters special?

Are the bias parameters special, or will any random subset do? We randomly sampled the same amount of parameters as in BitFit from the entire model, and fine-tuned only them ("rand uniform" line in Table 3). The results are substantially worse across all tasks; similar patterns are observed when the random parameters are sampled as complete rows/columns in the parameter matrices ("rand row/col" line in Table 3).

Fewer bias parameters (Table 3)

Can we fine-tune on only a subset of the bias-parameter?

We define the amount of change in a bias vector $\mathbf{b}$ to be $\frac{1}{\mathbf{dim(b)}}\left\lVert\mathbf{b}_0 - \mathbf{b}F\right\rVert_1$, that is, the average absolute change, across its dimensions, between the initial LM values $\mathbf{b}0$ and its fine-tuned values $\mathbf{b}F$. Figure 1 shows the change per bias term and layer, for the RTE task (other tasks look very similar, see Appendix § A.4). The 'key' bias $\textbf{b}{k}$ has zero change, consistent with the theoretical observation in [8]. In contrast, $\textbf{b}q$, the bias of the queries, and $\textbf{b}{m2}$, the bias of the intermediate MLP layers (which take the input from 768-dims to 3072), change the most. Table 3 reports dev-set results when fine-tuning only the $\textbf{b}q^{(\cdot)}$ and $\textbf{b}{m2}^{(\cdot)}$ bias terms, for the BERT $\text{BASE}$ model. Results are only marginally lower than when tuning all bias parameters. Tuning either $\textbf{b}q^{(\cdot)}$ or $\textbf{b}{m2}^{(\cdot)}$ alone yields substantially worse results, indicating both bias types are essential. As expected, using a frozen BERT $\text{BASE}$ model yields much worse results.

Generalization gap.

While in most cases full fine-tuning reaches nearly 100% train accuracy, we find that the generalization gap [9]—the difference between training error and test error—is substantially smaller for the BitFit models.

Token-level tasks.

The GLUE tasks are all sentence level. We also experimented with token-level PTB POS-tagging. Full-FT results for BERT $\text{BASE}$, BERT $\text{LARGE}$ and RoBERTa $_\text{BASE}$ are 97.2, 97.4, 97.2, while BitFit results are 97.2, 97.4, 97.1.

Size of training data.

The GLUE results suggest a reverse correlation between BitFit ability to reach Full-FT performance, and training set size. To test this (and to validate another token-level task), we train on increasing-sized subsets of SQuAD v1.0 [10]. The results on Figure 2 show a clear trend: BitFit dominates over Full-FT in the smaller-data regime, while the trend is reversed when more training data is available. We conclude that BitFit is a worthwhile targetted fine-tuning method in small-to-medium data regimes.

5. Related Work

Section Summary: This section explores research on fine-tuning large language models by updating only a small subset of parameters, linking the approach to both practical model compression and deeper questions about what knowledge pre-training captures. It reviews work on over-parameterization, showing that models often contain far more parameters than needed and that sparse sub-networks can transfer effectively to new tasks, consistent with ideas like the lottery-ticket hypothesis. It also highlights the under-studied role of bias terms, noting that while they are usually ignored, recent evidence indicates that tuning biases alone can successfully adapt pre-trained models in both language and vision settings.

The problem of identifying the minimal set of parameters that need to be fine-tuned to achieve good performance in end-tasks relates both to practical questions of model compression, and also to more fundamental question on the nature of the pre-training and finetuning process, the "linguistic knowledge" induced by each of them, and the extent to which it generalizes to different tasks.

Over-parameterization. Large LM models were shown to be over-parameterized: they contain more parameters than needed in inference ([11, 12, 13, 14, 15, 16, 17, 18, 19]). [20] have demonstrated that overparmeterization can be exploited in finetuning: pruned network perform well in transfer setting. We work in a complementary setting, where the entire model is kept, but only some parameters are updated. The remarkable success of those works have sparked interest the lottery-ticket hypothesis ([21, 22, 23]): the conjecture that large models are needed in pretraining only to induce (in high probability) the existing of sub-networks initialized with the correct inductive bias for learning, and the findings that those sparse networks often transfer well to different tasks.

Bias terms. Bias terms and their importance are rarely discussed in the literature[^5]. [24] describe a masking-based fine-tuning method, and explicitly mention ignoring the bias terms, as handling them "did not observe a positive effect on performance".

in dent An exception is the work of [25] who analyzed bias terms from the perspective of attribution method. They demonstrate that the last layer bias values are responsible for the predicted class, and propose a way to back-propagate their importance. [26] finetuned the biases of the output softmax in an NMT systems, to personalize the output vocabulary, and [27] have demonstrated that randomly-initialized CNNs achieve reasonable accuracy after training the batch-norm layers alone. Finally, and closest to our work, [28] demonstrate that bias-only fine-tuning similar to ours is effective also for adaptation of pre-trained computer vision models. Our work empirically shows the importance and power of the bias parameters to substantially change the networks' behavior, calling for further analysis and attention on the bias terms.

[^5]: Indeed, the equations in the paper introducing the Transformer model [29] do not include bias terms at all, and their existence in the BERT models might as well be a fortunate mistake.

6. Conclusions

Section Summary: BitFit is a method for adapting large pre-trained transformer models to new tasks by changing only the bias parameters rather than the full set of weights. This focused adjustment delivers competitive results on standard language-understanding benchmarks, simplifies sharing a single model across many tasks, and supports hardware designs that keep most computations fixed. Its effectiveness also raises broader questions about how fine-tuning actually shifts model behavior and what role bias terms play in that process.

We propose BitFit, a novel method for localized, fast fine-tuning of pre-trained transformers for end-tasks. The method focuses the finetuning on a specific fraction of the model parameters—the biases—and maintains good performance in all GLUE tasks we evaluated on. The focus on modifying a small group of parameters eases deployment, as the vast majority of the parameters of the model are shared between various NLP tasks. It also allows for efficient hardware implementations that hard-wire most of the network computation with the pre-trained weights, while only allowing few changeable parts for inference time.

Besides its empirical utility, the remarkable effectiveness of bias-only fine-tuning raises intriguing questions on the fine-tuning dynamics of pre-trained transformers, and the relation between the bias terms and transfer between LM and new tasks.

7. Acknowledgments

This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme, grant agreement No. 802774 (iEXTRACT).

Appendix

Section Summary: The appendix supplies supporting technical details for the BitFit experiments, including a mapping between the paper's layer notation and the parameter names in the Hugging Face BERT implementation. It then describes the training setup, such as attaching a linear classifier to the CLS token, using AdamW with specific learning rates and batch sizes for both full fine-tuning and bias-only updates, and notes on convergence and stability across GLUE tasks as well as RoBERTa and larger BERT models. Additional content covers the specific GLUE tasks and metrics employed, tables of optimal learning rates, visualizations showing how bias parameters change during training, and a performance comparison figure on the SQuAD benchmark.

A.1 Layer naming

For convenience, we relate the notation used in the paper with the names of the corresponding parameters in the popular HuggingFace [7] implementation.

::: caption="Table 4: Mapping the HuggingFace's BertLayer bias parameters names to BitFit paper bias notation."

:::

A.2 Training Details

To perform classification with BERT, we follow the approach of [1], and attach a linear layer to the contextual embedding of the [CLS] token to predict the label. The GLUE tasks are fed into BERT using the standard procedures.

We optimize using AdamW [30], with batch sizes of 16. For full fine-tuning, we used initial learning rates in 1e-5, 2e-5, 3e-5, 5e-5, and for the bias-only experiments we used initial learning rates in 1e-4, 4e-4, 7e-4, 1e-3 as the smaller rates took a very long time to converge on some of the tasks. With the larger learning rates, the bias-only fine-tuning converged in 8 or fewer epochs for most tasks, and up to 20 epochs on the others. We did not perform hyperparameter optimization beyond the minimal search over 4 learning rates. In each evaluation we report X $\pm$ Y where X is the average result for training 5 models with 5 different random seeds, Y is the standard deviation.

To perform classification with RoBERTa $_\text{BASE}$, we follow the above details but without hyperparameter search over the learning rates, for bias-only fine-tuning we used 1e-4 as learning rate and for full fine-tuning we used 1e-5 as learning rate.

As [31] show, fine-tuning BERT $\text{LARGE}$ and RoBERTa $\text{BASE}$ is a unstable due to vanishing gradients. BitFit allows for the usage of bigger learning rates, and overall the optimization process is much more stable, when compared with a full fine-tuning.

A.3 GLUE Benchmark

We provide information on the GLUE tasks we evaluated on, as well as on the evaluation metrics. We test our approach on the following subset of the GLUE [6] tasks: The Corpus of Linguistic Acceptability (CoLA; [32]), The Stanford Sentiment Treebank (SST-2; [33]), The Microsoft Research Paraphrase Corpus (MRPC; [34]), The Quora Question Pairs (QQP; [35]), The Semantic Textual Similarity Benchmark (STS-B; [36]), The Multi-Genre Natural Language Inference Corpus (MNLI; [37]), The Stanford Question Answering Dataset (QNLI; [38]) and The Recognizing Textual Entailment (RTE; [39]).

The metrics that we used to evaluate GLUE Benchmark are in Table 5. Learning rate configurations for best performing models are in Table 6. For all the experiments we used the common train:dev:test partition of GLUE.

: Table 5: Metrics that we use to evaluate GLUE Benchmark.

Task Name	Metric
QNLI	acc.
SST-2	acc.
MNLI	matched acc./mismatched acc.
CoLA	Matthews corr.
MRPC	F1
STS-B	Spearman corr.
RTE	acc.
QQP	F1

: Table 6: Learning rate configurations for best performing models.

Task Name	BERT $_\text{BASE}$	BERT $_\text{LARGE}$
QNLI	1e-4	7e-4
SST-2	4e-4	4e-4
MNLI	1e-4	1e-4
CoLA	7e-4	4e-4
MRPC	7e-4	1e-3
STS-B	1e-4	1e-4
RTE	1e-3	4e-4
QQP	4e-4	4e-4

A.4 Amount of change in bias terms

A.5 SQuAD F1 Results

References

Section Summary: This section compiles citations to research papers on language models such as BERT and its variants, along with methods for efficient training, compression, and pruning of neural networks. It also includes benchmarks for testing natural language understanding and tools for implementing transformer models. The references focus on improving model performance while reducing computational demands through techniques like knowledge distillation and sparse training.

[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.

[2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.

[3] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. CoRR, abs/1907.10529.

[4] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. CoRR, abs/1902.00751.

[5] Demi Guo, Alexander M. Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff pruning.

[6] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.07461.

[7] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

[8] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. 2020. Multi-head attention: Collaborate instead of concatenate. CoRR, abs/2006.16362.

[9] Shai Shalev-Shwartz and Shai Ben-David. 2014. Understanding machine learning: From theory to algorithms. Cambridge university press.

[10] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016a. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250.

[11] Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535–541.

[12] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.

[13] Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Özlem Aslan, Shengjie Wang, Abdelrahman Mohamed, Matthai Philipose, Matthew Richardson, and Rich Caruana. 2017. Do deep convolutional nets really need to be deep and convolutional?In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.

[14] Ehud D. Karnin. 1990. A simple procedure for pruning back-propagation trained neural networks. IEEE Trans. Neural Networks, 1(2):239–242.

[15] Russell Reed. 1993. Pruning algorithms-a survey. IEEE Trans. Neural Networks, 4(5):740–747.

[16] M. Gethsiyal Augasta and T. Kathirvalavakumar. 2013. Pruning algorithms of neural networks - a comparative study. Central Eur. J. Comput. Sci., 3(3):105–115.

[17] Chao Liu, Zhiyong Zhang, and Dong Wang. 2014. Pruning deep neural networks by optimal brain damage. In INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singapore, September 14-18, 2014, pages 1092–1095. ISCA.

[18] Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28:1135–1143.

[19] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2017. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.

[20] Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: studying the effects of weight pruning on transfer learning. CoRR, abs/2002.08307.

[21] Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.

[22] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre-trained BERT networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

[23] Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When BERT plays the lottery, all tickets are winning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3208–3229. Association for Computational Linguistics.

[24] Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020. Masking as an efficient alternative to finetuning for pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2226–2241, Online. Association for Computational Linguistics.

[25] Shengjie Wang, Tianyi Zhou, and Jeff A. Bilmes. 2019. Bias also matters: Bias attribution for deep neural network explanation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6659–6667. PMLR.

[26] Paul Michel and Graham Neubig. 2018. Extreme adaptation for personalized neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 312–318. Association for Computational Linguistics.

[27] Jonathan Frankle, David J. Schwab, and Ari S. Morcos. 2020. Training batchnorm and only batchnorm: On the expressive power of random features in cnns. CoRR, abs/2003.00152.

[28] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. 2020. Tiny transfer learning: Towards memory-efficient on-device learning. CoRR, abs/2007.11622.

[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.

[30] Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR, abs/1711.05101.

[31] Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines.

[32] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471.

[33] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.

[34] William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).

[35] Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017. First quora dataset release: Question pairs.

[36] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055.

[37] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.

[38] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.

[39] Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer.