AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Qingru Zhang$^{\dagger}$ $^{*}$, Minshuo Chen$^{\ddagger}$, Alexander Bukharin$^{\dagger}$, Nikos Karampatziakis$^{\diamond}$, Pengcheng He$^{\diamond}$, Yu Cheng$^{\diamond}$, Weizhu Chen$^{\diamond}$ and Tuo Zhao$^{\dagger}$
$^{\dagger}$Georgia Institute of Technology $^{\ddagger}$Princeton University $^{\diamond}$Microsoft Azure AI
{qingru.zhang, abukharin3, tourzhao}@gatech.edu
[email protected]
{nikosk, penhe, yu.cheng, wzchen}@microsoft.com
$^{*}$ Work was done during Qingru Zhang's internship at Microsoft Azure AI.
Abstract
Fine-tuning large pre-trained language models on downstream tasks has become an important paradigm in NLP. However, common practice fine-tunes all of the parameters in a pre-trained model, which becomes prohibitive when a large number of downstream tasks are present. Therefore, many fine-tuning methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e.g., low-rank increments. These methods often evenly distribute the budget of incremental updates across all pre-trained weight matrices, and overlook the varying importance of different weight parameters. As a consequence, the fine-tuning performance is suboptimal. To bridge this gap, we propose AdaLoRA, which adaptively allocates the parameter budget among weight matrices according to their importance score. In particular, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition. Such a novel approach allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact SVD computations. We conduct extensive experiments with several pre-trained models on natural language processing, question answering, and natural language generation to validate the effectiveness of AdaLoRA. Results demonstrate that AdaLoRA manifests notable improvement over baselines, especially in the low budget settings. Our code is publicly available at https://github.com/QingruZhang/AdaLoRA.
Executive Summary: Executive Summary
Large pre-trained language models deliver strong results on many NLP tasks but become impractical to adapt when dozens of downstream tasks must be supported. Full fine-tuning stores a separate copy of every model (hundreds of millions to billions of parameters), while existing parameter-efficient methods such as LoRA and adapter tuning allocate a fixed parameter budget evenly across layers and modules. This uniform allocation is suboptimal because weight matrices differ markedly in importance; experiments show that FFN layers and top transformer layers contribute far more to task performance than attention projections or bottom layers.
The paper introduces AdaLoRA, a method that dynamically reallocates the total parameter budget to the most valuable weight matrices during fine-tuning. It represents each incremental update as a low-rank SVD triplet and repeatedly scores every triplet with a sensitivity-plus-uncertainty metric. Low-scoring singular values are pruned, automatically concentrating capacity on critical modules while preserving the possibility of later recovery. A global cubic budget schedule starts above the target rank and gradually reduces it, improving training stability.
Across GLUE, SQuAD v1/v2, XSum and CNN/DailyMail, AdaLoRA consistently outperforms strong baselines. Gains are largest under tight budgets (0.08–0.3 % of total parameters): on SQuAD v2.0 it improves F1 by 1.2 points over the prior best method at the lowest budget; on GLUE it raises average score by 0.7–1.2 points while using comparable or fewer parameters. The method automatically assigns higher ranks to FFNs and upper layers, confirming that importance-aware allocation is both measurable and beneficial. Training-time overhead remains modest (11–16 % longer per epoch) and memory footprint is essentially unchanged.
These results indicate that adaptive rank allocation can materially improve the accuracy–efficiency trade-off for deploying large models on many tasks. Organizations facing memory or latency constraints should evaluate AdaLoRA on their target workloads, with particular attention to the lowest-budget regime where the relative gains are greatest. Follow-up work could test the approach on encoder–decoder or decoder-only models beyond BART and DeBERTa, explore task-specific importance metrics, or combine AdaLoRA with quantization for further compression.
The main limitations are the modest extra training cost, restriction to transformer weight matrices, and evaluation on a limited set of English benchmarks. Results are statistically significant (p < 0.05) and robust across multiple random seeds, giving reasonable confidence for the reported tasks and model families.
1. Introduction
Section Summary: Pre-trained language models like BERT and GPT-3 deliver strong results on language tasks but require costly full fine-tuning for each new application, since every task needs its own complete copy of the huge model. Researchers have developed more efficient approaches that either insert small trainable modules while freezing the base model or represent weight updates with compact low-rank matrices such as those used in LoRA. The introduction notes that these methods still distribute the limited training budget evenly across layers and modules, even though some parts of the model matter far more than others for good performance.
Pre-trained language models (PLMs) have manifested superior performance in various natural language processing tasks ([1, 2, 3, 4, 5]). The most common way to adapt pre-trained models to down-stream tasks is to fine-tune all the parameters (full fine-tuning, [6, 7]). However, pre-trained models typically incurs large memory footprint. For example, BERT model ([1]) consists up to 300 million parameters; T5 ([7]) comprises up to 11 billion parameters and GPT-3 ([5]) contains up to 175 billion parameters. When building a NLP system upon these pre-trained models, we usually handle multiple tasks that arrive simultaneously ([4]). Given a large number of down-stream tasks, full fine-tuning requires that each task maintains a separated copy of large models. The resulting memory consumption is prohibitively expensive.
To address this issue, researchers have proposed two main lines of research to reduce the fine-tuning parameters, while maintaining or even improving the performance of PLMs. Specifically, one line of research focuses on adding small neural modules to PLMs and fine-tune only these modules for each task – the base model is kept frozen and shared across tasks. In this way, only a small number of task-specific parameters are introduced and updated, greatly enhancing the practicality of large models. For example, adapter tuning ([8, 9, 10, 11]) inserts small neural modules called adapters between the layers of the base model. Prefix tuning ([12]) and prompt tuning ([13]) attach additional trainable prefix tokens to the input or hidden layers of the base model. These methods have shown to achieve comparable performance to full fine-tuning, while only updating less than $1%$ of the original model parameters, significantly releasing the memory consumption.
Another line of research proposes to model the incremental update of the pre-trained weights in a parameter-efficient way, without modifying the model architecture ([14, 15, 16]). Given a pre-trained weight matrix[^1] $W^{(0)} $, for example, diff pruning ([15]) models its incremental update $ \Delta $ as a sparse matrix. Diff pruning initializes $ \Delta $ as the same dimension as $ W^{(0)} $ and then prunes $ \Delta $ element-wise based on the magnitude of the entries. As such, diff pruning can increase the parameter efficiency substantially by adaptively retaining important updates and pruning unimportant ones. Nonetheless, diff pruning has several limitations. First, it relies on low-level implementation to speed up the computation of unstructured sparse matrices, which is not well supported by existing deep learning frameworks. Therefore, we have to store $ \Delta $ as a dense matrix during training. Second, it needs to update every entry of $\Delta$ with their gradients and then prune them. This results in similar computational cost as full fine-tuning ([15]).
[^1]: Unless specified otherwise, we use $ W^{(0)}$ to denote any pre-trained weight matrix.
To overcome these drawbacks, [16] propose a method named LoRA, which parameterizes $ \Delta $ as a low-rank matrix by the product of two much smaller matrices:
$ W = W^{(0)} + \Delta = W^{(0)} + B A,\tag{1} $
where $ W^{(0)}, \Delta \in \mathbb{R}^{d_1\times d_2} $, $ A\in \mathbb{R}^{r \times d_2} $ and $ B\in \mathbb{R}^{d_1\times r} $ with $r\ll { d_1, d_2 } $. During fine-tuning, only $ A $ and $ B $ are updated. The rank $ r $ is chosen to be much smaller than the dimension of $ W $ (e.g., $ r=8 $ when $ d_1 = d_2 = 1024 $). With less than $ 0.5% $ additional trainable parameters, the training overhead can be reduced up to $70%$, compared to full fine-tuning. However, LoRA achieves comparable or even better performance than full fine-tuning ([16]). Meanwhile, the product of two samll matrices is more friendly to implement and deploy than unstructured sparse matrices in diff pruning.
LoRA still has limitations as it prespecifies the rank $ r $ of each incremental matrix $\Delta$ identical. This ignores the fact that the importance of weight matrices varies significantly across modules and layers when fine-tuning pre-trained models. To illustrate this point, we present an concrete example in Figure 1. We compare the performance of LoRA when fine-tuning specific modules or layers with the same number of trainable parameters. Figure 1a shows that fine-tuning feed-forward networks (FFN) achieves better performance than self-attention modules. In addition, Figure 1b demonstrates that weight matrices in top layers are more important than those in bottom layers.
Adding more trainable parameters to the critical weight matrices can lead to better model performance. In contrast, adding more parameters to those less important weight matrices yields very marginal gains or even hurt model performance. Given the parameter budget, i.e., the number of total trainable parameters, we always prefer to allocate more parameters to those important modules. Distributing the budget evenly to all weight matrices/layers, like LoRA and other methods (e.g., adapter and prefix tuning), often gives suboptimal performance. To this end, a natural question is:
of modules to improve the performance of parameter-efficient fine-tuning?
To answer this question, we propose a new method – AdaLoRA ($\underline{Ada}$ ptive $\underline{Lo}w-\underline{R}$ ank $\underline{A}$ daptation), which dynamically allocates the parameter budget among weight matrices during LoRA-alike fine-tuning. Specifically, AdaLoRA adjusts the rank of incremental matrices to control their budget. Critical incremental matrices are assigned with high rank such that they can capture more fine-grained and task-specific information. Less importance ones are pruned to have lower rank to prevent overfitting and save the computational budget. There are some methods to control the rank of matrices in the existing literature of matrix approximation ([17, 18, 19]). Most of them directly compute singular value decomposition (SVD) of a matrix and then truncate the smallest singular values. Such an operation can manipulate the rank explicitly and, more importantly, minimize the difference between the resulting matrix and the original matrix. However, for fine-tuning large models, it becomes prohibitively expensive to iteratively apply SVD for a large number of high-dimensional weight matrices. Therefore, instead of computing SVD exactly, we parameterize $\Delta$ as $\Delta = P \Lambda Q$ to mimic SVD. The diagonal matrix $ \Lambda $ contains singular values while the orthogonal matrices $ P $ and $ Q $ represent left/right singular vectors of $ \Delta $. To regularize the orthogonality of $ P $ and $ Q $, an additional penalty is added to training loss. Such a parameterization avoids the intensive computations of SVD. Besides, another advantage is that we only need to drop the unimportant singular values while the singular vectors are maintained. This preserves the possibility of future recovery and stabilizes the training. See a detailed comparison to LoRA in Section 3.
Based on our SVD parameterization, AdaLoRA dynamically adjusts the rank of $\Delta = P \Lambda Q$ by importance scoring. Specifically, we divide the incremental matrix $P \Lambda Q$ into triplets, where each triplet $\mathcal{G}_i$ contains the $i$-th singular value and the corresponding singular vectors. To quantify the importance of triplets, we propose a novel importance metric, which takes account of the contribution of every entry in $ \mathcal{G}_i $ to the model performance ([20, 21, 22]). Triplets with low importance scores are granted low priority and hence the singular values are zeroed out. Triplets with high importance are retained for fine-tuning. Moreover, we also propose a global budget scheduler to facilitate the training. In particular, we start from an initial parameter budget, which is slightly higher than the final budget, and then gradually reduce it until matching the target. Such a scheduler can improve the training stability and model performance. Please see Section 3 for a detailed description of our importance metric and budget scheduler.
We conduct extensive experiments on a wide range of tasks and models to demonstrate the effectiveness of AdaLoRA. Specifically, we evaluate the performance using DeBERTaV3-base ([23]) on natural language understanding (GLUE, [24]) and question answering (SQuADv1, [25] and SQuADv2, [26]) datasets. We also apply our methods to BART-large ([27]) and evaluate the performance on natural language generation (XSum, [28] and CNN/DailyMail, [29]) tasks. We show AdaLoRA consistently outperforms the baseline, especially under low budget settings. For example, with less than $0.1%$ trainable parameters of full fine-tuning, AdaLoRA achieves a 1.2% F1 improvement on the SQuAD2.0 dataset compared with state-of-the-art approaches.
2. Background
Section Summary: Transformer models are built from repeated layers that use attention mechanisms to let each part of an input sequence influence the others, along with feed-forward networks that further transform the data, all stabilized by residual links and normalization. Low-rank adaptation (LoRA) offers an efficient way to fine-tune such large pretrained models by expressing weight changes as the product of two much smaller matrices rather than altering the full original weights. These small matrices are usually applied only to the attention and feed-forward parts, beginning at zero so training starts from the unchanged model.
Transformer-based Models. A typical transformer model consists of $ L $ stacked blocks, where each block contains two submodules: a multi-head attention (MHA) and a fully connected FFN. Given the input sequence $ X \in \mathbb{R}^{n\times d} $, MHA performs the attention function in parallel $ h $ heads:
$ \text{MHA}\left(X \right) = \text{Concat}(\text{head}1, ..., \text{head}h)W{o}, \quad \text{head}i = \text{Softmax}\left({X W{q_i} (X W{k_i})^{\top} } /{\sqrt{d_h}} \right) X W_{v_i}, $
where $ W_{o}\in \mathbb{R}^{d\times d} $ is an output projection and $ W_{q_i}, W_{k_i}, W_{v_i} \in \mathbb{R}^{d\times d_h} $ are query, key and value projections of head $ i $. $ d_h $ is typically set to $ d/h $. The other important module is a FFN which consists of two linear transformations with a ReLU activation in between: $\text{FFN}(X) = \text{ReLU}(X W_{f_1} + \bm{b}1)W{f_2} + \bm{b}2$, where $ W{f_1} \in \mathbb{R}^{d\times d_m} $ and $ W_{f_2} \in \mathbb{R}^{d_m \times d} $. Finally, a residual connection is used followed by a layer normalization ([30]).
Low Rank Adaptation. LoRA ([16]) models the incremental update of the pre-trained weights by the product of two small matrices. For $ \bm{h}= W^{(0)}\bm{x} $, the modified forward pass is:
$ \bm{h} = W^{(0)}\bm{x} + \Delta \bm{x} = W^{(0)} \bm{x} + B A \bm{x},\tag{2} $
where $ W^{(0)}, \Delta \in \mathbb{R}^{d_1\times d_2} $, $ A\in \mathbb{R}^{r \times d_2} $ and $ B\in \mathbb{R}^{d_1\times r} $ with $r\ll { d_1, d_2 } $. $ A $ typically adopts a random Gaussion initialization while $ B $ is initialized with zero to have $ \Delta = 0 $ at the beginning of training. We further denote $ A_{i*} $ as the $ i $-th row of $ A $, $ B_{* i} $ as the $ i $-th column of $ B $, and $ \mathcal{G}i = { A{i * }, B_{* i} } $ as the $ i $-th doublet. [16] only apply LoRA to query and value projections (i.e, $ W_{q} $ and $ W_{v} $) in the MHAs. [11] extend it to weight matrices of FFNs (i.e, $ W_{f_1} $ and $ W_{f_2} $), leading to the performance improvement . Meanwhile, they propose a unified view of various efficient tuning methods including adapter tuning, prefix tuning and LoRA.
3. AdaLoRA Method
Section Summary: AdaLoRA adapts large pre-trained models by expressing each weight update as a product of singular vectors and a diagonal matrix of singular values, which avoids costly recomputation of decompositions during training. It then iteratively prunes the least important singular values across all adapted matrices according to a learned importance score, thereby concentrating a limited parameter budget on the most useful updates while preserving the ability to restore pruned values later. A simple orthogonality regularizer keeps the decomposition well-behaved, yielding both greater stability and finer rank control than standard low-rank adaptation methods.
Our method contains two important components: (i) SVD-based adaptation, which formulates the incremental matrices in the form of singular value decomposition; (ii) Importance-aware rank allocation, which prunes redundant singular values based on our newly-designed importance metric.
3.1 SVD-Based Adaptation
As mentioned in Section 1, we propose to parameterize the incremental updates of the pre-trained weight matrices in the form of singular value decomposition:
$ W = W^{(0)} + \Delta = W^{(0)} + P \Lambda Q,\tag{3} $
where $ P\in \mathbb{R}^{d_1 \times r} $ and $ Q\in \mathbb{R}^{r \times d_2} $ represent the left/right singular vectors of $ \Delta $ and the diagonal matrix $ \Lambda \in \mathbb{R}^{r\times r} $ contains the singular values $ { \lambda_{i} }{1\leq i \leq r} $ with $ r\ll \min(d_1, d_2) $. We further denote $ \mathcal{G}i = {P{*i}, \lambda{i}, Q_{i*} } $ as the triplet containing the $i$-th singular value and vectors. In practice, since $\Lambda$ is diagonal, we only need to save it as a vector in $\mathbb{R}^r$. $ \Lambda $ is initialized with zero while $ P $ and $ Q $ adopt a random Gaussian initialization to ensure $ \Delta = 0 $ at the beginning of training. To enforce the orthogonality of $ P $ and $ Q $, i.e., $ P^{\top}P = Q Q^{\top} = I $, we utilize the following regularizer[^2]:
[^2]: We present the experiments in Appendix G to verify the effectiveness of the regularization.
$ R(P, Q) = \lVert P^{\top}P-I \rVert_{\sf F}^2 + \lVert QQ^{\top}-I \rVert_{\sf F}^2.\tag{4} $
In our method, $ \Lambda $ is iteratively pruned to adjust the rank after each gradient decent step. As mentioned in Section 1, one can directly compute SVD for every $ \Delta $ to manipulate singular values. The computational complexity, however, is $ O(\min(d_1, d_2) d_1 d_2) $. It becomes extremely expensive to iteratively apply SVD for a large number of high-dimensional incremental matrices. In contrast, our parameterization avoids intensive SVD computation, greatly releasing the computational overhead.
We remark that one can also apply structured pruning to LoRA to control the rank (i.e., prune $ B A $ doublet-wise in Equation (1)), whereas it has the following disadvantages. First, when a doublet is measured as unimportant, we have to prune all of its elements. It makes scarcely possible to reactivate the pruned doublets as their entries are all zeroed out and not trained. In contrast, AdaLoRA only masks out the singular values based on Equation (3) while the singular vectors are always maintained. It preserves the potential of future recovery for the triplets dropped by mistake. Second, $ A $ and $ B $ of LoRA are not orthogonal, meaning the doublets can be dependent with each other. Discarding the doublets can incur larger variation from the original matrix than truncating the smallest singular values. Therefore, the incremental matrices are often altered dramatically after each step of rank allocation, which causes training instability and even hurts generalization. To demonstrate this point, we present an ablation study in Section 4.4, which compares AdaLoRA with structured pruning for LoRA.
3.2 Importance-aware Rank Allocation
We apply the SVD-based adaptation Equation (3) to every weight matrix including $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_1}$ and $W_{f_2}$ of each transformer layer. In order to control the budget, we iteratively prune singular values in correspondence to their importance score during the training. For clear reference, we use $ k $ to index the incremental matrix, i.e., $ \Delta_{k} = P_{k} \Lambda_{k} Q_{k}$ for $k=1, \dots, n $, where $ n $ is the number of adapted weight matrices. We denote the $ i $-th triplet of $ \Delta_{k} $ as $ \mathcal{G}{k, i} = {P{k,i}, \lambda_{k, i}, Q_{k, i} } $ and its importance score as $ S_{k, i} $. We further denote the parameter sets $ \mathcal{P} = {P_{k} }{k=1}^{n} $, $ \mathcal{E} = {\Lambda{k} }{k=1}^{n} $, $ \mathcal{Q} = { Q{k} }{k=1}^{n} $ and training cost as $ \mathcal{C}(\mathcal{P}, \mathcal{E}, \mathcal{Q}) $. With the regularization Equation (4), the training objective is given by $ \mathcal{L}(\mathcal{P}, \mathcal{E}, \mathcal{Q}) = \mathcal{C}(\mathcal{P}, \mathcal{E}, \mathcal{Q}) + \gamma \sum{k=1}^{n} R(P_{k}, Q_{k}) $, where $\gamma>0$ is the regularization coefficient. At the $ t $-th step, we first take a stochastic gradient step to update $ P^{(t)}{k}, \Lambda^{(t)}{k} \text{ and } Q^{(t)}{k}$ for $ k=1, \dots, n $. Specifically, for $ \Lambda^{(t)}{k} $
$ \tilde{\Lambda}^{(t)}{k} = \Lambda^{(t)}{k} - \eta \nabla_{\Lambda_{k}} \mathcal{L}(\mathcal{P}^{(t)}, \mathcal{E}^{(t)}, \mathcal{Q}^{(t)}), $
where $ \eta > 0 $ is learning rate. Then, given importance score $ S^{(t)}_{k} $, the singular values are pruned following
$ \begin{aligned} \Lambda^{(t+1)}{k} = \mathcal{T}(\tilde{\Lambda}^{(t)}{k}, S^{(t)}{k}), \text{ with } \mathcal{T}(\tilde{\Lambda}^{(t)}{k}, S^{(t)}{k}){ii} = \left{ \begin{array}{lc} \tilde{\Lambda}^{(t)}{k, ii} & S^{(t)}{k, i} \text{ is in the top-}b^{(t)} \text{ of } S^{(t)}, \ 0 & \text{ otherwise,} \end{array} \right. \end{aligned} $
where $ S^{(t)} = { S^{(t)}{k, i} }{1\leq k \leq n, 1\leq i \leq r} $ contains the importance score of all triplets. Here $ b^{(t)} $ is the budget of remaining singular values at the $ t $-th step, which we explain more in Section 3.3. In this way, we leave more budget to the incremental matrices of higher priority by pruning the singular values of less important ones. In the sequel, we introduce several options to design the importance score.
Magnitude of singular values is the most straightforward way to quantify the importance of every triplet, i.e., $ S_{k, i} = \lvert \lambda_{k, i} \rvert $. In this way, only the least significant singular values are discarded. It minimizes the deviation from the original matrix and further stabilizes the training. Many existing methods use this criterion to control the rank of matrix ([17, 18, 19]). However, we remark that such a simple metric cannot properly quantify the contribution of parameters to model performance.
Sensitivity-based importance is another option for importance scoring, which quantifies the sensitivity of parameters to the training loss ([31, 20, 21, 22]). The prior work, however, leverages the sensitivity to quantify the importance of single entries and applies it for unstructured pruning that prunes weights element-wise. When it turns to our case, we have to design a new metric as the triplets are discarded group-wise. Every entry's sensitivity ought to be considered and properly combined to quantify the overall contribution of the triplet to model performance. Therefore, we propose a newly-designed importance metric in account of both the singular value and vectors in triplet $ \mathcal{G}_{k, i} $:
$ S_{k, i} = s(\lambda_{k, i}) + \frac{1}{d_1} \sum_{j=1}^{d_1} s(P_{k, ji}) + \frac{1}{d_2} \sum_{j=1}^{d_2} s(Q_{k, ij}),\tag{5} $
where we calculate the mean importance of $ P_{k,i} $ and $ Q_{k, i} $ such that $ S_{k, i} $ does not scale with the number of parameters in $ \mathcal{G}_{k, i} $. Here $ s(\cdot) $ is a specific importance function for single entries. We can adopt the sensitivity for $ s(\cdot) $, which is defined as the magnitude of the gradient-weight product:
$ I(w_{ij}) = | w_{ij} \nabla_{w_{ij}}\mathcal{L} |,\tag{6} $
where $ w_{ij} $ is any trainable parameter. Equation (6) essentially approximates the change in loss when a parameter is zeroed out. If the removal of a parameter has a large influence, then the model is sensitive to it and we should retain it ([31, 21, 22]).
However, [22] point out that the sensitivity in Equation (6) is not yet a reliable importance indicator. Such a score is estimated on the sampled mini batch. The stochastic sampling and complicated training dynamics incur high variability and large uncertainty for estimating the sensitivity with Equation (6). Therefore, [22] propose to resolve this issue by sensitivity smoothing and uncertainty quantification:
$ \begin{aligned} \overline{I}^{(t)}(w_{ij}) = & \beta_1 \overline{I}^{(t-1)}(w_{ij}) + (1-\beta_1) I^{(t)}(w_{ij}) \quad\text{(a)} \ \overline{U}^{(t)}(w_{ij}) = & \beta_2 \overline{U}^{(t-1)} (w_{ij}) + (1-\beta_2) \Big\lvert I^{(t)}(w_{ij}) - \overline{I}^{(t)}(w_{ij}) \Big\rvert \quad\text{(b)}, \end{aligned}\tag{7} $
where $ 0<\beta_1, \beta_2 <1 $. $ \overline{I}^{(t)} $ is the smoothed sensitivity by exponential moving average and $ \overline{U}^{(t)} $ is the uncertainty term quantified by the local variation between $ I^{(t)} $ and $ \overline{I}^{(t)} $. Then they define the importance as the product between $ \overline{I}^{(t)} $ and $ \overline{U}^{(t)} $, which can be another option for $ s(\cdot) $:
$ s^{(t)}(w_{ij}) = \overline{I}^{(t)}(w_{ij}) \cdot \overline{U}^{(t)}(w_{ij}).\tag{8} $
We present a detailed ablation study in Section 4.4 to compare the performance of different importance metrics. We find the proposed metric Equation (5) based on the sensitivity variant Equation (8) generally performs best. We summarize the detailed algorithm in Algorithm 1.
**Input:** Dataset $ \mathcal{D} $; total iterations T; budget schedule ( b(t) ) for t=0..T; hyperparameters η, γ, beta₁, beta₂.
**for** t = 1, …, T **do**
Sample a mini-batch from $\mathcal{D}$ and compute the gradient ∇ L(P, E, Q);
Compute the sensitivity I(t) in Equation (6) for every parameter in ( P, E, Q );
Update $ \overline{I}^{(t)} $ as Equation (7a) and $ \overline{U}^{(t)} $ as Equation (7b) for every parameter in ( P, E, Q );
Compute S(t)(k,i) by Equation (5), for k=1,…,n and i=1,…,r ;
Update P(k)(t+1) = P(t)(k) - η ∇(P(k))L(P, E, Q) and Q(k)(t+1) = Q(k)(t) - η ∇(Q(k))L(P, E, Q);
Update Lambda(t+1)(k) = T(Lambda(t)(k) -η ∇(Lambda(k))L(P, E, Q), S(t)(k)) given the budget b(t).
**end for**
**Output:** The fine-tuned parameters ( P(T), E(T), Q(T) ).
3.3 Global Budget Scheduler
As mentioned in Section 1, adjusting the rank is naturally to control the parameter budget in the context of low-rank adaptation. Hence we define the budget $ b^{(t)} $ as the total rank of all incremental matrices, i.e., the number of total singular values. Recall that the budget allocation is iteratively conducted during the fine-tuning. To facilitate the training, we propose a global budget scheduler. Specifically, we start from an initial budget $ b^{(0)} $ that is slightly higher than the target budget $ b^{(T)} $ (e.g., 1.5 times of $ b^{(T)} $). We set the initial rank of each incremental matrix as $r = b^{(0)} / n$. We warm up the training for $t_i$ steps, and then follow a cubic schedule to decrease the budget $b^{(t)}$ until it reaches $b^{(T)}$. Finally, we fix the resulting budget distribution and fine-tune the model for $ t_f $ steps. The exact equation for the budget schedule is presented in Appendix A. This allows AdaLoRA to explore the parameter space first and then focus on the most important weights later.
4. Experiments
Section Summary: The experiments apply AdaLoRA to fine-tune DeBERTaV3-base and BART-large on natural language understanding, question answering, and generation benchmarks such as GLUE, SQuAD, XSum, and CNN/DailyMail. The approach is implemented in PyTorch with the Hugging Face library and compared against full fine-tuning, BitFit, two adapter variants, and an extended version of LoRA across several low parameter budgets on NVIDIA V100 GPUs. Results show that AdaLoRA matches or exceeds the baselines in nearly all settings, with especially strong gains at the smallest budgets, and all improvements pass statistical significance tests.
The section details periodic pruning of singular values and consistent scaling drawn from LoRA to maintain stable training dynamics.
We implement AdaLoRA for fine-tuning DeBERTaV3-base ([23]) and BART-large ([27]). We evaluate the effectiveness of the proposed algorithm on natural language understanding (GLUE, [24]), question answering (SQuADv1, [25] and SQuADv2, [26]), and natural language generation (XSum, [28] and CNN/DailyMail [29]). All the gains have passed significant tests with $ p<0.05 $.
Implementation Details. We use PyTorch ([32]) to implement all the algorithms. Our implementation is based on the publicly available Huggingface Transformers^3 ([33]) code-base. All the experiments are conducted on NVIDIA V100 GPUs.
LoRA scales $ \Delta \bm{x} $ by $ \alpha/r $ where $\alpha$ is a constant in $ r $. As a result, the magnitude of output can be consistent given different $ r $. It reduces the efforts of retuning learning rate when varying $ r $. Typically $ \alpha $ is set as $16$ or $32$ and never tuned ([16, 34]). Following LoRA, we add the same scaling for Equation (3) and fix $ \alpha $ as LoRA. Besides, in Algorithm 1, we prune singular values every $ \Delta_{T} $ steps (e.g., $ \Delta_{T} = 100 $) such that the pruned triplets can still get updated within these intervals and possibly reactivated in future iterations.
Baselines. We compare AdaLoRA with the following methods:
$ \bullet $ Full fine-tuning is the most common approach for adaptation. During fine-tuning, the model is initialized with pre-trained weights and biases, and all model parameters undergo gradient updates.
$\bullet$ Bitfit ([14]) is an effective parameter-efficient fine-tuning method. The method only fine-tunes bias vectors in the pre-trained model.
$ \bullet $ Adapter tuning ([8, 10]) inserts two-layer adapters between transformer blocks. We compare with two types of adapter. Houlsby adapter as proposed in [8] is inserted between the self-attention module and the FFN module followed by a subsequent residual connection. Recently, [10] propose a more efficient design with adapters only applied after FFN modules and LayerNorm modules ([30]), which we call Pfeiffer adapter. The number of trainable parameters is determined by the number of layers, the hidden dimension of adapters and the dimension of their inputs.
$ \bullet $ LoRA ([16]) is a state-of-the-art method for parameter-efficient fine-tuning. The method parameterizes incremental updates by two small matrices and only fine-tune them. The number of trainable parameter is controlled by the rank $ r $ and the number of adapted weight matrices $ n $. [16] apply LoRA to query and value projections only. In empirical, we find that applying LoRA to all weight matrices, i.e., $ W_{q}, W_{k}, W_{v}, W_{f_1} \text{ and } W_{f_2} $, can further improve its performance (Please see Appendix F). Hence, we compare with this generalized LoRA to maximize its performance. We use publicly available implementation ^4 to run all the baselines. Please refer to [16] and reference therein for details.
::: {caption="Table 1: Results with DeBERTaV3-base on GLUE development set. The best results on each dataset are shown in bold. We report the average correlation for STS-B. Full FT, HAdapter and PAdapter represent full fine-tuning, Houlsby adapter, and Pfeiffer adapter respectively. We report mean of 5 runs using different random seeds."}
:::
4.1 Natural Language Understanding
Models and Datasets. We evaluate the fine-tuning performance of DeBERTaV3-base ([23]) using the proposed algorithm. We conduct experiments on the General Language Understanding Evaluation (GLUE, [24]) benchmark. The benchmark includes two single-sentence classification tasks, three similarity and paraphrase tasks and four natural language inference tasks. Dataset details are summarized in Appendix B.
Implementation Details. DeBERTaV3-base consists of 183 millions parameters. We compare AdaLoRA with the baselines under different budget levels, for example, given the total trainable parameters as 0.3/0.6/1.2 million. In order to match the parameter budget, we select the hidden dimensions of adapters from $ { 8, 16, 32, 64 } $, set the rank $ r $ of LoRA as $ {2, 4, 8} $, and choose the final budget $ b^{(T)} $ of AdaLoRA from $ {144, 288, 576} $. Then we set $b^{(0)}$ as 1.5 times of $b^{(T)}$ for AdaLoRA and select the regularization coefficient $ \gamma $ from $ { 0.1, 0.3, 0.5 } $. We set the exponential moving average parameters $ \beta_1 $ and $ \beta_2 $ as their default value $0.85$. We select the learning rate from ${5\times 10^{-5}, 8\times 10^{-5}, 1\times 10^{-4}, 2\times 10^{-4} }$. More details are presented in Appendix C.
Main results. We compare AdaLoRA with the baseline methods under different budget settings. Table 1 shows experimental results on the GLUE development set. We see that AdaLoRA achieves better or on par performance compared with existing approaches on all datasets under all budget levels. For example, when the parameter budget is 0.3M, AdaLoRA achieves 87.36% accuracy on RTE, which is 1.8% higher than the best-performing baseline. Besides, AdaLoRA with extreme low budget can often perform better than the baselines with higher budget. For example, AdaLoRA achieve 70.04% Mcc. score on CoLA with 0.3M fine-tuning parameters, which is higher than all baseline methods with lager budget (e.g., 0.6M and 1.2M).
4.2 Question Answering
Models and Datasets. We evaluate performance of the proposed algorithm on two question answering (QA) datasets: SQuAD v1.1 ([25]) and SQuADv2.0 ([26]), where we use AdaLoRA to fine-tune DeBERTaV3-base. These tasks are treated as a sequence labeling problem, where we predict the probability of each token being the start and end of the answer span. Dataset details can be found in Appendix D.
Implementation Details. We compare AdaLoRA with the baseline methods under different parameter budgets. That is we have the number of trainable parameters as $ 0.08% / 0.16% / 0.32% / 0.65% $ of total pre-trained parameters. To match the budget requirements, we select the hidden dimensions of adapters from $ {4, 8, 16, 32, 64 } $, set the rank $ r $ of LoRA as $ {1, 2, 4, 8} $ and choose the final total rank $ b^{(T)} $ of AdaLoRA from $ {72, 144, 288, 576} $. We set the batch size as $16$. We use AdamW ([35]) as the optimizer and we set the learning rate as $ 1\times 10^{-3} $ for AdaLoRA. Please refer to Appendix D for more details.
Main Results. Table 2 summarizes experimental results when we fine-tune DeBERTaV3-base under 4 different budget settings: 0.08%, 0.16%, 0.32% and 0.65% of total pre-trained parameters. From the result, we see that AdaLoRA consistently outperforms existing approaches under all the budget levels in term of two evaluation metrics: exact match (EM) and F1. Notice that the performance of Houlsby adapter and Pfeiffer adapter are notably decreased when we reduce the parameter budget. In contrast, our method shows the consistent performance under different budget levels. For example, AdaLoRA achieves 88.7% F1 on SQuADv2.0 with the smallest budget 0.08%. It is close to its performance under the high budget and it is also 1.2% higher than the best-performing baseline.
\begin{tabular}{l|cccc|cccc}
\toprule
& \multicolumn{4}{|c}{\bf SQuADv1.1} & \multicolumn{4}{|c}{\bf SQuADv2.0}
\\
\midrule
{\small Full FT} & \multicolumn{4}{|c}{86.0 / 92.7} & \multicolumn{4}{|c}{85.4 / 88.4}
\\
\midrule
{\small # Params} & {0.08\%} & {0.16\%} & {0.32\%} & {0.65\%} & {0.08\%} & {0.16\%} & {0.32\%} & {0.65\%}
\\
\midrule
{\small HAdapter} & 84.4/91.5 & 85.3/92.1 & 86.1/92.7 & 86.7/92.9 & 83.4/86.6 & 84.3/87.3 & 84.9/87.9 & 85.4/88.3
\\
{\small PAdapter} & 84.4/91.7 & 85.9/92.5 & 86.2/92.8 & 86.6/93.0 & 84.2/87.2 & 84.5/87.6 & 84.9/87.8 & 84.5/87.5
\\
{\small LoRA} & 86.4/92.8 & 86.6/92.9 & 86.7/93.1 & 86.7/93.1 & 84.7/87.5 & 83.6/86.7 & 84.5/87.4 & 85.0/88.0
\\
\midrule
{\small AdaLoRA} & {\bf 87.2/93.4} & {\bf 87.5/93.6} & {\bf 87.5/93.7} & {\bf 87.6/93.7} & {\bf85.6/88.7} & {\bf85.7/88.8} & {\bf85.5/88.6} & {\bf86.0/88.9}
\\
\bottomrule
\end{tabular}
4.3 Natural Language Generation
\begin{tabular}{l|c|c|c}
\toprule
{\bf # Params} & {\bf Method} & {\bf XSum} & {\bf CNN/DailyMail}
\\
\midrule
{\bf100\%} & Full FT & {\bf 45.49 / 22.33 / 37.26} & 44.16 / 21.28 / 40.90
\\
\midrule
\multirow{2}*{\bf 2.20\%} & LoRA & 43.95 / 20.72 / 35.68 & {{\bf 45.03} / 21.84 / 42.15}
\\
& AdaLoRA & {\bf 44.72 / 21.46 / 36.46} & {45.00 / {\bf 21.89} / {\bf 42.16}}
\\
\midrule
\multirow{2}*{\bf 1.10\%} & LoRA & 43.40 / 20.20 / 35.20 & 44.72 / 21.58 / 41.84
\\
& AdaLoRA & {\bf 44.35 / 21.13 / 36.13} & {\bf 44.96 / 21.77 / 42.09}
\\
\midrule
\multirow{2}*{\bf 0.26\%} & LoRA & 43.18 / 19.89 / 34.92 & 43.95 / 20.91 / 40.98
\\
& AdaLoRA & {\bf 43.55 / 20.17 / 35.20} & {\bf 44.39 / 21.28 / 41.50}
\\
\midrule
\multirow{2}*{\bf 0.13\%} & LoRA & 42.81 / 19.68 / 34.73 & 43.68 / 20.63 / 40.71
\\
& AdaLoRA & {\bf 43.29 / 19.95 / 35.04} & {\bf 43.94 / 20.83 / 40.96}
\\
\bottomrule
\end{tabular}
Models and Datasets. To provide a comparison with the state-of-the-art in natural language generation (NLG) tasks, we apply AdaLoRA to fine-tune a BART-large model ([27]). We evaluate model performance on two datasets: XSum ([28]) and CNN/DailyMail ([29]).
Implementation Details. Similarly as DeBERTav3-base, we apply low-rank/SVD-based adaptation to every weight matrix of both encoder and decoder layers. We report ROUGE 1/2/L scores (R-1/2/L, [36]). We set the training epochs as 15. For XSum, we set the beam length as 8 and batch size as 64. For CNN/DailyMail, we set the beam length as 4 and batch size as 32. Please see Appendix E for the detailed configuration.
Main Results. Experimental results are summarized in Table 3, where we compare the fine-tuning performance under four budget levels: the number of trainable parameters is 0.13%, 0.26%, 1.10% and 2.20% of total pre-trained parameters. We see that AdaLoRA achieves better or on par performance compared with the baseline on both datasets (XSum and CNN/DailyMail) under all the budget levels. For example, AdaLoRA achieves 21.13 R-2 score when budget level is 1.10%, compared with 19.89 for LoRA.
4.4 Analysis
Different budget levels. Figure 2 illustrates experimental results of fine-tuning DeBERTaV3-base under different budget levels. We see that on all the three datasets (MNLI-m, SQuADv2.0 and XSum), AdaLoRA achieves consistent performance improvement under all the budget levels compared with the baseline. The performance gain is more significant when increasing the budget for the XSum task, suggesting a high budget can help NLG tasks. Note that on the MNLI and SQuADv2.0 datasets, the performance of AdaLoRA under low budget levels ($ \leq 1% $) can match the results of high budget settings. For example, AdaLoRA achieves $ 88.78% $ F1 on SQuADv2.0 when the budget is $ 0.16% $. It is close to the performance (88.89% F1) of the highest budget ($ 4.65% $) with a more significant gain over the baseline.
Comparison to low-rank parameterization. As mentioned in Section 3.1, one can alternatively prune LoRA doublet-wise to conduct the rank allocation. In this case, the doublets are zeroed out entirely, raising the barrier to reactivate them. It can cause training instability and hurt the generalization when some crucial doublets are pruned by mistake. In Table 4, we compare AdaLoRA with pruning LoRA on three datasets (SST-2, RTE, and CoLA) to illustrate this point. We apply the same importance score, budget scheduler and training setups as Section 4.1 for pruning LoRA. We can see that AdaLoRA outperforms pruning LoRA on all the datasets under all the budget levels.
\begin{tabular}{c|ccc|ccc|ccc}
\toprule
& \multicolumn{3}{|c}{\bf SST-2} & \multicolumn{3}{|c}{\bf RTE} & \multicolumn{3}{|c}{\bf CoLA}
\\
\midrule
{# Params}
& {0.08\%} & {0.16\%} & {0.65\%}
& {0.08\%} & {0.16\%} & {0.65\%}
& {0.08\%} & {0.16\%} & {0.65\%}
\\
\midrule
{\small Prune LoRA}
& 94.84 & 94.50 & 94.95
& 86.28 & 86.15 & 87.00
& 66.71 & 69.29 & 69.57
\\
\midrule
AdaLoRA
& 95.52 & 95.80 & 96.10
& 87.36 & 87.73 & 88.09
& 70.21 & 70.04 & 71.45
\\
{$ s(\cdot) = I(\cdot) $}
& 94.61 & 95.30 & 95.64
& 87.36 & 87.71 & 88.10
& 66.71 & 68.83 & 70.19
\\
{$ S_{i} = |\lambda_{i}| $}
& 95.41 & 95.41 & 95.87
& 87.00 & 86.28 & 88.00
& 67.67 & 68.44 & 70.38
\\
\bottomrule
\end{tabular}
Variants of the importance score. Recall that in AdaLoRA, the importance score is defined by the sensitivity and uncertainty of every entry in the triplet Equation (5). In Table 4, we examine two variants of the importance score: (i) changing $ s(\cdot) $ in Equation (5) to sensitivity-only; (ii) directly defining $ S_{i} $ as $ |\lambda_{i}| $. From the results, we can see that the proposed importance score generally performs best. The other two variants can degenerate the model performance up to $ 0.9% $.
The role of two components. We remark that both two components of our method - SVD adaptation and adaptive budget allocation, play vital roles for the performance gain. To demonstrate it, we compare AdaLoRA with the following variants: (i) SVD-LoRA: fine-tuning only with the proposed SVD-based adaptation in Equation (3) and (4); (ii) LoRA $\text{regu}$: LoRA with orthogonal regularization Equation (4) on $A$ and $B$; (iii) AdaLoRA ${\gamma=0}$: AdaLoRA without orthogonal regularization Equation (4). Table 5 present the results when fine-tuning DeBERTaVe-base on SST-2 and MNLI. We can see that fine-tuning only with SVD adaptation shows an improvement over LoRA but cannot match the performance of AdaLoRA. Meanwhile, without SVD orthogonal regularization, the performance of AdaLoRA can degenerate. These results validate that both components contribute to the model performance.
\begin{tabular}{c|cccc|cccc}
\toprule
& \multicolumn{4}{|c}{\bf SST-2} & \multicolumn{4}{|c}{\bf MNLI}
\\
\midrule
{# Params}
& {0.08\%} & {0.16\%} & {0.32\%} & {0.65\%}
& {0.08\%} & {0.16\%} & {0.32\%} & {0.65\%}
\\
\midrule
LoRA
& 94.38 & 94.95 & - & 94.95
& 90.19 & 90.34 & - & 90.57
\\
{LoRA\textsubscript{regu}}
& - & 94.61 & 94.72 & 94.61
& - & 90.30 & 90.40 & 90.66
\\
SVD-LoRA
& 95.33 & 95.18 & 95.07 & 95.53
& 90.28 & 90.25 & 90.52 & 90.62
\\
\midrule
{{AdaLoRA}\textsubscript{$\gamma=0$}}
& 95.41 & 95.10 & 95.30 & 95.10
& 90.37 & 90.34 & 90.56 & 90.43
\\
{{AdaLoRA}}
&95.64 & 95.80 & 96.10 & 96.10
&90.65 & 90.68 & 90.66 & 90.77
\\
\bottomrule
\end{tabular}
The resulting budget distribution. Figure 3 shows the resulting rank of each incremental matrix of DeBERTaV3-base fine-tuned with AdaLoRA. We find that AdaLoRA always prefers to allocating more budget to FFNs and top layers. Such behavior aligns with our empirical conclusions presented in Figure 1 that weight matrices of FFN moduels and top layers are more important for model performance. Hence, it validates that our proposed importance metric can guide AdaLoRA to focus on crucial modules. Meanwhile, the rank distribution generated by AdaLoRA is consistent across different budget levels, tasks and models. It means the number of remaining parameters is linearly scaled with $b^{(T)}$ and hence we can tune $b^{(T)}$ to control the remaining parameters.
5. Conclusion
Section Summary: The researchers introduce AdaLoRA, a technique for efficiently adapting large models to new tasks by automatically assigning a limited number of adjustable parameters according to their importance. The approach represents weight updates through singular value decomposition and dynamically tunes them to boost both performance and efficiency. Experiments across language understanding, question answering, and text generation tasks show that it outperforms prior methods.
We propose a parameter-efficient fine-tuning method – AdaLoRA that adaptively allocates the parameter budget according to importance scoring. In AdaLoRA, we parameterize the incremental updates of weight matrices in the form of singular value decomposition. Then, we dynamically allocate the parameter budget among incremental matrices by manipulating the singular values based on a new importance metric. Such an a pproach effectively improves the model performance and parameter efficiency. We conduct extensive experiments on natural language processing, question answering and natural language generation tasks. Results show that AdaLoRA outperforms existing approaches.
Appendix
Section Summary: The appendix supplies supporting technical details for the paper’s experiments, beginning with the cubic formula used to gradually reduce the global budget parameter over training steps. It then includes tables listing dataset sizes and metrics for the GLUE benchmark, SQuAD question-answering sets, and summarization tasks, along with the precise hyperparameter choices, adapter dimensions, rank values, learning rates, and epoch counts applied to each method and dataset under different parameter budgets. These elements allow readers to reproduce the reported training configurations for natural-language understanding, question answering, and generation.
A. Global Budget Schedule
As mentioned in Section 3.3, we propose a global budget scheduler to gradually decrease the budget $b^{(t)}$ following a cubic schedule. The detailed equation is given as follows:
$ b^{(t)} = \begin{cases} b^{(0)} & 0 \leq t<t_{i} \ b^{(T)}+ \left(b^{(0)}-b^{(T)}\right)\left(1-\frac{t-t_{i}-t_{f}}{T-t_i - t_f}\right)^{3} & t_{i} \leq t<T-t_{f} \ b^{(T)} &\text o.w. \end{cases}.\tag{9} $
B. GLUE Dataset Statistics
We present the dataset statistics of GLUE ([24]) in the following table.
\begin{tabular}{l|l|c|c|c|c|c}
\toprule
\bf Corpus &Task& #Train & #Dev & #Test & #Label &Metrics\\ \midrule
\multicolumn{6}{@{\hskip1pt}r@{\hskip1pt}}{Single-Sentence Classification (GLUE)} \\ \hline
CoLA & Acceptability&8.5k & 1k & 1k & 2 & Matthews corr\\ \hline
SST & Sentiment&67k & 872 & 1.8k & 2 & Accuracy\\ \midrule
\multicolumn{6}{@{\hskip1pt}r@{\hskip1pt}}{Pairwise Text Classification (GLUE)} \\ \hline
MNLI & NLI& 393k& 20k & 20k& 3 & Accuracy\\ \hline
RTE & NLI &2.5k & 276 & 3k & 2 & Accuracy \\ \hline
QQP & Paraphrase&364k & 40k & 391k& 2 & Accuracy/F1\\ \hline
MRPC & Paraphrase &3.7k & 408 & 1.7k& 2&Accuracy/F1\\ \hline
QNLI & QA/NLI& 108k &5.7k&5.7k&2& Accuracy\\ \midrule
\multicolumn{5}{@{\hskip1pt}r@{\hskip1pt}}{Text Similarity (GLUE)} \\ \hline
STS-B & Similarity &7k &1.5k& 1.4k &1 & Pearson/Spearman corr\\ \bottomrule
\end{tabular}
C. Natural Language Understanding
C.1 Budget Configuration
For each budget level, we tune the final budget $b^{(T)}$ for AdaLoRA, the rank $r$ for LoRA, the hidden dimension $d$ for two adapters to match the budget requirements.
\begin{tabular}{l|cccc}
\toprule
{# Params} & Houlsby Adapter ($d$) & Pfeiffer Adapter ($d$) & LoRA ($r$) & {{AdaLoRA} ($b^{(T)}$)}
\\
\midrule
1.2M & 32 & 64 & 8 & 576
\\
0.6M & 16 & 32 & 4 & 288
\\
0.3M & 8 & 16 & 2 & 144
\\
\bottomrule
\end{tabular}
Alternatively, we can also set the final average rank $ \bar{r}^{(T)} = b^{(T)} / n $ for AdaLoRA to control the budget, which is set as 2, 4, and 8 given the final budget as 144, 288, and 576 respectively. Then we select the initial rank $r$ from ${4, 6, 12}$ for the final average rank ${ 2, 4, 8}$ respectively.
C.2 Training Details
We tune the learning rate from $ { 8\times10^{-5}, 5\times10^{-5}, 3\times10^{-5}, 1\times10^{-4}, 3\times10^{-4}, 5\times10^{-4}, 8\times10^{-4}, 1\times10^{-3} } $ and pick the best learning rate for every method. For each dataset, the batch size is set as identical for every method.
\begin{tabular}{l|ccccccc}
\toprule
Dataset & learning rate & batch size & {# epochs} & {$\gamma$} & $t_i$ & {$\Delta_{T}$} & $t_f$
\\
\midrule
{\bf MNLI} & {$5\times 10^{-4}$} & 32 & 7 & 0.1 & 8000 & 100 & 50000
\\
{\bf RTE} & $ 1.2\times 10^{-3} $ & 32 & 50 & 0.3 & 600 & 1 & 1800
\\
{\bf QNLI} & $ 1.2\times 10^{-3} $ & 32 & 5 & 0.1 & 2000 & 100 & 8000
\\
{\bf MRPC} & $ 1\times 10^{-3} $ & 32 & 30 & 0.1 & 600 & 1 & 1800
\\
{\bf QQP } & $5\times 10^{-4}$ & 32 & 5 & 0.1 & 8000 & 100 & 25000
\\
{\bf SST-2} & $ 8\times 10^{-4} $ & 32 & 24 & 0.1 & 6000 & 100 & 22000
\\
{\bf CoLA} & $ 5\times 10^{-4} $ & 32 & 25 & 0.5 & 800 & 10 & 3500
\\
{\bf STS-B} & $ 2.2\times 10^{-3} $ & 32 & 25 & 0.1 & 800 & 10 & 2000
\\
\bottomrule
\end{tabular}
D. Question Answering
D.1 Budget Configuration
Given the budget, we control the trainable parameters for each method as the following table.
\begin{tabular}{l|cccc}
\toprule
{# Params} & Houlsby Adapter & Pfeiffer Adapter & LoRA & {{AdaLoRA}}
\\
& $ d $ & $ d $ & $ r $ & $ b^{(T)} / \bar{r}^{(T)} / r $
\\
\midrule
{0.65\%} & 32 & 64 & 8 & 576 / 8 / 12
\\
{0.32\%} & 16 & 32 & 4 & 288 / 4 / 6
\\
{0.16\%} & 8 & 16 & 2 & 144 / 2 / 4
\\
{0.08\%} & 4 & 8 & 1 & 72 / 1 / 2
\\
\bottomrule
\end{tabular}
D.2 Training Details
We set the batch size as 16. We select the learning rate from $ { 8\times10^{-5}, 5\times10^{-5}, 3\times10^{-5}, 1\times10^{-4}, 3\times10^{-4}, 5\times10^{-4}, 8\times10^{-4}, 1\times10^{-3} } $ and pick the best-performing learning rate for every method. The configuration of AdaLoRA is listed in the following table.
\begin{tabular}{l|ccccccc}
\toprule
Dataset & learning rate & batch size & {# epochs} & {$\gamma$} & $t_i$ & {$\Delta_{T}$} & $t_f$
\\
\midrule
{\bf SQuADv1.1} & {$1\times 10^{-3}$} & 16 & 10 & 0.1 & 5000 & 100 & 25000
\\
{\bf SQuADv2.0} & $ 1\times 10^{-3} $ & 16 & 12 & 0.1 & 5000 & 100 & 50000
\\
\bottomrule
\end{tabular}
D.3 Dataset
The statistics of question answering datasets are summarized in Table 11.
: Table 11: Statistics of the SQuAD dataset.
| No. Train | No. Validation | |
|---|---|---|
| SQuAD v1.1 | 87, 599 | 10, 570 |
| SQuAD v2.0 | 130, 319 | 11, 873 |
E. Natural Language Generation
E.1 Budget Configuration
Given the budget, we control the trainable parameters for each method as the following table.
\begin{tabular}{l|cccc}
\toprule
{# Params} & Houlsby Adapter & Pfeiffer Adapter & LoRA & {{AdaLoRA}}
\\
& $ d $ & $ d $ & $ r $ & $ b^{(T)} / \bar{r}^{(T)} / r $
\\
\midrule
{0.65\%} & 32 & 64 & 8 & 576 / 8 / 12
\\
{0.32\%} & 16 & 32 & 4 & 288 / 4 / 6
\\
{0.16\%} & 8 & 16 & 2 & 144 / 2 / 4
\\
{0.08\%} & 4 & 8 & 1 & 72 / 1 / 2
\\
\bottomrule
\end{tabular}
E.2 Training Details
We set the batch size as 16. We select the learning rate from $ { 8\times10^{-5}, 5\times10^{-5}, 3\times10^{-5}, 1\times10^{-4}, 3\times10^{-4}, 5\times10^{-4}, 8\times10^{-4}, 1\times10^{-3} } $ and pick the best-performing learning rate for every method. The configuration of AdaLoRA is listed in the following table.
\begin{tabular}{l|ccccccc}
\toprule
Dataset & learning rate & batch size & {# epochs} & {$\gamma$} & $t_i$ & {$\Delta_{T}$} & $t_f$
\\
\midrule
{\bf XSum} & {$5\times 10^{-4}$} & 64 & 25 & 0.1 & 6000 & 100 & 50000
\\
{\bf CNN/DailyMail} & $ 5\times 10^{-4} $ & 32 & 15 & 0.1 & 5000 & 100 & 85000
\\
\bottomrule
\end{tabular}
F. Ablation Study for LoRA
As mentioned in Section 4, we find that the performance of LoRA can be further improved when applying it to every weight matrix, compared to fine-tuning $ W_{q} $ and $ W_{v} $ only ([16]). This observation aligns with the empirical results of [11]. In Table 14, we follow the same training configuration as Section 4.1 and present an ablation study to illustrate this point.
: Table 14: We compare the fine-tuning performance when apply LoRA to every weight matrix or $W_q, W_v$ only. The parameter budget is fixed as 0.3M. We report accuracy for QQP and MRPC, accuracy(m) for MNLI, and average correlation for STS-B.
| MNLI | QQP | CoLA | RTE | QNLI | SST-2 | MRPC | STS-B | |
|---|---|---|---|---|---|---|---|---|
| LoRA ($W_q, W_k$) | 89.80 | 90.48 | 67.04 | 83.75 | 93.69 | 94.84 | 90.20 | 91.05 |
| LoRA (all) | 90.30 | 91.61 | 68.71 | 85.56 | 94.31 | 94.95 | 90.44 | 91.68 |
G. Orthogonal Regularization
To verify the effectiveness of Equation (4), we plot $\lVert P^{\top}P-I \rVert_{\sf F}^2$ and $\lVert Q Q^{\top}-I \rVert_{\sf F}^2$ to show whether $P$ and $Q$ are regularized to be orthogonal. We fine-tune a DeBERTaV3-base model on SST-2 with AdaLoRA and follow the same training configuration as Section 4.1. We set $\gamma$ as 0.1 and plot the two terms along the training horizon. From Figure 4, we can see that two regularization terms can be optimized to a very small value (e.g., 0.001) at the beginning of training. Therefore, both $P$ and $Q$ can be enforced to be orthogonal quickly during the initial warm-up of AdaLoRA. It ensures that the triplets are not dependent with each other.
H. Comparison of Training Cost
We compare the training cost between AdaLoRA and LoRA in the following table. We use two methods to fine-tune DeBERTaV3-base on a single NVIDIA V100 GPU. We do training only and set hyperparameters, e.g., batch size and training epochs, the same as in Section 4.
\begin{tabular}{c|c|ccc}
\toprule
{\bf Dataset} & {\bf # Param} & {\bf Method} & {\bf GPU Mem} & {\bf Time/epoch}
\\
\midrule
\multirow{6}*{\bf MNLI}
& \multirow{2}*{\bf 0.08\%} & LoRA & 11.094 GB & 105 min
\\
& & AdaLoRA & 11.104 GB & 116 min
\\
& \multirow{2}*{\bf 0.16\%} & LoRA & 11.098 GB & 105 min
\\
& & AdaLoRA & 11.110 GB & 117 min
\\
& \multirow{2}*{\bf 0.65\%} & LoRA & 11.128 GB & 105 min
\\
& & AdaLoRA & 11.188 GB & 117 min
\\
\midrule
\multirow{6}*{\bf SST-2}
& \multirow{2}*{\bf 0.08\%} & LoRA & 13.138 GB & 60 min
\\
& & AdaLoRA & 13.148 GB & 71 min
\\
& \multirow{2}*{\bf 0.16\%} & LoRA & 13.142 GB & 61 min
\\
& & AdaLoRA & 13.164 GB & 71 min
\\
& \multirow{2}*{\bf 0.65\%} & LoRA & 13.170 GB & 61 min
\\
& & AdaLoRA & 13.226 GB & 71 min
\\
\bottomrule
\end{tabular}
Table 15 shows that AdaLoRA incurs 11% additional training time on MNLI and 16% on SQuADv2 under different budgets. The memory footprint of two methods are quite close. Such results demonstrate that AdaLoRA does not incur significant training overheads. The reason behind is that we only evaluate the importance score for small incremental matrices $P \Lambda Q$. Their total number of parameters is usually less than 1% of pre-trained weights. Therefore, it does not lead to significant computational cost to update the importance scores of these well-structured small matrices, compared to forward-backward pass of full model.
References
Section Summary: This section consists of a numbered bibliography listing academic papers, conference proceedings, and preprints primarily from the fields of natural language processing and machine learning. The entries cover foundational work on large language models such as BERT and GPT, along with studies on parameter-efficient adaptation techniques like adapters, prompt tuning, and low-rank methods, as well as related mathematical approaches to model optimization and sparsity. These references provide the scholarly sources supporting the article's discussions and methods.
[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi:10.18653/v1/N19-1423.
[2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
[3] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021b.
[4] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[6] Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872–1897, 2020.
[7] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020.
[8] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790–2799. PMLR, 2019.
[9] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
[10] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247, 2020.
[11] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=0RDcd5Axok.
[12] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 4582–4597. Association for Computational Linguistics, 2021. doi:10.18653/v1/2021.acl-long.353. URL https://doi.org/10.18653/v1/2021.acl-long.353.
[13] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.emnlp-main.243. URL https://aclanthology.org/2021.emnlp-main.243.
[14] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021.
[15] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463, 2020.
[16] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.
[17] Jian-Feng Cai, Emmanuel J Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on optimization, 20(4):1956–1982, 2010.
[18] Vladimir Koltchinskii, Karim Lounici, and Alexandre B Tsybakov. Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. The Annals of Statistics, 39(5):2302–2329, 2011.
[19] Kim-Chuan Toh and Sangwoon Yun. An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pacific Journal of optimization, 6(615-640):15, 2010.
[20] Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. 2020.
[21] Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, and Weizhu Chen. Super tickets in pre-trained language models: From model compression to improving generalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6524–6538, Online, 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.acl-long.510.
[22] Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference on Machine Learning, pp. 26809–26823. PMLR, 2022.
[23] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021a.
[24] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[25] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, Austin, Texas, 2016. Association for Computational Linguistics. doi:10.18653/v1/D16-1264.
[26] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789, Melbourne, Australia, 2018. Association for Computational Linguistics. doi:10.18653/v1/P18-2124.
[27] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
[28] Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.
[29] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
[30] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[31] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 11264–11272. Computer Vision Foundation / IEEE, 2019. doi:10.1109/CVPR.2019.01152.
[32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024–8035, 2019.
[33] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. ArXiv preprint, abs/1910.03771, 2019.
[34] Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522, 2020.
[35] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[36] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74–81, 2004.