VeRA: Vector-based Random Matrix Adaptation

mboxVeRA: Vector-based Random Matrix Adaptation

Dawid J. Kopiczko$^{2}$ $^{*}$ QUVA Lab University of Amsterdam

Tijmen Blankevoort Qualcomm AI Research$^1$

Yuki M. Asano QUVA Lab University of Amsterdam

$^{*}$ [email protected]; $^1$Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.

Abstract

Low-rank adapation (LoRA) is a popular method that reduces the number of trainable parameters when finetuning large language models, but still faces acute storage challenges when scaling to even larger models or deploying numerous per-user or per-task adapted models. In this work, we present Vector-based Random Matrix Adaptation (VeRA), which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. It achieves this by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead. We demonstrate its effectiveness on the GLUE and E2E benchmarks, image classification tasks, and show its application in instruction-tuning of 7B and 13B language models.

Executive Summary: VeRA (Vector-based Random Matrix Adaptation) addresses the growing storage and memory burden of adapting very large language and vision models to many users or tasks. Standard full fine-tuning stores a complete copy of the model for each adaptation, while even efficient methods like LoRA still require tens to hundreds of megabytes per adapted model. At the scale of personalized assistants or per-task deployments, this quickly reaches hundreds of terabytes, limiting practical deployment on GPUs or edge devices.

The work evaluates whether a new parameterization can match LoRA’s downstream performance while reducing the number of trainable parameters by roughly an order of magnitude. The method freezes a single pair of random matrices shared across all layers and instead learns two small scaling vectors per layer; these vectors modulate the shared matrices without ever storing them explicitly, since they can be regenerated from a seed.

Experiments span the GLUE and E2E benchmarks with RoBERTa and GPT-2 models, instruction tuning of 7 B and 13 B Llama models, and image classification with Vision Transformers on four datasets. Results are reported against full fine-tuning, BitFit, multiple adapter variants, and LoRA under matched hyper-parameter regimes, with five random seeds per configuration.

Across these settings, VeRA delivers comparable or better task performance while using 3–100× fewer trainable parameters. On GLUE with RoBERTa-large it matches LoRA at roughly 1/10 the parameter count; on E2E it outperforms LoRA with 3–4× fewer parameters; on Llama instruction tuning it stays within 0.3–0.5 MT-Bench points of LoRA despite a ~100× reduction (1.6 M versus 160 M parameters for the 7 B model). Similar patterns hold for vision tasks, and scaling curves show higher accuracy per stored parameter than LoRA at every budget examined.

These findings matter because storage and loading time become the dominant bottleneck when hundreds or thousands of adapted models must coexist in memory. VeRA makes it feasible to keep many specialized versions resident on a single GPU, eliminates extra inference latency, and modestly lowers training memory. The approach therefore directly improves serving economics for personalized or multi-task deployments.

Organizations should adopt VeRA for any scenario that stores numerous adapted checkpoints, especially where GPU memory is constrained. Where maximum accuracy on a single high-value task is paramount, a modest increase in rank or hybrid use with LoRA on the final layers can be explored as a next step. Further gains are likely from dynamic rank allocation or refined initialization, both identified as immediate follow-on work.

The results rest on standard academic benchmarks and models up to 13 B parameters; extension to other architectures and larger scales remains untested, and a few large GLUE tasks were omitted for compute reasons. Reported performance differences are consistent across seeds and comparable to or better than prior LoRA baselines.

1. Introduction

Section Summary: Large language models are powerful but extremely memory-intensive to customize for different users or tasks, since storing separate fine-tuned versions of models like GPT-4 quickly becomes impractical. Existing efficient methods such as LoRA still add substantial storage costs, yet research on low-dimensional features and random projections suggests far leaner approaches are possible. The authors introduce VeRA, which freezes shared random matrices and learns only lightweight scaling vectors, cutting the number of trainable parameters well below prior techniques while maintaining performance and imposing no extra cost at inference time.

In the era of increasingly large and complex language models, the challenge of efficient adaptation for specific tasks has become more important than ever. While these models provide powerful capabilities, their extensive memory requirements pose a significant bottleneck, particularly when adapting them for personalized use. Consider, for example, a cloud-based operating system assistant that continuously learns from and adapts to individual user behaviors and feedback. The need to store multiple checkpoints of finetuned models for each user rapidly escalates the required storage, even more so when multiple tasks come into play.

The situation is further exacerbated when we look at the state-of-the-art models like GPT-4 ([1]). Finetuning techniques like LoRA ([2]), while effective, still introduce considerable memory overhead. As an illustrative example, applying LoRA with a rank of 16 to the query and value layers of GPT-3 ([3]) would demand at least 288MB of memory, if stored in singe-precision – at a million finetuned weights, e.g., one per user, that would amount to 275TB.

Given the recent proliferation of language models and their deployment in personalized assistants, edge devices, and similar applications, efficient adaptation methods are paramount. We believe there is untapped potential for even more efficient approaches. Previous work ([4]) pointed out the low intrinsic dimensionality of pretrained models' features. These studies reported numbers much lower than the trainable parameters used in LoRA, suggesting there is room for improvement.

In parallel to this, recent research has shown the surprising effectiveness of models utilizing random weights and projections ([5, 6, 7, 8, 9]). Such models serve as the basis of our proposed solution, Vector-based Random Matrix Adaptation (VeRA), which minimizes the number of trainable parameters introduced during finetuning by reparametrizing the weights matrices. Specifically, we employ "scaling vectors" to adapt a pair of frozen random matrices shared between layers. With this approach, many more versions of the model can reside in the limited memory of a single GPU.

In summary, our main contributions are as follows:

We introduce a novel finetuning method with no additional inference time cost. Our method further reduces the number of trainable parameters compared to the state-of-the-art LoRA method, while yielding comparable results.
We compare our approach with LoRA and other parameter-efficient adaptation methods on the natural language understanding (GLUE) and natural language generation (E2E) benchmarks, and compare against LoRA on instruction-following and image classification tasks.
We perform an ablation study to better understand the individual components of our method and their effects on performance.

2. Related Work

Section Summary: Low-Rank Adaptation (LoRA) and its extension AdaLoRA provide efficient ways to finetune large language models by using low-rank matrices to approximate weight changes, which cuts down on trainable parameters, reduces hardware demands, and avoids extra costs at inference time. Despite these gains, such methods still employ far more parameters than necessary according to studies on the intrinsic dimensions of models, leaving room for greater efficiency with only minor trade-offs in performance. Supporting research on random neural networks and projections shows that frozen random matrices and small trainable subnetworks can often achieve strong results, laying groundwork for new parameter-saving finetuning strategies.

Low-Rank Adaptation (LoRA).

LoRA offers an innovative solution to the computational challenges posed by the finetuning of large pretrained language models. Introduced by [2], the method employs low-rank matrices to approximate the weight changes during finetuning, effectively reducing the number of parameters that need to be trained. Among its advantages, LoRA significantly lowers the hardware barrier for finetuning by reducing the need for gradient calculation and optimizer state maintenance for most parameters. It can also work with quantized model weights ([10]), reducing the requirements even further. Furthermore, LoRA modules are easily swappable, making task-switching efficient and less resource-intensive. Importantly, and different to adapter-based finetuning approaches ([11, 12, 13, 14]), LoRA incurs no additional inference time cost when deployed, as the trainable matrices can be merged with the frozen weights.

Based on this, AdaLoRA ([15]) extends the LoRA method, introducing dynamic rank adjustment for the low-rank matrices during finetuning. The core idea is to optimally distribute the parameter budget by selectively pruning less important components of the matrices based on an importance metric.

Parameter Efficiency in Existing Methods

While methods such as LoRA have shown significant improvements in finetuning performance, they still require a considerable amount of trainable parameters. According to [4], the upper bound for intrinsic dimensions is much smaller than what is typically utilized in such methods. For instance, the $d_{90}$ [^1] for RoBERTa $_\text{base}$ is reported to be $896$, whereas authors of the LoRA paper reported using $0.3$ M trainable parameters for this model, suggesting that the parameter count could be reduced further.

[^1]: The smallest dimension $d$ that provides a satisfactory solution, which is 90% of the full training metric, as defined by [16].

Although AdaLoRA takes steps in this direction by dynamically allocating parameters to more critical layers, we posit that a different approach could achieve substantial parameter reduction, while tolerating a marginal performance degradation. This sets the stage for the method we introduce in the following section.

Random Models and Projections.

The concept of using random matrices and projections for model efficiency is supported by multiple strands of research. [17] identified that randomly-initialized neural networks contain subnetworks that are capable of reaching high performance when trained. Meanwhile, [6] revealed that there exist subnetworks that can achieve impressive results even in the absence of training. [4] showed that training only a small number of parameters, randomly projected back into the full space, could achieve 90% of the full-parameter model performance. [18] introduced a parameter-efficient finetuning method for personalization of text-to-image models, utilising random frozen matrices inside LoRA. Other works ([7, 8, 9]) have shown that frozen, randomly initialized models, with small sections finetuned, can perform surprisingly well.

Collectively, these works create a compelling case for the utilization of frozen random matrices in finetuning methods, providing both a theoretical and an empirical foundation for the approach taken in this paper.

3. Method

Section Summary: VeRA is a parameter-efficient fine-tuning technique that improves on LoRA by freezing a single pair of randomly initialized low-rank matrices and sharing them across all layers of a model, then adapting each layer only through small trainable scaling vectors. This change reduces the number of parameters that must be learned and stored—often by orders of magnitude—while the frozen matrices can be recreated from a random seed and the learned vectors can be merged back into the original weights with no added inference cost. The method also specifies initialization choices that keep training stable across different ranks without per-rank hyperparameter tuning.

In this section, we introduce Vector-based Random Matrix Adaptation, a novel parameter-efficient finetuning method that builds upon and extends the state-of-the-art method, LoRA. The central innovation in VeRA lies in the reparameterization of the low-rank matrices. Specifically, we freeze a single pair of randomly initialized matrices, shared across all adapted layers, and introduce trainable scaling vectors that allow for layer-wise adaptation, as shown in Figure 1. Similarly to LoRA, trained scaling vectors along with low-rank matrices can be merged into original weights, eliminating additional inference latency.

3.1 Method Formulation

LoRA ([2]) finetunes a matrix product of two low-rank matrices to adapt large-language models for a new task. Formally, for a pretrained weight matrix $W_0 \in \mathbb{R}^{m \times n}$, the weight update $ \Delta W $ is constrained to a low-rank decomposition, as expressed in Equation 1

$ h = W_0x + \Delta W x = W_0x + {\underline{B A}} x,\tag{1} $

where we undeline the parameters updated via gradient descent. This approximation enables the model to keep the original weight $ W_0 $ frozen while optimizing only the new low-rank matrices $ A $ and $ B $ . These matrices are much smaller in size than the original matrix due to their rank-reduced nature. $ A $ has shape $m \times r$ and $ B $ has shape $r \times n$, where $r \ll \min(m, n)$ serves as the bottleneck dimension. In contrast, our VeRA method is expressed as:

$ h = W_0x + \Delta W x = W_0x + {\underline{\Lambda_{b}}} B {\underline{\Lambda_{d}}} A x\tag{2} $

In this approach, $B$ and $A$ are frozen, random, and shared across layers, while the scaling vectors $b$ and $d$ are trainable, and formally denoted by diagonal matrices $\Lambda_b$ and $\Lambda_d$ . This approach can effectively scale and disable rows and columns of both $A$ and $B$, allowing for layer-wise adaptation with a minimal number of trainable parameters. Note that in this setup, $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times n}$ are not required to be low-rank. This is because they remain static and we do not need to store their values. Instead, varying $r$ leads to a linear increase in the number of trainable parameters via $d \in \mathbb{R}^{1 \times r}$.

3.2 Parameter Count

\begin{tabular}{l c | r r | r r}
\toprule
& & \multicolumn{2}{c|}{LoRA} & \multicolumn{2}{c}{VeRA} \\
& Rank & # Trainable Parameters & Required Bytes & # Trainable Parameters & Required Bytes \\
\midrule 
\parbox[t]{4mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{\textsc{Base}}}} 
& 1 & 36.8K & 144KB & 18.4K & 72KB \\
& 16 & 589.8K & 2MB & 18.8K & 74KB \\
& 256 & 9437.1K & 36MB & 24.5K & 96KB \\
\midrule
\parbox[t]{4mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{\textsc{Large}}}} 
& 1 & 98.3K & 384KB & 49.2K & 192KB \\
& 16 & 1572.8K & 6MB & 49.5K & 195KB \\
& 256 & 25165.8K & 96MB & 61.4K & 240KB \\
\midrule
\parbox[t]{4mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{\textsc{GPT-3}}}} 
& 1 & 4.7M & 18MB & 2.4M & 9.1MB \\
& 16 & 75.5M & 288MB & 2.8M & 10.5MB \\
& 256 & 1207.9M & 4.6GB & 8.7M & 33MB \\
\bottomrule
\end{tabular}

We use $ L_{\text{tuned}} $ to denote the number of finetuned layers and $ d_{\text{model}} $ to represent the dimension of these layers. The number of trainable parameters in VeRA is then governed by $ |\Theta| = L_{\text{tuned}} \times (d_{\text{model}} + r) $, contrasting with LoRA's $ |\Theta| = 2 \times L_{\text{tuned}} \times d_{\text{model}} \times r $ . Specifically, for the lowest rank (i.e., $ r=1 $), VeRA requires approximately half the trainable parameters of LoRA. Moreover, as the rank increases, VeRA's parameter count increases by $ L_{\text{tuned}} $ for each increment, a substantial saving compared to LoRA's $ 2L_{\text{tuned}}d_{\text{model}} $ . This parameter efficiency becomes notably significant in the context of extremely deep and wide models, such as GPT-3 ([3]), which has 96 attention layers and a hidden size of 12288.

Building on this efficiency, the main advantage of VeRA is its minimal memory footprint for storing the trained weight adjustments. Because the random frozen matrices can be regenerated from a random number generator (RNG) seed, these do not need to be stored in memory. This substantially reduces the memory requirement, which is now limited to the bytes needed for the trained $b$ and $d$ vectors and a single RNG seed. The memory efficiency in comparison to LoRA is shown in Table 1.

3.3 Initialization Strategies

Shared Matrices: In our method, we employ Kaiming initialization ([19]) for the frozen low-rank matrices $ A $ and $ B $ . By scaling the values based on matrix dimensions, it ensures that a matrix product of $ A $ and $ B $ maintains a consistent variance for all ranks, eliminating the need to finetune the learning rate for each rank.
Scaling Vectors: The scaling vector $ b $ is initialized to zeros, which aligns with the initialization of matrix $ B $ in LoRA and ensures that the weight matrix is unaffected during the first forward pass. The scaling vector $ d $ is initialized with a single non-zero value across all its elements, thereby introducing a new hyperparameter that may be tuned for better performance.

Figure 1 illustrates example initializations for the low-rank matrices and scaling vectors in VeRA. Specifically, the low-rank matrices are initialized using a normal distribution, and the $d$ vector is initialized with ones. Note that alternative initializations, such as uniform distribution for $A$ and $B$, and other non-zero constants for $d$, are also explored in our experiments.

4. Experiments

Section Summary: The experiments section evaluates the VeRA finetuning method through comparisons against full finetuning, BitFit, various adapter approaches, and LoRA on the GLUE and E2E benchmarks using models such as RoBERTa and GPT-2. Additional tests apply VeRA to instruction tuning of Llama models and image classification with Vision Transformers, while also examining how performance changes when varying the rank and number of trainable parameters, plus an ablation study of the method's components. Results show VeRA matching or exceeding LoRA's performance on these tasks while using substantially fewer parameters, such as an order of magnitude reduction on GLUE and 3-4 times fewer on E2E.

In this section, we conduct a series of experiments to evaluate our finetuning method. We start by comparing our approach to LoRA and other baselines on the GLUE and E2E benchmarks. Following this, we turn our attention to instruction-tuning of Llama models, and image classification with Vision Transformers. Next, we select one task and vary the rank for both methods, LoRA and VeRA, to examine how performance scales with the number of trainable parameters. Lastly, an ablation study sheds light on the importance of each component in our method, including the influence of different initializations.

Baselines.

We compare VeRA to the following baselines:

Full finetuning - the model is initialized with pretrained weights and all parameters are being trained.
Bitfit - this baseline involves the sole finetuning of bias vectors, keeping all other parameters fixed. This technique has been investigated in depth by [20].
Adapter tuning - initially introduced by [11], involves the integration of adapter layers between the self-attention and MLP modules, followed by a residual connection. This setup includes two fully connected layers and a nonlinearity and is denoted as $\textbf{Adapter}^{\textbf{H}}$. A variation by [12], $\textbf{Adapter}^{\textbf{L}}$, employs the adapter layer solely after the MLP module and subsequent to a LayerNorm. This closely resembles an alternative design suggested by [13], referred to as $\textbf{Adapter}^{\textbf{P}}$. Another baseline, termed AdapterDrop by [14], enhances efficiency by omitting certain adapter layers and is represented as $\textbf{Adapter}^{\textbf{D}}$.
LoRA ([2]) - as introduced in the earlier section.

4.1 GLUE Benchmark

We evaluate our approach on the General Language Understanding Evaluation (GLUE) benchmark ([21]), employing the RoBERTa $\text{base}$ and RoBERTa $\text{large}$ models ([22]). For RoBERTa $\text{base}$ we use a rank of 1024, and for RoBERTa $\text{large}$ a rank of 256. The shared matrices are initialized using the uniform version of Kaiming initialization as implemented in PyTorch ([23]), with an initial value of 0.1 for the $ d $ vector.

Our experimental setup generally aligns with that of [2], applying our method to the query and value projection matrices in each self-attention module and fully training the classification head. Unlike [2], who used an additional hyperparameter $\alpha$ to adjust gradients for the adapted layers, we introduce separate learning rates for the classification head and the adapted layers. We determine the learning rates and the number of training epochs through hyperparameter tuning; for detailed settings, refer to the Table 8 in Appendix A. The batch size is set to 64 for RoBERTa $\text{base}$ and 32 for RoBERTa $\text{large}$, with maximum sequence lengths of 512 and 128 respectively.

Due to time constraints and budget limitations, we omit the time-intensive MNLI and QQP tasks, thus forgoing the use of the MNLI trick[^2] for tasks MRPC, RTE, and STS-B. In line with [2], we report the number of trainable parameters attributable to the finetuned layers, explicitly excluding the classification head, which is trained in a standard way. We perform 5 runs with different random seeds, recording the best epoch's outcome for each run, and report the median of these results.

[^2]: For the RoBERTa $_\text{base}$ model and MRPC, RTE and STS-B tasks, [2] initialized the model with the best weights finetuned on the MNLI task.

Results.

::: {caption="Table 2: Results for different adaptation methods on the GLUE benchmark. We report Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for the remaining tasks. In all cases, higher values indicate better performance. Results of all methods except VeRA are sourced from prior work ([2, 24]). VeRA performs on par with LoRA with an order of magnitude fewer parameters."}

:::

Table 2 reveals that VeRA performs competitively with LoRA across both models, yet achieves these results with an order of magnitude fewer parameters.

4.2 E2E Benchmark

For the E2E benchmark ([25]), we follow the experimental setup from [2] and finetune the GPT-2 ([26]) Medium and Large models. For LoRA we use the implementation and set of hyperparameters provided in [2], while for VeRA we change the rank and learning rate, both of which are tuned. Table with all hyperparameters used can be found in Appendix A.

Results.

::: {caption="Table 3: Results for different adaptation methods on the E2E benchmark and GPT2 Medium and Large models. Results with (^1, 2, 3) are taken from prior work: ^1([2]), ^2([27]), ^3([28]). VeRA outperforms LoRA with 3 and 4 times less trainable parameters, for GPT2 Medium and Large respectively."}

:::

We report results from the last epoch. Table 3 shows that VeRA outperforms LoRA with 3 and 4 times less trainable parameters, for GPT2 Medium and Large respectively.

4.3 Instruction tuning

Instruction tuning is a process by which language models are finetuned to follow specific instructions more effectively ([29]). We demonstrate the efficacy of VeRA in enabling Llama ([30]) and Llama2 ([31]) models to follow instructions using only $1.6$ M and $2.4$ M trainable parameters, for 7B and 13B variants respectively, in contrast to $159.9$ M and $250.3$ M trainable parameters when employing LoRA with a rank of 64 as proposed by [10].

We perform finetuning using both LoRA and VeRA, by applying both methods on all linear layers except the top one, similarly to [10]. Additionally, we leverage the quantization techniques from [10] to train the model on a single GPU.

For our experiment, we employ the Alpaca dataset ([32]), specifically its cleaned version^3. This dataset comprises 51K instructions and demonstrations and is suitable for instruction-tuning. The cleaned version corrects multiple issues such as hallucinations, merged instructions, and empty outputs. We train for one epoch, preceded by a learning rate sweep.

We evaluate finetuned models on MT-Bench ([33]), by generating model responses to a pre-defined set of 80 multi-turn questions and subsequently evaluating these using GPT-4 ([1]). GPT-4 reviews the answers and assigns a quantitative score on a scale of 10 to each response. We present the average scores alongside the number of trainable parameters in Table 4.

\begin{tabular}{l | l | r | c}
\toprule
Model & Method & # Parameters & Score \\
\midrule
Llama 13B & - & - & 2.61 \\
\midrule
\multirow{2}{*}{\textsc{Llama 7B}}
 & LoRA & 159.9M & 5.03 \\
 & VeRA & 1.6M & 4.77 \\
\midrule
\multirow{2}{*}{\textsc{Llama 13B}}
 & LoRA & 250.3M & 5.31 \\
 & VeRA & 2.4M & 5.22 \\
\midrule
\multirow{2}{*}{\textsc{Llama2 7B}}
 & LoRA & 159.9M & 5.19 \\
 & VeRA & 1.6M & 5.08 \\
\midrule
\multirow{2}{*}{\textsc{Llama2 13B}}
 & LoRA & 250.3M & 5.77 \\
 & VeRA & 2.4M & 5.93 \\
\midrule
\end{tabular}

We find that despite the 100x reduction in the number of trainable parameters, our method closely matches the performance of LoRA-based finetuning.

4.4 Image Classification

To evaluate the method on the image classification task, we adapt Vision Transformer (ViT) ([34]), Base and Large variants, on datasets - CIFAR100 ([35]), Food101 ([36]), Flowers102 ([37]), and RESISC45 ([38]). For each dataset we train on a subset of 10 samples per class, and evaluate on the full test set (CIFAR100, Food101, Flowers102) or on all the remaining samples (RESISC45). We use weights of ViT models pretrained on the ImageNet-21k ([39]) dataset.

We evaluated LoRA and VeRA methods applied on the query and value layers of ViT, along with two baselines - fully-finetuned model (referred to as Full), and training the classification head only (referred to as Head). Similarly to the GLUE benchmark, we use rank 8 for LoRA, and rank 256 for VeRA. We tuned learning rates for all methods and reported results after 10 epochs in Table 5. The reported parameter count excludes the classification head, which has to be trained in all methods.

::: {caption="Table 5: Vision models finetuned with VeRA and LoRA on different image classification datasets. VeRA approaches performance of LoRA for the smaller model, and outperforms it in the case of the large model, with over 10x fewer trainable parameters."}

:::

We find that VeRA approaches performance of LoRA on the Base model for three datasets and outperforms it for Flowers102, despite using over 10x fewer trainable parameters. For ViT-Large, it outperforms LoRA for three datasets: CIFAR100, Flowers102 and RESISC45.

4.5 Scaling the Number of Trainable Parameters

Finally, we investigate the trade-offs involved in parameter scalability for both LoRA and our method using the RoBERTa $_\text{large}$ model on the RTE task from the GLUE benchmark. We use a set of ranks $r={1, 4, 16, 64, 256, 1024}$ for VeRA and $r={1, 2, 4, 8, 16, 32, 64}$ for LoRA, and observe the trade-off between trainable parameters and the accuracy. We replicate each configuration five times for different random seeds, and report the median of results. For LoRA, we employ the HuggingFace PEFT ([40]) implementation, adhering to the hyperparameters specified in [2]. Our own method uses the same hyperparameters as employed in the RTE experiments from the previous subsection. The results, depicted in Figure 2, reveal that our method is significantly more parameter-efficient. Notably, when the higher-rank VeRA has the same number of parameters as standard LoRA, it outperforms LoRA by 4 accuracy percentage points.

**Figure 2:** Performance of LoRA and VeRA methods for varying ranks on RTE task.

**Figure 3:** Magnitude of the adapted $d$ vector for query and value matrices across layers for RoBERTa-L on the RTE task.

4.6 Ablation Study

In this section, we conduct an ablation study to examine the impact of individual components of our method. All subsequent experiments focus on the MRPC and RTE tasks and utilize the RoBERTa $_\text{large}$ model. We adhere to the hyperparameters used in previous experiments, modifying only the component under investigation for each test. Each experiment is run with 5 random seeds, and we report the mean and standard deviation of the results.

::: {caption="Table 6: Ablation study results for the impact of the d and b scaling vectors and different initialization strategies. Our default settings are highlighted with blue color."}

:::

\begin{tabular}{c | llll}
  \toprule
  Random Matrices & MRPC & RTE & CoLA & STS-B \\
  \midrule 
  Shared & $90.0_{\pm0.9}$ & $\textbf{84.6}_{\pm1.5}$ & $67.7_{\pm0.8}$ & $\textbf{91.5}_{\pm0.6}$ \\
  Unique & $\textbf{90.7}_{\pm0.3}$ & $\textbf{84.6}_{\pm0.8}$ & $\textbf{68.3}_{\pm1.8}$ & $\textbf{91.5}_{\pm0.2}$ \\
  \bottomrule
  \end{tabular}

Single Scaling Vector

We first investigate the necessity of both the $d$ and $b$ scaling vectors in our method. We create two ablation setups: one that excludes $d$ (termed as only $\boldsymbol{b}$) and another that omits $b$ (termed as only $\boldsymbol{d}$). In the only $\boldsymbol{d}$ setup, $d$ is initialized with zeros. As shown in Table 6, omitting either scaling vector compromises performance. The only $\boldsymbol{d}$ configuration performs slightly better than its only $\boldsymbol{b}$ counterpart. This disparity in performance underscores the higher expressiveness of the $d$ scaling vector over the $b$ vector. Specifically, $d$ modulates the rows of both low-rank matrices, thereby influencing a broader aspect of the final constructed matrix. In contrast, $b$ only scales the rows of the final matrix resulting from the product of the low-rank matrices.

Initialization of Shared Matrices

We examine three different initialization schemes for the shared matrices: Kaiming normal, Kaiming uniform, and uniform initialization within the range $[0, 0.1]$ . As per the results in Table 6, both Kaiming initializations outperform the uniform range initialization, with uniform variant having slightly better results than the normal one.

Initialization of Scaling Vector

We further explore the impact of the initialization values for the $d$ vector. Experiments are conducted with $d_{\text{init}}$ set at $1.0$, $10^{-1}$, and $10^{-7}$ . The results in Table 6 show that the choice of $d_{\text{init}}$ significantly influences the method's performance; in the settings we examined, values $10^{-1}$ and $10^{-7}$ outperformed $1.0$, potentially offering more flexibility in the optimization process through early sign changes in selected rows of the frozen matrices.

Magnitude of Adaptation

In Figure 3 we provide a visualisation of the magnitude of the changes of the $d$ vectors after finetuning on RTE task. Because the low-rank frozen matrices remain the same for each layer, we can directly compare the length of the $d$ vector across layers to account for its relative adaptation. Overall, we find that the largest adaptation happens for query matrices compared to the value ones, indicating a larger need or ease for finetuning a model there. Furthermore, similar to previous efficient adaptation methods' findings ([15, 41]) we also observe a higher adaptation for the later layers compared to earlier ones.

Sharing Random Matrices

We conduct experiments on RTE, MRPC, CoLA, and STS-B tasks to assess the impact of sharing random matrices on the performance. We evaluate two setups - one with random matrices shared across all adapted layers, and another with uniquely generated ones. Results in Table 7 show that the mean performance is identical in case of tasks RTE and STS-B, and there is a slight improvement for MRPC and CoLA when using unique matrices.

5. Conclusion

Section Summary: This paper presents a finetuning method that cuts the number of adjustable parameters by up to ten times compared with LoRA, while delivering equal or better results on language, vision, and text-generation tasks. Its compact design makes it especially useful for cloud services that must keep many personalized models ready in limited GPU memory, improving efficiency and reducing loading delays. Future work could test the approach on other model types and refine it through smarter ways of assigning parameters.

In this work, we introduce a finetuning method that significantly reduces the number of trainable parameters compared to LoRA, yielding similar or better results on downstream tasks. Specifically, it achieved ten-fold reduction in parameters yielding the same performance on the GLUE benchmark for RoBERTa $_\text{large}$, ten-fold reduction on image classification tasks, and three-fold reduction on the E2E benchmark. This method is particularly well-suited for scenarios that require frequent swapping of numerous finetuned models, such as cloud-based AI services personalized for individual users. Due to the minimal size of the scaling vectors, many versions can reside in the limited memory of a single GPU, thus substantially improving serving efficiency and removing the bottleneck of loading specific models into memory.

While the current study focuses on language and vision models with Transformer architecture, the applicability of the method across different architectures and domains remains an area for future research. Moreover, the performance of the method may benefit from additional refinements, such as dynamic parameter budget allocation, or different initialization and regularization techniques.

Acknowledgements

This work is financially supported by Qualcomm Technologies Inc., the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy. We also acknowledge the use of the National Supercomputer Snellius and Distributed ASCI Supercomputer 6 ([42]) for essential computational tasks.

Appendix

Section Summary: The appendix details the specific hyperparameter settings used for training models on the GLUE benchmark and for instruction-tuning experiments. It lists configurations such as optimizer type, learning rates, number of epochs, batch sizes, and ranks for both base and large models across various tasks, while noting that choices were constrained by limited computing resources and partly drawn from prior LoRA setups rather than exhaustive searches. A second table compares settings between LoRA and VeRA for the instruction-tuning stage.

A. Hyperparameters

\begin{tabular}{ll | cccccc}
\toprule
Model & Hyperparameter & SST-2 & MRPC & CoLA & QNLI & RTE & STS-B \\
\midrule
& Optimizer & \multicolumn{6}{c}{AdamW} \\
& Warmup Ratio & \multicolumn{6}{c}{0.06} \\
& LR Schedule & \multicolumn{6}{c}{Linear} \\
& Init. of Shared Matrices & \multicolumn{6}{c}{Kaiming Uniform} \\
& Initial Value of $d$ & \multicolumn{6}{c}{0.1} \\
\midrule
\parbox[t]{4mm}{\multirow{7}{*}{\rotatebox[origin=c]{90}{\textsc{Base}}}}
& # GPUs & \multicolumn{6}{c}{1} \\
& VeRA Rank & \multicolumn{6}{c}{1024} \\
& Epochs & 60 & 30 & 80 & 25 & 160 & 80 \\
& Learning Rate (Head) & 4E-3 & 4E-3 & 1E-2 & 4E-3 & 1E-2 & 1E-2 \\
& Learning Rate (VeRA) & 4E-3 & 1E-2 & 1E-2 & 1E-2 & 4E-3 & 1E-2 \\
& Max Seq. Len. & \multicolumn{6}{c}{512} \\
& Batch Size Per GPU & \multicolumn{6}{c}{64} \\
\midrule
\parbox[t]{4mm}{\multirow{7}{*}{\rotatebox[origin=c]{90}{\textsc{Large}}}}
& # GPUs & \multicolumn{6}{c}{4} \\
& VeRA Rank & \multicolumn{6}{c}{256} \\
& Epochs & 10 & 40 & 40 & 20 & 40 & 20 \\
& Learning Rate (Head) & 6E-3 & 3E-3 & 6E-3 & 2E-4 & 2E-3 & 2E-3 \\
& Learning Rate (VeRA) & 1E-2 & 3E-2 & 1E-2 & 1E-2 & 2E-2 & 2E-2 \\
& Max Seq. Len. & \multicolumn{6}{c}{128} \\
& Batch Size Per GPU & \multicolumn{6}{c}{32} \\
\bottomrule
\end{tabular}

In Table 8, we provide the hyperparameters used for the GLUE benchmark in the main paper. Note that due to our academic compute we were not able to run full grid searches on any hyperparameters. We only evaluated different learning rates and number of epochs and even relied on existing configurations of LoRA (Optimizer, Warmup ratio, LR schedule).

\begin{tabular}{l | cc}
\toprule
Hyperparameter & LoRA & VeRA \\
\midrule
## GPUs & \multicolumn{2}{c}{1} \\

> **Section Summary**: The provided section lists detailed hyperparameter settings for fine-tuning ViT models on image-classification tasks, including a shared setup of one GPU, the AdamW optimizer, a linear schedule, weight decay of 0.01, and separate learning rates for the classification head versus the rest of the network. Subsequent subsections report supporting experiments that quantify VeRA’s gains in parameter efficiency, modest trade-offs in training time and memory versus LoRA, and the similarity of weights learned by the two methods. Additional results cover expressivity tests and instruction-tuning performance on LLaMA with the Vicuna evaluation protocol.

Optimizer & \multicolumn{2}{c}{AdamW} \\
Warmup Ratio & \multicolumn{2}{c}{0.1} \\
Batch Size & \multicolumn{2}{c}{4} \\
Accumulation Steps & \multicolumn{2}{c}{4} \\
Epochs & \multicolumn{2}{c}{1} \\
LR Schedule & \multicolumn{2}{c}{Cosine} \\
Rank & 64 & 1024 \\
Learning Rate & 4E-4 & 4E-3 \\
\bottomrule
\end{tabular}

\begin{tabular}{l | cc}
\toprule
Hyperparameter & Medium & Large \\
\midrule
## GPUs & \multicolumn{2}{c}{1} \\

> **Section Summary**: The provided section lists detailed hyperparameter settings for fine-tuning ViT models on image-classification tasks, including a shared setup of one GPU, the AdamW optimizer, a linear schedule, weight decay of 0.01, and separate learning rates for the classification head versus the rest of the network. Subsequent subsections report supporting experiments that quantify VeRA’s gains in parameter efficiency, modest trade-offs in training time and memory versus LoRA, and the similarity of weights learned by the two methods. Additional results cover expressivity tests and instruction-tuning performance on LLaMA with the Vicuna evaluation protocol.

Optimizer & \multicolumn{2}{c}{AdamW} \\
Learning Rate Schedule & \multicolumn{2}{c}{Linear} \\
Weight Decay & \multicolumn{2}{c}{0.01} \\
Batch Size & \multicolumn{2}{c}{8} \\
Epochs & \multicolumn{2}{c}{5} \\
Warmup Steps & \multicolumn{2}{c}{500} \\
Label Smooth & \multicolumn{2}{c}{0.1} \\
Rank & \multicolumn{2}{c}{1024} \\
Learning Rate & 1E-1 & 2E-2 \\
\bottomrule
\end{tabular}

\begin{tabular}{ll | cccc}
\toprule
Model & Hyperparameter & CIFAR100 & Food101 & Flowers102 & RESISC45 \\
\midrule
& # GPUs & \multicolumn{4}{c}{1} \\
& Optimizer & \multicolumn{4}{c}{AdamW} \\
& LR Schedule & \multicolumn{4}{c}{Linear} \\
& Weight Decay & \multicolumn{4}{c}{0.01} \\
& VeRA Rank & \multicolumn{4}{c}{256} \\
& LoRA Rank & \multicolumn{4}{c}{8} \\
\midrule
\parbox[t]{4mm}{\multirow{7}{*}{\rotatebox[origin=c]{90}{\textsc{Base}}}}
& LR-Head (Head) & 4E-3 & 4E-3 & 4E-3 & 4E-2 \\
& LR (Full) & 4E-5 & 4E-5 & 4E-5 & 8E-5 \\
& LR-Head (Full) & 4E-3 & 4E-2 & 4E-3 & 4E-3 \\
& LR (VeRA) & 2E-2 & 4E-2 & 4E-2 & 7E-2\\
& LR-Head (VeRA) & 4E-3 & 4E-2 & 4E-3 & 5E-3 \\
& LR (LoRA) & 4E-3 & 4E-3 & 4E-3 & 4E-3 \\
& LR-Head (LoRA) & 4E-3 & 4E-3 & 4E-3 & 4E-3\\
\midrule
\parbox[t]{4mm}{\multirow{7}{*}{\rotatebox[origin=c]{90}{\textsc{Large}}}}
& LR-Head (Head) & 4E-4 & 4E-3 & 4E-3 & 4E-3 \\
& LR (Full) & 4E-5 & 4E-5 & 4E-5 & 8E-5 \\
& LR-Head (Full) & 4E-3 & 4E-3 & 8E-3 & 4E-3 \\
& LR (VeRA) & 4E-2 & 4E-2 & 4E-2 & 7E-2\\
& LR-Head (VeRA) & 2E-3 & 2E-3 & 2E-3 & 3E-3 \\
& LR (LoRA) & 4E-3 & 4E-3 & 4E-3 & 4E-3 \\
& LR-Head (LoRA) & 4E-3 & 4E-3 & 4E-3 & 4E-4\\
\bottomrule
\end{tabular}

B. Relative performance gain.

$**Figure 4:** Performance gains per 1K trainable parameters on the RTE task for RoBERTa $_\text{large}$ model relative to the baseline. Formula: $ (\text{accuracy}_{\text{method}} / \text{accuracy}_{\text{baseline}}) / \text{parameters}_{\text{method}} * 100$$

Figure 4 quantifies the efficiency of each method in terms of performance gains per 1K trainable parameters. For a focused comparison, we select the RTE task and RoBERTa $_\text{large}$ model.

To establish a baseline, we conduct auxiliary experiments where only the classification head is trained while the remainder of the model is frozen. This baseline is constructed using the same hyperparameters as in our VeRA method. We then evaluate the performance gain attributable to each method, normalized by the additional trainable parameters introduced, relative to the baseline. The results clearly show that VeRA yields the highest performance gain per 1K trainable parameters.

C. Impact on training time and memory usage

To evaluate the training time and GPU memory benefits of our method, we conducted a comparison between LoRA and VeRA while fine-tuning LLaMA 7B with the same rank (64) on instruction tuning dataset, introduced earlier in this work. The results are summarized in Table 12:

: Table 12: Impact on GPU memory usage and training time.

Method	Training Time	GPU Memory
LoRA	568 min	23.42GB
VeRA	578 min	21.69GB

While VeRA includes more operations than LoRA because of the additional vector multiplies in the forward pass, we find that it only results in a modest 1.8% increase in training time. For the GPU memory, we observe a 7.4% reduction in memory usage with VeRA, as it does not require storing optimizer states and gradients for shared random matrices.

D. Similarities of trained weights

We compared the weights trained with LoRA and VeRA at a single rank of 64 across all query layers. For each method and adapted layer, we constructed a weight difference. In LoRA’s case, this involved the multiplication of two low-rank matrices, while for VeRA, it also included multiplication by scaling vectors. We then calculated the cosine similarity of these flattened weights. Additionally, we compared the similarity between trained LoRA weights and randomly initialized matrices as a baseline: We find that similarities of VeRA to LoRA are on average 2e-3 while LoRA to random matrices is -8e-5.

**Figure 5:** Cosine similarity of LoRA, VeRA, and random weights across layers.

In Figure 5 we can see a notable increase in similarity between the trained weights, particularly in the latter layers. This observation aligns with our earlier findings (Figure 3) that the highest adaptation occurs in these layers. These results support the notion that VeRA can approximate the weights trained with LoRA.

E. Expressivity of VeRA

We conducted an experiment on the expressivity of LoRA and VeRA on the task of fitting random square 10x10 matrices, with results seen in Figure 6. For given number of trainable parameters, both methods perform equally well, with VeRA providing more flexibility, e.g. by allowing for much lower parametrization - below LoRA’s rank 1.

**Figure 6:** MSE loss on the task of fitting 10x10 random matrix. LoRA (PQ) and VeRA (XdYb) perform equally well for given number of trainable parameters.

F. Instruction-tuning with Vicuna Eval

Results and samples from evaluation of instruction tuned Llama 7B model with Vicuna Eval ([43]), predecessor of MT-Bench. The model has been finetuned on a 10K subset of cleaned Alpaca dataset.

: Table 13: Average scores assigned by GPT-4 to the answers generated by models fine-tuned with VeRA and LoRA methods. Number of trainable parameters and theoretical memory required to store them in single-precision are also shown.

Method	No. Parameters	Required Bytes	Avg. Score
LoRA	159.9M	610MB	7.36
VeRA	1.4M	5.4MB	7.48

::: caption="Table 14"

:::

::: caption="Table 15"

:::

::: caption="Table 16"

:::

::: caption="Table 17"

:::

References

Section Summary: This section provides a numbered bibliography of 16 academic papers, technical reports, and preprints primarily from the fields of machine learning and natural language processing. The works cited focus on large language models such as GPT-4, parameter-efficient fine-tuning methods like LoRA and adapters, and related concepts including intrinsic dimensionality and random feature techniques. Most references originate from conferences including NeurIPS, ICLR, ACL, and CVPR, with publication dates between 2019 and 2023.

[1] OpenAI. Gpt-4 technical report, 2023.

[2] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.

[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877–1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

[4] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7319–7328, Online, August 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.acl-long.568. URL https://aclanthology.org/2021.acl-long.568.

[5] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=QtTKTdVrFBB.

[6] V. Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari. What’s hidden in a randomly weighted neural network? In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11890–11899, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society. doi:10.1109/CVPR42600.2020.01191. URL https://doi.ieeecomputersociety.org/10.1109/CVPR42600.2020.01191.

[7] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. Proceedings of the AAAI Conference on Artificial Intelligence, 36(7):7628–7636, Jun. 2022. doi:10.1609/aaai.v36i7.20729. URL https://ojs.aaai.org/index.php/AAAI/article/view/20729.

[8] Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e2105646118, 2021. doi:10.1073/pnas.2105646118. URL https://www.pnas.org/doi/abs/10.1073/pnas.2105646118.

[9] Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in cnns. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=vYeQQ29Tbvx.

[10] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.

[11] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019.

[12] Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficient transfer learning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 441–459, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.findings-emnlp.41. URL https://aclanthology.org/2020.findings-emnlp.41.

[13] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487–503, Online, April 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.eacl-main.39. URL https://aclanthology.org/2021.eacl-main.39.

[14] Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7930–7946, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.emnlp-main.626. URL https://aclanthology.org/2021.emnlp-main.626.

[15] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=lq62uWRJjiY.

[16] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=ryup8-WCW.

[17] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.

[18] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023.

[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034, 2015. doi:10.1109/ICCV.2015.123.

[20] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2022.

[21] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJ4km2R5t7.

[22] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.

[23] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. $d'$ Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf.

[24] Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303, 2023a.

[25] Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 201–206, Saarbrücken, Germany, August 2017. Association for Computational Linguistics. doi:10.18653/v1/W17-5525. URL https://aclanthology.org/W17-5525.

[26] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

[27] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558, 2022.

[28] Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411, 2023.

[29] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=TG8KACxEON.

[30] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.

[31] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.

[32] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.

[33] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.

[34] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.

[35] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

[36] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In European Conference on Computer Vision, 2014.

[37] M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.

[38] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, 2017.

[39] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009. doi:10.1109/CVPR.2009.5206848.

[40] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022.

[41] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021.

[42] H. Bal, D. Epema, C. de Laat, R. van Nieuwpoort, J. Romein, F. Seinstra, C. Snoek, and H. Wijshoff. A medium-scale distributed system for computer science research: Infrastructure for the long term. Computer, 49(05):54–63, may 2016. ISSN 1558-0814. doi:10.1109/MC.2016.127.

[43] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.