GLU Variants Improve Transformer — Read with an AI tutor

GLU Variants Improve Transformer

Abstract

Gated Linear Units [1] consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer [2] sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.

Executive Summary: The paper examines how to improve the core feed-forward layers inside Transformer models used for language tasks. These layers currently rely on simple nonlinear activations such as ReLU or GELU. The authors test whether replacing them with gated linear units and several close variants can raise model quality without increasing parameter count or computation.

The work set out to measure the effect of these gated variants on both pre-training perplexity and downstream task performance. The authors ran controlled experiments inside the T5 text-to-text framework, keeping total parameters and operations fixed by shrinking the hidden dimension of the new layers by one-third.

They trained identical base-size encoder-decoder models on the C4 span-filling task for up to 524k steps, then fine-tuned each once on a combined mixture of GLUE, SuperGLUE, and SQuAD. Four short runs per architecture quantified run-to-run variability, and results were compared against the standard ReLU baseline.

GEGLU and SwiGLU produced the clearest gains. At 65k steps they lowered held-out perplexity by roughly 0.05 points relative to ReLU; at full training the gap remained about 0.04 points. On the downstream suite these two variants also led on most individual tasks, although the margins were smaller and noisier. Bilinear and ReGLU showed modest but consistent improvements as well, while plain GELU and Swish offered little or no benefit.

The results indicate that a simple change to the feed-forward sub-layer can raise both language-modeling quality and transfer performance at no extra cost. Because the modification requires only standard matrix multiplies and element-wise operations, it can be adopted immediately in existing codebases.

Teams training large language models should therefore replace the current ReLU or GELU feed-forward blocks with GEGLU or SwiGLU. The next practical step is to verify the same gains at larger model scales and on additional domains such as code or multilingual data. The main limitation is that all measurements come from a single base-size model trained on English web text; wider confirmation would increase confidence before broad deployment.

1. Introduction

Section Summary: The Transformer processes input sequences by alternating attention layers with position-wise feed-forward networks. Each network applies two linear transformations to a vector, separated by a nonlinearity such as ReLU, though some implementations omit bias terms. Later work has tested other activation functions, including GELU and Swish, in place of ReLU within the same two-layer structure.

The Transformer ([2]) sequence-to-sequence model alternates between multi-head attention, and what it calls "position-wise feed-forward networks" (FFN). The FFN takes a vector $x$ (the hidden representation at a particular position in the sequence) and passes it through two learned linear transformations, (represented by the matrices $W_1$ and $W_2$ and bias vectors $b_1$ and $b_2$). A rectified-linear (ReLU) ([3]) activation function applied between the two linear transformations.

$ \textrm{FFN}(x, W_1, W_2, b_1, b_2) = \textrm{max}(0, xW_1 + b_1)W_2 + b_2\tag{1} $

Following the T5 codebase ([4]) [^1], we use a version with no bias:

[^1]: Also in the interest of ML fairness.

$ \textrm{FFN}_\textrm{ReLU}(x, W_1, W_2) = \textrm{max}(xW_1, 0)W_2\tag{2} $

Subsequent work has proposed replacing the ReLU with other nonlinear activation functions such as Gaussian Error Linear Units, $\textrm{GELU}(x)=x\Phi(x)$ ([5]), and $\textrm{Swish}_\beta(x)=x\sigma(\beta x)$ ([6]).

$ \begin{split} \textrm{FFN}\textrm{GELU}(x, W_1, W_2) & = \textrm{GELU}(xW_1)W_2 \ \textrm{FFN}\textrm{Swish}(x, W_1, W_2) & = \textrm{Swish}_1(xW_1)W_2 \end{split}\tag{3} $

2. Gated Linear Units (GLU) and Variants

Section Summary: Gated linear units apply a sigmoid function to one of two parallel linear projections of the input and multiply the results elementwise, while a related bilinear version skips the sigmoid. The section extends this idea to other activations such as ReLU, GELU, and Swish, producing the ReGLU, GEGLU, and SwiGLU variants. These gated operations are then substituted for the first linear layer and nonlinearity inside a Transformer feed-forward block; because each new layer contains three weight matrices instead of two, the hidden dimension is reduced by two-thirds to hold total parameter count and computation fixed.

([1]) introduced Gated Linear Units (GLU), a neural network layer defined as the component-wise product of two linear transformations of the input, one of which is sigmoid-activated. They also suggest omitting the activation, which they call a "bilinear" layer and attribute to ([7]).

$ \begin{split} \textrm{GLU}(x, W, V, b, c) & = \sigma(xW + b) \otimes (xV + c) \ \textrm{Bilinear}(x, W, V, b, c) & = (xW + b) \otimes (xV + c) \end{split}\tag{4} $

We can also define GLU variants using other activation functions:

$ \begin{split}\textrm{ReGLU}(x, W, V, b, c) & = \textrm{max}(0, xW + b) \otimes (xV + c)\\textrm{GEGLU}(x, W, V, b, c) & = \textrm{GELU}(xW + b) \otimes (xV + c)\\textrm{SwiGLU}(x, W, V, b, c, \beta) & = \textrm{Swish}_\beta(xW + b) \otimes (xV + c)\end{split}\tag{5} $

In this paper, we propose additional variations on the Transformer FFN layer which use GLU or one of its variants in place of the first linear transformation and the activation function. Again, we omit the bias terms.

$ \begin{split}\textrm{FFN}\textrm{GLU}(x, W, V, W_2) & = (\sigma(xW) \otimes xV)W_2\\textrm{FFN}\textrm{Bilinear}(x, W, V, W_2) & = (xW \otimes xV)W_2\\textrm{FFN}\textrm{ReGLU}(x, W, V, W_2) & = (\textrm{max}(0, xW) \otimes xV)W_2\\textrm{FFN}\textrm{GEGLU}(x, W, V, W_2) & = (\textrm{GELU}(xW) \otimes xV)W_2\\textrm{FFN}_\textrm{SwiGLU}(x, W, V, W_2) & = (\textrm{Swish}_1(xW) \otimes xV)W_2\end{split}\tag{6} $

All of these layers have three weight matrices, as opposed to two for the original FFN. To keep the number of parameters and the amount of computation constant, we reduce the number of hidden units $d_{ff}$ (the second dimension of $W$ and $V$ and the first dimension of $W_2$) by a factor of $\frac{2}{3}$ when comparing these layers to the original two-matrix version.

3. Experiments on Text-to-Text Transfer Transformer (T5)

Section Summary: The experiments evaluate several modified feed-forward layers inside a standard T5-style encoder-decoder transformer that is first pre-trained to fill in deleted text spans from the C4 corpus and then fine-tuned on question-answering and language-understanding tasks. To keep parameter counts identical, the hidden size of the gated-linear-unit variants is reduced from 3072 to 2048. On a held-out perplexity measure the GEGLU and SwiGLU versions perform best, and after fine-tuning on GLUE, SuperGLUE and SQuAD they also tend to outperform the ReLU baseline on most individual tasks.

We test the FFN variants we have described on the transfer-learning setup from ([4]). An encoder-decoder transformer model ([2]) is trained on a denoising objective of predicting missing text segments, and subsequently fine-tuned on various language understanding tasks.

3.1 Model Architecture

We use the same code base, model architecture, and training task as the base model from ([4]). The encoder and decoder each consist of 12 layers, with $d_{model}=768$. For the attention layers, $h=12$ and $d_k = d_v = 64$. The FFN layers have hidden size $d_{ff}=3072$. As we describe above, for the GLU-variant-based FFN layers, which have thee weight matrices instead of two, we reduce the hidden layer to $d_{ff}=2048$, so as to maintain the same parameter and operation counts as the base model.

: Table 1: Heldout-set log-perplexity for Transformer models on the segment-filling task from ([4]). All models are matched for parameters and computation.

Training Steps	65, 536	524, 288
FFN_ReLU (baseline)	1.997 (0.005)	1.677
FFN_GELU	1.983 (0.005)	1.679
FFN_Swish	1.994 (0.003)	1.683
FFN_GLU	1.982 (0.006)	1.663
FFN_Bilinear	1.960 (0.005)	1.648
FFN_GEGLU	1.942 (0.004)	1.633
FFN_SwiGLU	1.944 (0.010)	1.636
FFN_ReGLU	1.953 (0.003)	1.645

3.2 Pre-Training and Perplexity Results

Identically to ([4]), we pre-train for 524, 288 steps on the span-filling objective on the C4 dataset. Each training batch consists of 128 examples, each of which has an input of 512 tokens and an output of 114 tokens, the output containing multiple spans of tokens which were deleted from the input[^2]. Similarly to ([4]), we use the Adafactor optimizer ([8]) and an inverse-square-root learning-rate schedule. We also decay the learning rate linearly for the final 10 percent of the training steps. Our main departure from ([4]) is that we use no dropout during pre-training. We find this to produce superior results. We compute the log-perplexity on the training objective on a heldout shard of C4, which we believe to be a good indicator of model quality. For each model architecture, we also trained four models for a shorter period (65, 536 steps) to measure inter-run variability. The results are listed in Table 1. The GEGLU and SwiGLU variants produce the best perplexities.

[^2]: Each training step took approximately 0.15 seconds on a 32-core TPUv2 cluster.

3.3 Fine-Tuning

We then fine-tune each fully-trained model once on an examples-proportional mixture of the Stanford Question-Answering Dataset (SQuAD) ([9]) and all the language understanding tasks in the GLUE ([10]) and SuperGlue ([11]) benchmarks.[^3] Fine-tuning consists of 131072 steps with a learning rate of $10^{-3}$. As in training, the input sequences for each step have a combined length of approximately 65, 536 tokens. Following ([4]), we use a dropout rate of $0.1$ on the layer outputs, feed-forward hidden-layers and attention weights. The embedding matrices are fixed during fine-tuning.

[^3]: This departs from ([4]), who fine-tuned separately on the different tasks. We chose one fine-tuning run for simplicity.

Table 2, Table 3 and Table 4 show results on the development sets. For each task, we report the best score of any of the checkpoints recorded during fine-tuning. While the results are noisy, the new GLU-variants perform best on most of the tasks. For comparison, at the bottom of each of the tables we list the reuslts from ([4]). The model is identical to our $\textrm{FFN}_\textrm{ReLU}$ model. Their results are notably worse, which we believe was caused by their use of dropout during pre-training. Also listed are the inter-run standard deviations measured by ([4]).

::: {caption="Table 2: GLUE Language-Understanding Benchmark ([10]) (dev)."}

:::

::: {caption="Table 3: SuperGLUE Language-Understanding Benchmark ([11]) (dev)."}

:::

::: {caption="Table 4: SQuAD ([9]) v1.1 (dev)."}

:::

4. Conclusions

Section Summary: Researchers have developed new variants of certain neural network layers and integrated them into Transformer models for language tasks. In experiments involving pre-training followed by transfer to other problems, these changes produced stronger results on both the initial training objective and a range of downstream language-understanding benchmarks, while remaining simple to add and free of extra computational cost. The authors note that they have no technical explanation for the improvements and instead credit their success to divine benevolence.

We have extended the GLU family of layers and proposed their use in Transformer. In a transfer-learning setup, the new variants seem to produce better perplexities for the de-noising objective used in pre-training, as well as better results on many downstream language-understanding tasks. These architectures are simple to implement, and have no apparent computational drawbacks. We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.

References

Section Summary: This references section consists of a numbered list of eleven academic papers and preprints drawn from machine learning and natural language processing research. The cited works address neural network architectures such as transformers and gated convolutions, along with activation functions, optimization techniques, and standard benchmarks for evaluating language understanding models. Most date from 2016 to 2019 and were published in venues like NeurIPS or on arXiv.

[1] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. CoRR, abs/1612.08083, 2016. URL http://arxiv.org/abs/1612.08083.

[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.

[3] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315–323, 2011.

[4] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.

[5] Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, 2016. URL http://arxiv.org/abs/1606.08415.

[6] Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.

[7] Andriy Mnih and Geoffrey Hinton. Three new graphical models for statistical language modelling. In Proceedings of the 24th international conference on Machine learning, pages 641–648, 2007.

[8] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235, 2018.

[9] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.

[10] Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.

[11] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019.