Recirculation

Michael C. Mozer$^{*}$
Google DeepMind
[email protected]

Shoaib Ahmed Siddiqui$^{*}$
Google DeepMind
[email protected]

Danny Sawyer
Google DeepMind
[email protected]

Sunny Sanyal$^{\dagger}$
University of Texas, Austin
[email protected]

Rosanne Liu
Google DeepMind
[email protected]

Abstract

We describe an inference-time architectural enhancement for off-the-shelf foundation models that markedly reduces perplexity and boosts accuracy across generation and reasoning tasks. Our approach incurs essentially no additional latency during generation, though it requires serial processing in the prefill phase. Motivated by the fundamental limitation that state updates in feedforward transformers are bounded by model depth, our technique, recirculation, introduces a specific form of recurrence that allows the model to act as a dynamical system and track belief states. We distinguish this technique from chain-of-thought computation—which is better reserved for complex inferences rather than basic state tracking—as well as from popular depth-recurrence techniques (looping) and the costly training of recurrent transformers. We also propose and evaluate an adaptive variant of recirculation which requires only light tuning of hyperparameters while freezing the original model weights. Relative to the off-the-shelf baseline, adaptive recirculation achieves remarkable gains on the Gemma3 family, including a 23% reduction in perplexity on a suite of datasets, a 21% increase in accuracy on GSM8k, and reliable improvements in accuracy on other downstream tasks. Our training-free approach succeeds by leveraging the model itself to inform architectural modifications, suggesting a route to architectural evolution guided by a trained network's properties rather than forced, arbitrary design choices.

$^{*}$ Shared first authorship. Our thanks to Lei Bao, Mukul Bhutani, and Lan Le who contributed to the shaping of this research.
$^{\dagger}$ Work performed while a student researcher at DeepMind.

Executive Summary: Recirculation is a lightweight inference-time change to standard transformer models that improves their ability to track evolving information across a sequence. The core problem is that transformers process text in a single forward pass through fixed layers, which limits how well they can maintain and update internal state. This leads to coherence failures in conversations, errors on contextual word meanings, and weaker performance on reasoning tasks that require working memory.

The authors set out to test whether a simple form of recurrence—leaking a controlled amount of activation from deeper layers back to shallower ones—could address this limitation without retraining the model. They evaluated the approach on the Gemma3 family (1B, 4B, and 12B) and several other model families, using both language-modeling perplexity and downstream tasks. The method was tested in its basic form and in an adaptive variant that learns only the mixing coefficients while leaving the original model weights frozen.

On language-modeling benchmarks, recirculation produced consistent perplexity reductions, reaching 16–35% on the larger Gemma3 models across most datasets. Accuracy improved on instruction-following, contextual disambiguation, and multiple-choice tasks, with the largest gains on GSM8k math problems (roughly 21% relative error reduction with the adaptive version). The benefits were clearest on longer sequences and content words that carry state information. Adaptive recirculation delivered roughly triple the average perplexity gain of the fixed-coefficient version while requiring only modest tuning data.

These results matter because they show meaningful capability gains at essentially no extra generation cost. The approach strengthens the model’s basic ability to maintain context, which supports better performance on every task built on that foundation. It also demonstrates that useful architectural improvements can be discovered by examining a trained model’s own internal dynamics rather than imposing new structures through expensive retraining.

For immediate use, teams should adopt adaptive recirculation on Gemma-style models when generation latency is the primary constraint and state-tracking errors are observed. When prefill speed or very long contexts are critical, blockwise recirculation or further hyperparameter search will be needed first. Future work should determine whether task-independent hyperparameters exist, test the method on a wider range of architectures, and explore multiple recirculation paths and iteration counts. The main limitations are that optimal layer pairs and mixing strengths appear somewhat task- and model-family dependent, and prefill remains serial. Results on the Gemma3 family are robust; broader claims should be treated as preliminary until additional families are evaluated at scale.

1. State tracking

Section Summary: Transformers struggle to track an evolving situation because their parallel architecture prevents the kind of step-by-step updating that recurrent networks and similar systems perform. As a result, they often lose or misapply key details such as the intended meaning of an ambiguous word or the current state of a conversation, producing inconsistent or incoherent output. While the models can sometimes resolve these ambiguities in deeper layers, that information is not reliably available to earlier stages that generate responses, and simple fixes like copying resolved representations downward help only modestly.

The ability to track a fluid, evolving state of affairs is essential for comprehending language, reasoning about situations, and modeling the world around us. Traditional approaches to state tracking, such as recurrent neural nets and Kalman filters, involve iterative, sequential updating of latent variables to capture dynamics. The parallel operation of transformers, during both training and prefill, precludes tracking state in the traditional manner ([1]). Transformers learn a variety of clever solutions that can make them effective at state tracking for finite sequence lengths ([2, 3, 4, 5]). However, transformers can also fail to track state information needed to understand the structure of arguments and social interactions. State tracking failures in foundation models lead to loss of coherence in multi-turn conversations ([6]), inefficiency in information gathering ([7]), and breakdowns in communication and cooperation in multi-agent settings ([8, 9]). Tracking deficits can lead to a range of reasoning failures arising from ineffective working memory, unstable theory-of-mind representations, and absence of a robust internal world model ([10]).

Without proper state tracking, models can flip-flop in their interpretations and fail to detect their inconsistencies, e.g., the meaning of a polysemous word in Figure 1 ([11]), known as a contextualization error. While it is perfectly sensible for a model to maintain a belief state that admits both interpretations of the word bank, after committing to the edge-of-a-river interpretation (river bank), a socially adept communicator should acknowledge the reversal of meaning.

In general, it becomes untenable for models—and people—to maintain and track belief states because the distributions explode in dimensionality. People adopt heuristics such as sampling (e.g., [12]), collapsing distributions into prototypical cases ([13]), or forming concrete mental models that are most consistent with premises ([14]), kind of like a MAP estimate. All of these approaches nonetheless require state tracking, albeit without explicit representation of uncertainty.

Figure 2 a illustrates the challenge of state tracking in a transformer using the bank dialog of Figure 1. Input steps are presented along the horizontal axis and blocks (or layers) of the transformer along the vertical axis. Each column represents the processing of an input token, where activation flows from bottom to top. The processing of the text day off work and fishing pole is depicted in columns on the left. Using a technique called Patchscopes ([15]), [11] show that when the polysemous word bank is processed, shallow layers reflect the word's ambiguity: the embedding is a mixture of its meanings—river edge and financial institution. In deeper layers of the stack (highlighted in yellow), the embedding is contextualized by the preceding text (depicted by the blue arrows), and the context-appropriate interpretation of the polysemous word is selected by nudging the embedding toward the river-edge representation. Because the architecture is feedforward, shallow layers of the transformer cannot access this interpretation. Thus, when the model is asked to formulate a response to the ATM question, the initial processing stages see only the ambiguous representation of bank. If the model commits to a response (highlighted yellow) using the ambiguous representation (blue arrows), it will choose the wrong response. Despite having previously interpreted bank as the river edge, it fails to access this state and responds yes based on a strong association between the ambiguous bank and ATM. [11] characterize this delayed disambiguation as a race in which the model's response generation can outpace the model's internal semantic convergence.

[11] obtained evidence in support of this account via an intervention experiment, illustrated in Figure 2 b. They processed the input sequence in the ordinary way up to the step at which the bank token is presented. At this step, they copied activation from a deep layer—after the ambiguity was resolved—down to a shallow layer and then continued processing the input sequence. This intervention, depicted in the Figure with an arrow, reduced contextualization errors by 60%.[^1]

[^1]: The full experiment was based on a large set of generated questions containing a critical token that needs to be properly contextualized in order to answer correctly, with activation patching on only the critical token. [16] have also shown that patching visual token activations back to a shallower layer in a vision-language model improves performances.

In this experiment, [11] replaced the embedding vector in a shallow layer with the corresponding vector from a deep layer, but only for one specific token pre-identified as critical in the given context. Suppose that instead of targeting this manipulation to a specific token, we did so in an undifferentiated fashion at every token position? Likely the outcome would be disastrous because the model was not trained to accommodate the resulting amplification of feedback. But what if instead of substituting one embedding vector for another, we merely leaked a small bit of activation from the deep layer to the shallow layer? The leakage might conceivably be sufficient to enrich the representation without pushing representations out of distribution. Consequently, we might observe benefits of this manipulation at inference without modifying the weights of a fully trained model.

This scheme might work even without fine tuning because a transformer's residual stream acts like a shared blackboard onto which all layers can write, which encourages alignment of representations across layers ([17]). By alignment, we mean that for a deep layer to communicate with a shallow layer, a 1:1 correspondence of features can be assumed and we do not require an arbitrary adapter such as a full-rank affine transformation or cross attention.

Why should there be alignment? Consider a particular feature of the embedding vector in the residual stream, and suppose that it is associated with a semantically meaningful concept such as moisture. Whether the feature is activated early or late in the stack, the direct effect on the output distribution is identical due to commutativity of addition. Some input tokens, e.g., pool or tears, may activate the moisture feature in the input embedding directly. An ambiguous token such as bank may not yield much intrinsic activation for moisture due to polysemy. However, if we feed back the disambiguated bank from a deep layer, the moisture feature will be amplified, providing useful information for subsequent processing.[^2]

[^2]: This argument does not deny that the layer of origin of a feature may change the feature's role, and in fact, the argument for leaking activation downwards is to leverage the changing roles. Layer normalization is one way that roles might depend on depth. However, while layer normalization may change a feature's magnitude and even polarity, it does not reorient the feature. One might also be concerned about feature erasing: a feature may be 'uncomputed' once it is no longer needed. This concern is alleviated by recent work showing that transformers learn just as well, if not better, when each layer is forced to output orthogonal directions from the earlier layers, preventing feature erasing ([18, 19]).

2. Proposed method: Recirculation

Section Summary: The proposed recirculation method runs a transformer step by step while periodically copying a portion of the activation from a deeper layer back into a shallower one, allowing information to flow both downward in depth and forward across successive inputs. This differs from a looped transformer, which simply reuses the same blocks multiple times within a single forward pass; recirculation instead merges operations across both depth and time steps so that the same layer can hold and update an evolving internal state. The result supports more flexible state tracking than weight-shared feed-forward stacks, at the cost of requiring strictly sequential computation even when the full input sequence is available.

We now formalize our method, which we refer to as recirculation, that involves running an LLM step-by-step and after each step, leaking a bit of the activation from a deep layer down to a shallow layer. Figure 3 a gives the general picture, where the recurrent arrow specifies one possible source- and destination-layer pair. Because this Figure does not specify how the recurrence is orchestrated with regard to input steps, the figure is ambiguous and could correspond to several distinct ideas ([1]), one of which is recirculation and another of which is a very popular idea in the literature, the looped transformer ([20, 21]). For didactic purposes, we first discuss looped transformers and then characterize their relationship to recirculation.

A looped transformer is a parameter-efficient variant of the standard architecture. Whereas a standard transformer stacks a deep sequence of unique transformer blocks, a looped transformer applies a set of shared blocks multiple times. Figure 3 b depicts a looped transformer by unrolling the architecture of Figure 3 a vertically in depth and unrolled horizontally in input steps. At step 1, the first input token is presented, the activation stack is computed, and at layer 6 in the Figure (the loop source), activation is passed to layer 3 (the loop destination) of a second copy of the architecture, which then propagates activity to the output. The colored rectangles denote the current input token, the dark outlined rectangles are blocks whose activation is computed at a given step, the shaded blocks are frozen (or are replaced by KV cache), and the faint outlined rectangles are placeholders that are irrelevant at the current step. As each input step is processed, a single stack operates and the earlier stacks are frozen (or in KV cache). The Figure shows step-by-step operation of the model, as would be used with autoregressive decoding. However, when an input sequence is fixed—as during pretraining or in the prefill stage—the entire sequence can be computed in parallel.

Recurrence in a looped transformer is solely in depth: the arrows in Figure 3 b are directed within a stack. In contrast, recurrence in recirculation is in both depth and step, as depicted in Figure 3 c. Figure 3 c can be viewed as collapsing together the operation of Figure 3 b's top stack at step $i$ and the bottom stack of step $i+1$. In recirculation, two input stacks are run in parallel at each recurrence step (except for the very first step, which serves as a warm up). The difference between Figure 3 b and Figure 3 c looks to be a minor reorganization, but it fundamentally changes the nature of the computation when it comes to arbitrary state tracking. Figure 4 shows the same unrolled models but superimposes colored rectangles to indicate state propagation. To implement an arbitrary state updating function over time $t$, $z(t+1) = f(z(t), x(t))$, where $z$ is state and $x$ is input, $z(t+1)$ must be one layer deeper in the looped transformer, which is just a deeper feedforward transformer with weight-sharing constraints. However, in recirculation, where there is projection in both depth and step, the same layer can hold both $z(t)$ and $z(t+1)$. The cost of state tracking is that sequential passes must be made through the architecture, and because of this sequential updating, recirculation cannot be parallelized, even when an entire input sequence is provided, such as during prefill.

In Figure 3 and Figure 4, the model is unrolled to execute the transformer stack twice at each step, i.e., one more iteration than a standard transformer. This number of iterations can be increased, both for looping and recirculation. We depict two-iteration recirculation in Figure 14. However, all experiments reported in this article are with the one-additional-iteration variant. If the number of iterations is unbounded, recirculation behaves as a true recurrent neural net.

We note one other key difference between looping and recirculation. In looping, the activation that gets looped is the entire residual stream and it acts as the direct replacement for the input that would normally come from a preceding layer. In recirculation, activation is mixed from the source and destination layers. Initially, we formalize one-iteration recirculation as a mixture:

$ {\boldsymbol z}{t+1, t, d} = \alpha f({\boldsymbol z}{t, t, s} | d, t) + \beta {\boldsymbol z}_{t, t, d},\tag{1} $

where $t$ is an index over the order of updates, $s$ and $d$ are the source and destination layer indices, respectively, $\alpha$ and $\beta$ are mixture coefficients, $f(.)$ is a renormalization function, and $\boldsymbol{z_{i, j, l}}$ is the residual stream output after incorporating the computation of layer $l$ at unrolling step $i$ and input step $j$. These three indices correspond to the three sequence dimensions depicted in Figure 3 c: $l$ and $j$ are the row and column indices of the transformer layer grid, respectively, and $i$ is an index over copies of architecture. The motivation for the renormalization function $f$ is to accommodate the possibility that embedding magnitudes grow as the layer outputs are combined. We always use a convex mixture with $\beta \equiv 1-\alpha$ and rescale source to have the same $L_2$ norm as the destination unless mentioned otherwise:

$ f({\boldsymbol z} | d, t) = \frac{||{\boldsymbol z}_{t, t, d} ||_2}{||{\boldsymbol z}||_2} ~{\boldsymbol z}.\tag{2} $

3. Related research

Section Summary: Researchers have explored various ways to enhance transformers for tasks requiring sequential state tracking, such as language modeling or planning, where standard models struggle due to limited depth and parallelism. Approaches include looped transformers that reuse layers for greater effective depth, recurrent or state-space models that maintain hidden states across steps, and chain-of-thought methods that allow models to iteratively refine their own outputs. Other work focuses on specialized training losses to encourage stateful representations, along with activation steering techniques that intervene on internal signals, of which recirculation is presented as a form of self-guided inference-time adjustment.

Looped transformers. Looped transformers run single layers or a range of layers multiple times, obtaining a deeper architecture with no addition in free parameters ([20, 21]). Looping, which can be deterministic or adaptive, is a very popular and successful approach. Looping can increase the expressivity of a transformer ([22]), but it does not guarantee indefinite state tracking. Some methods are designed and trained to allow for inference time scaling (e.g., [23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34]); others incorporate recurrence via pretraining ([35]) or fine tuning a pretrained model ([36, 37]); and surprisingly, several operate purely as an inference-time method to improve reasoning ([38, 39, 40]). The inference-time methods have the greatest similarity to recirculation, although the notion of looping (greater depth) is conceptually distinct from the recurrence that occurs in recirculation (see Figure 3 and Figure 4).

Training objectives. Training losses have been proposed that aim to make embeddings in a given layer of the transformer more stateful, i.e., interpretable in terms of state updating functions. Particular losses have been proposed to steer models toward such solutions, to the extent they exist exactly or approximately ([41, 42, 43]).

State tracking. [44] point to the weakness of modern massively parallel architectures on problems that are inherently sequential, problems where combinatorics make it impractical to parallelize, such as state tracking, multihop inference, and planning. Transformers are bounded in their serial capacity based on model depth (e.g., [45, 46, 47]) and also by the fact that effectively utilizing the state representation becomes more challenging as it shifts upwards to deeper layers ([48, 11, 1, 7, 49]). [47] prove the necessity and sufficiency of $\log n$ layers to recognize regular language strings of up to length $n$ and graph-connectivity problems with $n$ vertices. However, this proof addresses only the constructability of solutions, not their learnability. In practice, many researchers have identified clever solutions obtained by training depth-limited models on specific finite sequence-length problems ([2, 3, 4, 5]).

Recurrent transformers. Models with sequential recurrent updates can express arbitrary state dynamics, $z_t = f(z_{t-1}, x_t)$. Some of these models operate with token-by-token recurrence (e.g., [50]), but most operate with blockwise recurrence ([51, 52, 53, 54, 55, 56]). State-space models (SSMs) (e.g., [57, 58, 59, 60, 61, 62, 63, 64]) are often touted as a means of state propagation, but many SSMs are no more expressive than an ordinary transformer ([65]) and none are as expressive as a generic recurrent net.

Thinking models. One solution to the depth dilemma is chain-of-thought style "thinking" where a model can talk to itself by sequentially sending signals from deep in the transformer to shallow layers, thereby propagating state forward. This form of recurrence enhances model expressivity ([66, 67]). Thinking can be performed in natural language tokens or in latent space (e.g., [68, 69]). As with other recurrent transformers, training thinking models can be costly because it restricts parallelism.

Activation steering. Recent work in activation steering demonstrates that a language model's behavior can be predictably modulated by intervening on its latent representations ([70, 71, 72]). This representation space encodes complex behavioral directions, including those governing truthfulness and refusal ([73, 74]). In this context, recirculation can be viewed as an inference-time mechanism for self-steering. Rather than modifying the residual stream with a static, externally derived steering vector ([75]), recirculation leverages the model's own contextualized deep-layer activations to guide representation trajectories in shallower layers.

4. Experiments

Section Summary: Experiments explored the use of recirculation in Gemma3 language models ranging from 1B to 12B parameters by systematically testing different source and destination layers along with mixing strengths on datasets such as arXiv, PG-19, and C4. Researchers identified layer pairs and settings that lowered perplexity relative to baseline models, achieving average reductions around 5% during tuning and as much as 35% on some evaluation sets, though gains were weaker for very short sequences. Supporting adjustments such as embedding normalization and gradual ramping of the mixing strength helped stabilize and extend these improvements across model sizes and ten language-modeling benchmarks.

4.1 Hyperparameter sweeps

To explore the feasibility of recirculation, we begin by sweeping over the three hyperparameters of recirculation: the mixture coefficient $\alpha$ (with $\beta=1-\alpha$ unless mentioned otherwise), the source layer $s$, and the destination layer $d$. We incorporate recirculation into the Gemma3 1B PT (pretrained) model and compute perplexity for documents from the arXiv dataset. (Details of the simulation can be found in Appendix B.1.) The four heatmaps of Figure 5 correspond to $\alpha \in {0.04, 0.07, 0.10, 0.16}$; the vertical and horizontal axes indicate the source and destination layers. We examine all source-destination pairs that are no more than 12 layers apart. The heatmap is coded blue-to-red to indicate perplexity lower-to-higher than a baseline no-recirculation Gemma3 1B model, which has perplexity 16.6 on this dataset. Notably, these heatmaps are fairly smooth and reveal systematic patterns. Increasing $\alpha$ amplifies the effect of recirculation but results in more source-destination pairs that harm perplexity. Layer 4 is desirable as a destination with a source 5-7 layers higher. For the moment, focus on the pattern of results and not on the magnitude of reduction in perplexity.

The top row of Figure 6, shows perplexity for three datasets—arXiv, PG19, and C4—sweeping over source and destination layers and fixing $\alpha=0.10$. We transform absolute perplexity estimates into percentage change relative to the baseline model and average across the three datasets to get the boxed sweep showing percentage change. Blue indicates a reduction in perplexity, with the best source-destination pair obtaining an average reduction of 4.72%. Similarly to the 1B model, we identify the source and destination layers for the 4B and 12B models that yield the largest mean percentage perplexity reduction on the roughly 1.5M tokens in our tuning set. For the 1B, 4B, and 12B Gemma3 models, we found the optimal source and destination pairs based on our tuning set to be: 11, 4, 18, 9, and 35, 16, respectively.

4.2 Perplexity evaluation

Having chosen source and destination layer based on our grid search results, we evaluate perplexity reduction on ten language modeling datasets with $\alpha=0.15$. Our evaluations include the three datasets used to select hyperparameters—arXiv, C4, and PG19—but the evaluation split is distinct from 'training' split used for hyperparameter tuning. Appendix B.2 presents details of the evaluation procedure, which included the Gemma3 1B, 4B, and 12B PT models.

Table 1 presents results for the three model sizes and ten data sets. For each model size, columns indicate perplexity of the baseline and recirculation models, and the percentage reduction in perplexity by incorporating recirculation. The 1B and 4B models obtain reductions up to 16% and the 12B model up to 35%[^3]. For nine of the ten data sets, we see robust improvements across model scales. The lambada set is an anomaly due to the presence of very short sequences and tokenization artifacts. We show later that recirculation has greater benefits for longer sequences.

[^3]: Note that the 12B model is relatively weak in terms of language modeling, but strong in downstream tasks post instruction-tuning as highlighted by its performance on standard benchmarks ([76]).

::: {caption="Table 1: Reduction in perplexity with recirculation for Gemma3 models across ten language-modeling datasets."}

:::

4.3 Normalization and ramping

Because the magnitude of residual-stream embeddings tends to grow over layers, we have found that renormalizing the source embedding before feeding it to the destination (Equations better conditions the model to obtain a consistent and reliable pattern of perplexity reduction over the hyperparameter sweep (Figure 5 and Figure 6). Appendix B.3 presents a range of normalization schemes we considered. The normalization scheme does not have much impact on the maximum perplexity reduction, only on the robustness of improvements over the source-destination landscape, which increases our confidence in being able to identify the optimal source-destination pair. The Appendix shows sweeps for multiple renormalization schemes, including an identity mapping (no renormalization). The Appendix also addresses the Gemma3 4B and 12B models, which turn out to critically require a non-convex mixture with $\beta=1$ instead of $\beta = 1-\alpha$.

In Section 4.4.4, we report results indicating that recirculating tokens at the beginning of the Gemma3 1B context window can be harmful. This finding does not seem surprising given there is little state information to propagate at the start of the window; and without a benefit from recirculation, the cost of potentially pushing the internal representations out of distribution may overwhelm. However, we find no harm of early-token recirculation in the 4B and 12B models. Nonetheless, we introduce ramping in the 1B model to attenuate $\alpha$ for the first tokens (see Appendix B.3), which yields a small reduction in perplexity.

4.4 Analysis of recirculation

4.4.1 Robustness across model families

Is Gemma3 somehow special in its receptiveness to recirculation? In Figure 7, we observe that four other model families—Ministral3, Pythia, Qwen3, and Phi2—all show a robust region in the source-destination heatmap where reductions in perplexity are observed, analogous to what we observe with Gemma3 1B. The models are roughly the same size and all benefit most from recirculation in the middle region of the architecture.

The range of layers in this heatmap spans the full range of layers, which explains the difference in heatmap shape[^4]. The plot also includes recirculation to the output of layer 0 (leftmost column), whose representations apparently have not been contextualized to the point that deeper layers can interpret the recirculated signal.

[^4]: Note that this Gemma3 1B heatmap is a full independent replication of Figure 5 with a different implementation, written in PyTorch and HuggingFace eco-system instead of JAX.

The percentage reduction in perplexity is significantly larger for Gemma3 than for the other families—about 5% versus less than 0.5%. However, we did not explore normalization adjustments or values of the mixture coefficient $\alpha$ for the other model families. Thus, while receptivity to recirculation seems universal, it seems likely that the magnitude of effects depends on specifics of an architecture or training procedure.

The second and fourth generations of the Gemma family show gains as pronounced as those observed with Gemma3 (Appendix C.1), highlighting that Gemma models are particularly compatible with recirculation.

4.4.2 Recirculation versus temperature tuning

Because perplexity is affected by the sharpness of a softmax distribution, we wanted to rule out the possibility that recirculation is merely sharpening or smoothing the distribution in an undifferentiated manner. We evaluated perplexity across a range of softmax temperatures and indeed found that with the base Gemma3 1B model, a temperature of 1.2 (versus the default of 1.0) reduced perplexity by 8.48% (see left panel of Figure 19). Recirculation alone reduces perplexity by 14.21%, and therefore recirculation must not be reducible to temperature adjustment. When we combine recirculation with temperature tuning, we obtain a reduction in perplexity of 19.55%, also with temperature 1.2. The fact that the two effects are nearly additive allows us to rule out the artifactual explanation for recirculation in terms of token-independent temperature tuning.

4.4.3 Recirculation versus looping

To verify that recirculation is influencing model dynamics in a structurally different manner than looped transformers ([20, 21, 26]), we compare recirculation and looped transformers. As we did with recirculation, we sweep over all $(\ell_1, \ell_2)$ layer pairs with $\ell_2 > \ell_1$. To implement looping, we inserted into the transformer stack a copy of the layers from $\ell_1+1$ up to and including $\ell_2$ immediately following the original $\ell_2$ in the stack. To be comparable to recirculation, we perform training-free evaluation. Although the literature indicates that models have been improved with training-free looping ([38, 39, 40]), the heatmaps in Figure 8 indicate that, for the Gemma3 family at least, looping does not produce robust benefits. Furthermore, while recirculation shows benefits across model scales, looping a pretrained model shows benefits only at larger model scales, as suggested by our plots and observed by [40]. These qualitative differences in the heatmaps confirm that looping and recirculation are operating on different principles.

4.4.4 Which tokens benefit from recirculation?

In all experiments reported to this point, tokens in every position of the context window are recirculated. Now we ask whether we can identify which tokens most contribute to recirculation performance improvements. Using a 1024-token context window and the Gemma3 1B PT model, we recirculate only the token in position $t$ and examine the benefit at lag $k$, i.e., to the token in position $t+k$. The increase in target log likelihood (equivalently, reduction in perplexity) relative to the no-recirculation model is a power function of $k$, with large increases at short lags but a residual tail even at long lags. Figure 9 a shows the recirculation benefit averaged over token position $t \in [0, 767]$ for lag $k \in [1, 256]$, relative to the baseline condition in which no recirculation occurs. Figure 9 b teases apart the effect of the recirculated token position, $t$, shown along the horizontal axis of the Figure, and the heatmap indicates the benefit magnitude over lags $k \in [1, 256]$. At the earliest positions, roughly $t<10$, recirculation reduces log likelihood, [^5] but at all later position, recirculation yields increases, particularly at short lags. Roughly, positions 20 to 200 in the window appear to have the most persistent effects, with a measurable benefit even at lag 256.

[^5]: In contrast to the Gemma3 1B model, we did not observe harmful effects of recirculating the early token positions with the larger Gemma3 4B and 12B models.

The effectiveness of recirculation depends not only on token position, but also token content. We tag each token with a grammatical part-of-speech (PoS) and then determine the mean recirculation benefit for each PoS regardless of token position. Figure 9 c indicates that adverbs, adjectives, and verbs show the biggest effects, whereas fixed classes such as numerals, determiners, and pronouns show the smallest effects. We obtain supporting evidence in experiments where we recirculate all and only tokens of a given PoS (Appendix C.4) compared to a count matched random set. Interestingly, for nouns we find that plural forms yield robust benefits whereas singular forms do not. The distinction in how models process singular and plural nouns was also noted by [77].

In contrast to the above experiment, in which we recirculated only a single token, we also conducted an experiment in which we recirculated all-but-one token. We obtain results complementary to Figure 9 a, b, hinting that the benefit of recirculating individual tokens is additive in log likelihood. In all, the position and content effects seem to further rule out artifactual explanations for the phenomenon, and to support the story that recirculation helps construct persistent state representations.

4.5 Generative tasks

We evaluated a range of downstream tasks that require models to generate responses, both single-token choices and thinking responses. Broadly, we observe modest to significant improvements in model accuracy.

4.5.1 Instruction following

In a simple instruction-following task, we prompted models with text that included:

Let's play a game. I will say two words. 
If the first word is a fruit, you say first.
If the second word is a fruit, you say second.

Exact prompts, which included two-shot examples, and additional details are presented in Appendix. This task is appealing as a test of executive function in children and neurological patients. The baseline Gemma3 1B IT (instruction tuned) model performs at chance, which is why we focus on the larger Gemma3 4B and 12B models. Larger models, despite being better, still have room for improvement as evident in Figure 10. Using the hyperparameters we previously selected to minimize perplexity on a tuning set, recirculation reduces the error rate by about 25% for the 4B model and by 75% for the 12B model. To get an upper bound on recirculation's potential, we overfit by performing a hyperparameter sweep specifically to maximize accuracy on this instruction-following task. Although this sweep constrains just two parameters (layer indices), recirculation improves significantly, as shown by the task-specific bars of Figure 10.

4.5.2 Contextualization

[11] curated a dataset to explore model contextualization failures of the sort that occur in the bank dialog (Figure 2 a). Each dataset instance consists of a context, a cue, and a question, includes zero or more embedded distractor sentences, and demands a yes/no answer, e.g.,

Examples were counterbalanced to ensure that biases (e.g., to prefer one sense of the word bank) did not influence results. Specifically, for model responses to count as correct, the model must respond to two variants of the question, e.g., Is it a financial institution? and Is it a geographical feature? Thus, chance accuracy is 25%.

In addition to questions with polysemous words like the example above, there were factual and gender-related questions, e.g.,

Figure 11 shows results for the Gemma3 1B, 4B, and 12B IT models. The abscissa of each graph is the number of distractor sentences; performance tends to drop with more distractors. Shown separately in each graph are the three question types and results for the baseline (solid) and recirculation (dashed) models. The chance rate of 25% is depicted as a dotted horizontal line.

For 1B, two of the three question types show a clear improvement with recirculation; the third is at chance for both baseline and recirculation. For 4B, two of the three question types again show a reliable improvement with recirculation; the third is about the same. For 12B, two of the three question types are at a disadvantage with recirculation; the third is at ceiling for both. On balance, recirculation is a win, though its failure on 12B is disappointing. We used source and destination hyperparameters based on perplexity evaluations in Section 4.2 for a pretrained not instruction tuned model; sweeping hyperparameters with the instruction tuned model did result in larger gains (see Appendix D.2).

4.5.3 Multiple-choice and single-token response tasks

::: {caption="Table 2: Accuracy of Gemma3 4B PT baseline model (second column) and model with recirculation (third column) on eight standard single-token response datasets. Green shading indicates results for which recirculation matches or beats the baseline, red shading indicates otherwise. Recirculation improves performance on 3/4 of the datasets. The last four columns report accuracy for adaptive recirculation (Section 4.6), a variant of recirculation involving fine tuning of recirculation coefficients alpha and beta. The success of adaptive recirculation is highly dependent on the dataset used for fine tuning (shown at the top of the column); consistent improvements in accuracy are observed when the MMLU test set is used for fine tuning. * denotes overlap between the train and test distributions."}

:::

We turn now to standard benchmark datasets, including those from Open LLM Leaderboard (v1) ([78]), tested using the eval-harness package ([79]). The tasks we evaluate are: ARC easy ([80]), ARC challenge ([80]), MMLU ([81]), Winogrande ([82]), BoolQ ([83]), PiQA ([84]), HellaSwag ([85]), and Lambada ([86]). Lambada is a fill-in-the-blank single token response task. All the rest are multiple-choice tasks. Accuracy is measured as the percent correct responses. We use the MMLU development set to search over hyperparameters, as detailed in Appendix D.3. All results are based on zero-shot evaluation.

Table 2 shows accuracy for the eight datasets in the first column for the baseline (second column) and recirculation (third column) models. Recirculation improves performance on six of the eight datasets, although the accuracy differences—both positive and negative—are modest. Disregard the last four columns of Table 2 for the moment; we discuss them in Section 4.6.

4.5.4 GSM8k

In contrast to the single-token-response datasets examined in the previous section, we turn to problems that are solved with chain-of-thought reasoning ([87, 88]), specifically the grade-school math problems in GSM8k ([89]). With single-token responses, any benefit of recirculation must depend on inferences made in processing the problem statement; with chain of thought, recirculation has the additional opportunity to support extended reasoning.

GSM8k also contrasts with the previous datasets in that it allows for a large set of candidate responses. When the number of candidates is large, improvements in model performance can be characterized in terms of either capability sharpening or capability expansion ([90]). The distinction is made with pass@k metrics ([90]): improved model performance with pass@1 (greedy decoding) indicates sharpening—the correct response beating out other responses in the set of candidates; improved model performance with pass@128 (accepting any of 128 samples if they are correct) indicates expansion of the set of possibilities. We therefore examine model performance with pass@1 (greedy decoding) and pass@128 (using a higher temperature and nucleus sampling, as recommended by [76], [76]). Presented results are based on zero-shot evaluation. Following [87], we use the zero-shot chain-of-thought prompt, which is important without few-shot examples as we are only evaluating base models in our case.

Figure 12 compares baseline and recirculated Gemma3 4B models (brown and blue bars, respectively) for pass@1 and pass@128. We turn to the green bar in Section 4.6. Recirculation improves both pass@1 and pass@128 performance, indicating its support for both capability sharpening and capability expansion. Recirculation appears to benefit extended generative responses more than single-token responses (Section 4.5.3), thus revealing its promise for advanced problem solving and complex reasoning.

4.6 Adaptive recirculation

Our strategy to this point has been to determine how far we could push recirculation as a pure inference-time technique for an off-the-shelf model. Recirculation is surprising and intriguing specifically because it works out-of-the-box. The blindingly obvious next step is to improve the robustness and gains from recirculation via model adaptation. Because training a recurrent architecture comes at a high cost and there is a risk of overfitting if the entire model is fine tuned, we begin by taking small steps away from the training-free setting, focusing on modulating the recirculation hyperparameters $\alpha$ and $\beta$ (. Our goal is to make changes as minimal as possible that show benefits over the training-free recirculation setting.

For training experiments, we use 250 documents from each of arXiv, C4, and PG19. Experiments are conducted with Gemma3 1B PT. We fix the source and destination layer indices that produced earlier recirculation results (Table 1) and explore methods that learn representation-mixing terms $\alpha$ and $\beta$ to minimize prediction loss.

We score six different adaptation methods on nine datasets by the reduction in evaluation-set perplexity they achieve relative to the baseline (no recirculation) model. Experimental details are presented in Appendix D.5. The six alternative methods, shown from left to right in Figure 13, are as follows.

• Fixed. We use the fixed, previously determined coefficients ($\alpha=0.15$, $\beta=0.85$), thereby replicating earlier results.
• Learned constant scalars. We perform gradient descent in the scalars $\alpha$ and $\beta$ to best fit the training data.
• Learned conditional scalars. We train an MLP that maps the token-specific source and destination embeddings to scalar $\alpha$ and $\beta$ values, allowing the model to self-determine the strength of recirculation for each token.
• Learned constant vectors. Instead of scaling the source and destination vectors by constants when recirculating, we scale by learned vector-valued $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$:

$ {\boldsymbol z}{t+1, t, d} = \boldsymbol{\alpha} \circ f({\boldsymbol z}{t, t, s}) + \boldsymbol{\beta} \circ {\boldsymbol z}_{t, t, d},\tag{3} $

where $\circ$ is the Hadamard product.

• Learned conditional vectors. We train an MLP that maps the token-specific source and destination embeddings to vector-valued $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$.
• Model fine tuning. We fine tune the Gemma3 1B architecture augmented with the recurrent connections of recirculation. We fix the mixture coefficients at $\alpha=0.15$ and $\beta=0.85$.

Of the methods that learn coefficients $\alpha$ and $\beta$, the two that learn vector valued coefficients (fourth and fifth sets) perform better than those that learn scalar coefficients (second and third sets). And the two methods that learn token-conditional coefficients (third and fifth set) outperform the two methods that learn static coefficients (second and fourth set). Of the four methods, the best (fifth set: learned conditional vector $\boldsymbol{\alpha}, \boldsymbol{\beta}$) trains an MLP to generate token-specific mixture vectors. The other methods (second, third, and fourth sets) can be considered ablations demonstrating that both vector-valued coefficients and conditioning on the current token is essential. Henceforth, we refer to the superior method as adaptive recirculation.

Adaptive recirculation obtains a mean 23.0% reduction in perplexity, relative to the 8.5% reduction for basic recirculation. Adaptive recirculation beats recirculation for each of the nine datasets (see Figure 26). Adaptive recirculation performs at least as well as full Gemma3 1B fine tuning (23.0% versus 21.6%). The advantage of adaptive recirculation is that the Gemma3 model itself is untouched and therefore there is less risk of overfitting the model via fine tuning.

On downstream tasks, adaptive recirculation also shows promise, with the caveat that the dataset used for training the MLP is critical. Returning to the eight single-token-response datasets, the last four columns of Table 2 present accuracy for adaptive recirculation on these evaluation datasets for four different training datasets. Adapting to a small fraction of the MMLU test split (3600 of about 14000 examples) yields robust across-the-board improvements, and adapting to MMLU (auxiliary train split) yields on balance improvements. For comparability, we use the full test set of MMLU for evaluation even when training on the MMLU test split, which includes the examples used for training; performance on other datasets highlights generalization. Note that the auxiliary train set for MMLU is in general much lower quality than the well-curated MMLU test set, which may be the reason for its inferior performance. Finally, the MMLU train set is based on a combination of existing datasets, with potential contamination ([81]). Training directly on other datasets such as ARC Easy or ARC Challenge produces significant drops in performance, although these datasets have a much smaller number of examples. Details of these experiments are provided in Appendix D.5.

Although neither recirculation nor adaptive recirculation yield robust accuracy gains on single-token response tasks, the gains for GSM8k offer a promising signal for extended generative response tasks. The green bar in each panel of Figure 12 shows GSM8k performance with adaptive recirculation. Adaptive recirculation greatly benefits GSM8k, yielding 8.8% and 20.9% reductions in error rate with pass@1 and pass@128, respectively—a staggering result considering the model itself is untouched.

5. Discussion

Section Summary: The paper presents recirculation as a simple, training-free way to add recurrence to existing transformer models like Gemma3 by routing activations between layers at inference time. This change improves perplexity and task performance while adding almost no extra cost during normal generation, though it slows down the initial processing of long contexts, and the authors frame it as a way to discover a model’s natural “design affordances” rather than imposing changes through retraining. They also note key limitations, including the need for task-specific tuning, uncertain generalization across model families, and remaining computational challenges for very large inputs.

Our work explores the consequences of recirculation, a training-free, inference-time architectural modification to incorporate recurrence into a transformer for improved state tracking. We have shown that for the Gemma3 pretrained and instruction-tuned architectures, recirculation yields notable reductions in model perplexity and enhances models' ability to follow instructions, answer questions, and problem solve. When generating responses, recirculation incurs almost no additional computation cost; although recirculation requires running two transformer stacks in parallel instead of a single stack, modern AI hardware parallelizes efficiently in this case. However, there is an additional cost of processing the prefill context autoregressively, which for large-context problems, can be quite slow.

Although there is a robust literature on architectural improvements to the transformer, this literature focuses almost exclusively on training models from scratch or incorporating modifications mid-training. The notable exception is the notion of training-free looping ([38, 39, 40]), a related but distinct architectural modification. Both looping and recirculation exploit a key property of the transformer's residual pathway: the alignment of representations across layers ([17]). Recirculation is complementary to and can be combined with other methods in common practice, including looping, variable computation time, and the coarser-grain recurrence that occurs with both latent thought and chain of thought.

5.1 Model-design affordances

We view recirculation not as a `shovel ready' technique to be incorporated into state-of-the-art models, but rather as a methodological or philosophical contribution. By focusing on training-free architectural modifications, we are essentially asking the model to reveal to us its ingrained paths for improvement. Our search over where to place recurrence in the architecture and how to mix and normalize activations is informed by the model itself. The alternative—the typical course of research in machine learning—is to propose an arbitrary modification to an architecture and then evaluate the potential of this modification via costly training. Our key hypothesis is that if we identify how the pretrained model wants us to tweak it without modifying its weights, we also identify an inductive bias that will facilitate and simplify training, whether from scratch or with fine tuning. Our adaptive recirculation experiments support this hypothesis.

In product design, practitioners talk about design affordances ([91]), which are properties of an object that inform us about how it should be used. For example, a door handle affords grasping and pulling; a door plate affords pressing. Analogously, our research explores model-design affordances—the natural properties of a foundation model that we can exploit and amplify to improve the model's basic operation. By improving this basic substrate, we improve all capabilities that build on it. Most approaches to incorporating recurrence within the internal layers of a foundation model do so via adapters or cross attention. Our observations suggest that the simpler strategy of mixing activation vectors may suffice.

5.2 Limitations and future directions

We close by discussing limitations of the present work and promising future directions that will address these limitations.

• Determination of optimal hyperparameters. The optimal model hyperparameters—the source and destination layers and the recirculation coefficients $\alpha$ and $\beta$ —appear to be domain or problem dependent. However, hyperparameters selected based on one dataset or one criterion often generalize to another. For example, hyperparameters based on perplexity evaluations were used for downstream tasks; hyperparameters based on one specific dataset (MMLU) were used for a range of other single-token-response tasks; and small sets of examples are often sufficient to select hyperparameters. However, unless we can identify task-universal hyperparameters or an automatic mapping from task to hyperparameters, the practical applicability of recirculation will be limited.
• Dependence on model family. While recirculation shows the same qualitative pattern across model families, it is possible that Gemma's Peri-LN architecture and training optimization process yield larger benefits than other model families (Appendix C.1). Generalizability to other architectures requires further investigation.
• Normalization. We find that normalization of source activation vectors makes recirculation behave more robustly. For the Gemma3 family, we find that of the many candidate normalization schemes we investigated, one in particular seems to work well across model sizes (Appendix B.3). Nonetheless, further experiments are needed to determine whether normalization is necessary and how to normalize for other model families.
• Blockwise recurrence. Recirculation is computationally cheap during autoregressive generation because running two stacks in parallel is nearly as efficient with modern hardware as one stack. However, during prefill, token-by-token processing is also required. For state-of-the-art foundation models, which may include a large amount of contextually relevant information, sequential processing of the prefill may be infeasible. One way to alleviate the computational burden would be to perform blockwise recirculation, i.e., to process a set of $K$ tokens in parallel, and then recirculate those $K$ tokens simultaneously with the next $K$ as they are processed for the first pass. We have yet to explore the trade off between $K$ and recirculation performance.
• Multiple recirculation iterations. A true recurrent network would recirculate the activation in stacks 1 to $t-1$ at step $t$. In contrast, the variant of recirculation we explored recirculates only stack $t-1$, such that each stack is recirculated exactly one time. We have yet to study an intermediate approach in which each stack is recirculated exactly $r$ times. Figure 14 shows an unrolled architecture with $r=2$ iterations per stack. We wished to start with the simplest instantiation, but given the low cost of recirculating $r$ stacks in parallel, increasing the number of iterations is worthwhile.
• Multiple recirculation paths. We picked a single source, destination pair for recirculation, but one could recirculate along multiple paths simultaneously. Indeed, for some architectures (e.g., see Gemma3 4B and 12B recirculation sweeps in Figure 8), there appeared to be distinct, disjoint regions of the hyperparameter sweep where recirculation was effective. One hypothesis is that these distinct regions are complementary, e.g., they may convey state information at different levels of abstraction.
• Adaptation. The present work has only scratched the surface of possible approaches to adapting models to enhance recirculation effects. Our results with adaptive recirculation are extremely promising considering that the tuning set we used was quite small and that tuning of model weights was not required to obtain robust performance improvements. Before turning to model-weight fine tuning, many avenues for improvement still exist, e.g., token conditioned recirculation pathways and normalization methods.

By listening to the model's own internal dynamics rather than forcing costly architectural overhauls, recirculation offers a powerful, computationally inexpensive path forward. Unlocking these intrinsic affordances may solidify the model's basic contextual understanding, which in turn provides the vital scaffolding required for extended, multi-turn reasoning.

Appendix

Section Summary: The appendix describes an unrolled transformer architecture that performs multiple recirculation passes through the model stacks to refine outputs, along with the specific data sampling and evaluation procedures used to test it. It details hyperparameter choices for different model sizes, including which layers serve as sources and destinations for recirculation, and reports perplexity results across ten datasets after processing fixed-length text windows. The section also examines several normalization methods for blending activations between layers, presenting sweeps that show how scaling and orthogonalization techniques affect performance by preventing norm mismatches.

A. Unrolled recirculation architecture

Figure 14 shows an unrolled transformer with two iterations of recirculation. In general, with $k$ iterations of recirculation, it will be necessary to run $k+1$ stacks for each input step.

B. Recirculation implementation details

B.1 Hyperparameter sweeps

For the hyperparameter sweeps (Figure 5 and Figure 6), we used a context window of 1024 tokens and pulled roughly 500 windows from training documents in three datasets: arXiv, C4, and PG19. We used at most two windows from each document, requiring that the windows had no filler tokens (i.e., the document extended at least to the end of the window). This requirement yielded 484 windows for arXiv (495132 predicted tokens), 488 windows for C4 (499224 tokens), and 500 windows for PG19 (511000 tokens).

B.2 Perplexity evaluation

For our perplexity evaluation (Table 1), we used the entire evaluation set from nine data sets (arXiv, billsum, booksum/books, C4/webtextlike, gov report, lambada, newsroom, PG19, and pubmed) and the first 10000 documents from a tenth data set (big patent). The evaluation split was labeled 'validation'for C4 and 'test' for all other datasets. We partitioned each document into chunks of 1024 tokens and excluded partially filled windows (i.e., less than 1024 tokens), except for three data sets whose documents were typically too short (c4/webtextlike, lambada, and newsroom). This procedure resulted in the evaluation of the number of tokens listed in the second column of Table 1.

Based on hyperparameter sweeps for the 1B, 4B, and 12B models, we used the source and destination pairs shown in Table 3. These hyperparameters were used for all results presented in the article unless otherwise mentioned.

: Table 3: Optimal hyperparameters based on tuning dataset

Model	Source	Destination
Gemma3 1B PT	11	4
Gemma3 4B PT	18	9
Gemma3 12B PT	35	16

B.3 Normalization and ramping

Figures 15-17 show hyperparameter sweeps for multiple renormalization techniques, i.e., scaling of the source vector when recirculating to the destination layer. Scaling helps to ensure that the source activations do not overwhelm the destination activations, due to the fact that embedding norms increase over transformer layers ([92, 93]). The heatmaps depict percentage reduction (negative) or increase (red) in perplexity over our three tuning datasets (arXiv, C4, PG-19).

\begin{tabular}{p{3in} l} 
  \hline
  \textbf{description} & \textbf{formula} \\
  \hline
  no normalization & $d' = (1-\alpha) d + \alpha s$ \\
  \hline
  convex combination, magnitude ratio & $d' = (1-\alpha) d + \alpha \frac{||d||}{||s||} s$ \\
  \hline
  nonconvex combination, magnitude ratio & $d' = d + \alpha \frac{||d||}{||s||} s$ \\
  \hline
  convex combination, magnitude ratio reweighting, destination renormalized
   {} & $\begin{aligned} d' & = (1-\alpha) d + \alpha \frac{||d||}{||s||} s \\ d' & \leftarrow \frac{||d||}{||d'||} d' \end{aligned} $ \\
  \hline
  nonconvex combination, recirculate only component of source orthogonal to destination &
  $ \begin{aligned} d' &= d + \alpha \left(s- \frac{s^\textsc{t}d}{d^\textsc{t}d} d\right) \\ &= \left(1 - \alpha \frac{||s||}{||d||} \cos(\theta)\right) d + \alpha s \\ \theta &= \arccos \left(\frac{s^\textsc{t} d}{||d|| ||s||} \right) \\ \end{aligned} $ \\
  \hline
  convex combination (when aligned) or nonconvex (when orthogonal), magnitude ratio reweighting &
  $ \begin{aligned} d' &= d + \alpha \frac{||d||}{||s||} \left(s- \frac{s^\textsc{t}d}{d^\textsc{t}d} d\right) \\ &= (1 - \alpha \cos(\theta)) d + \alpha \frac{||d||}{||s||} s \\ \end{aligned} $ \\
  \hline
  nonconvex combination, magnitude ratio reweighting, scale source by novelty
   {} &
  $ \begin{aligned} d' &= d + \alpha \frac{d^\textsc{t} d}{s^\textsc{t} d} s \\ &= d + \alpha \frac{||d||}{||s||} \frac{s}{\cos(\theta)} \end{aligned} $ \\
  \hline
  rotate destination embedding toward source embedding &
  $ \begin{aligned} d' &= \frac{1}{\sin(\theta)} \left[\sin((1-\alpha)\theta) d + \sin(\alpha \theta) \frac{||d||}{||s||} s \right] \end{aligned} $
  \end{tabular}

The normalization schemes are summarized in the Figures and in Table 4. We simplify the notation used in the main paper, where $ \boldsymbol{z}_{i, j, l} $ referred to the embedding at unrolling step $ i $ for input step $ j $ and layer $ l$. Instead, we denote:

$ \begin{aligned} d \equiv~ & \boldsymbol{z}{t, t, d}, \ s \equiv~ & \boldsymbol{z}{t, t, s}\text{, and} \ d' \equiv~ & \boldsymbol{z}_{t+1, t, d}. \end{aligned} $

The normalization schemes in Table 4 include the simple scheme in which no normalization is applied. The no-normalization heatmaps (top row, middle panel in Figures 15-17) attain reasonable outcomes but clearly the schemes with $L_2$ normalization (top row, right panel and second row, left panel) are better behaved in the sense that there are fewer hyperparameters that result in poorer performance. Oddly, a convex combination of source and destination vectors (top row, right panel) is superior to a nonconvex combination (second row, left panel) for the Gemma3 1B model, but the nonconvex combination is superior for Gemma3 4B and 12B. Other candidate schemes either had pathologies (non-smooth heatmaps, large red regions) or mimicked the simple norm-ratio schemes. Based on the Gemma family, we recommend evaluating at least the convex and non-convex mixtures with norm-ratio adjustment of the source.

Ramping. As we mentioned in the main article, for the Gemma3 1B model, we found a small additional reduction in perplexity if we ramped up the recirculation coefficient over the first 10 steps. Specifically, we defined the $\alpha$ coefficient at step $t \ge 0$ to be $\alpha_t \equiv \min (t/10, 1) \alpha$.

C. Basic results

C.1 Robustness across architectures

In Figure 7, we contrasted recirculation hyperparameter sweeps for Gemma3 1B to four other model families: Ministral3, Qwen3, Pythia, and Phi2. The Figure compares architectures using only the arXiv training set. Due to the fact that all of these models were relatively small, we utilized the same hyperparameters as we did for Gemma3 1B PT: scaling of the source layer norm to match the target layer norm, $\alpha=.07$ and $\beta = 1-\alpha$. Using $\beta = 1$ —the choice for Gemma3 4B and 12B—produced qualitatively similar results.

In addition to the comparison across different model families, we also compared results from Gemma3 with those from other generations of the Gemma model family, including the older Gemma2 and the newer Gemma4 (Figure 18). The arXiv train set is again used in this Figure. The gains for Gemma2 and Gemma4 are as pronounced as those for Gemma3 in the main text, despite the fact that there are regions of the hyperparameter space where recirculation is quite harmful. Note that smaller variants of Gemma4 (i.e., E2B and E4B) uses cross-layer KV cache sharing as well as per-layer embedding, which can explain the instability in these plots ([94]).

We hypothesize that this favorability of Gemma architectures can be attributed to two distinct explanations:

• Gemma architectures use a Peri-LN architecture ([95]) which adds layer-norm to both the input of the layer (i.e., self-attention or MLP), as well as the output of the layer. In contrast, other architectures only use input normalization ([93]). The use of output normalization might improve the model's compatibility with recirculation by avoiding diminishing contribution from later layers ([96]).
• The compatibility with recirculation is strongly dependent on the optimization process used for training the model. Hence, the scheme used to train Gemma models is particularly favorable for recirculation.

C.2 Recirculation versus temperature tuning

Gemma3 1B with the PG-19 evaluation ('test') set was used for temperature tuning experiments. The documents were split into 1024 token chunks per context window; partially filled sequences were excluded. We swept over a wide enough range of softmax temperatures to identify the range in which adjusting model temperature improved perplexity. Figure 19 shows the sweep without recirculation in the left panel and with recirculation in the right panel. Whether with or without recirculation, the optimal temperature was about 1.2. Although we performed a more granular sweep for the combined experiment, we report in the main paper the optimum over only the range of temperatures used for the temperature-alone experiment.

C.3 Recirculation versus looping

The dataset is comprised of 250 documents from the arxiv train set, with 2 subsequences taken from each document starting from the beginning of the document, each of size 1024 tokens. This is identical to our setting for the grid search results in Figure 5. Results for recirculation are shown with $\alpha=0.07$, with convex combination ($\beta=0.93$) for the 1B model and a non-convex combination ($\beta=1.0$) for the 4B and 12B models, as discussed in the main text.

C.4 Which tokens benefit from recirculation?

Experiments in the main paper are based on Gemma3 1B PT with the arXiv data set, training split, with a context window of 1024 tokens extracted from a randomly selected position within each document. Part-of-speech tags are extracted with nltk.pos_tag. We processed 24960 documents and for each, we recirculated tokens 0-767 individually and examined downstream effects at lags 1-256.

We also ran an experiment in which we recirculated all and only tokens tagged with a given part of speech (Figure 20). For this experiment, 3120 documents from the arXiv train set were used, and from each, a sequence of 1024 tokens was taken from a random starting position within the document. Each of these sequences was tested with recirculation of only PoS-selected tokens.

D. Downstream evaluations

D.1 Instruction following

We generated 800 queries of the following form:

<start_of_turn>user
Let's play a game. I will say two words.
If the first word is a fruit, you say first.
If the second word is a fruit, you say second.
For example, if I say `pomegranate kangaroo', respond first.
And if I say `koala fig', respond second.
The word pair is `monkey banana'. Your answer?<end_of_turn>
<start_of_turn>model

On half the trials, the model was asked instead to identify the position of the animal ("If the first word is an animal, you say first..."). We formed 400 trials by combining twenty different fruit names with twenty different animal names. With the respond-to-fruit and respond-to-animal variants, this yields 800 trials total. The fruits are: apple, avocado, banana, blueberry, cantaloupe, cherry, grape, honeydew, kiwi, lemon, lime, mango, orange, peach, pear, pineapple, plum, raspberry, strawberry, watermelon. The animals are: bear, bird, cat, deer, dog, dolphin, elephant, fox, giraffe, lion, lizard, monkey, penguin, shark, snake, spider, tiger, whale, wolf, zebra.

We selected the most likely response among eight candidate tokens, which consisted of the words first and second, both in upper- and lower-case form and with and without a leading space.

For the Gemma3 4B and 12B IT, we used the source and destination layers determined by our previous perplexity hyperparameter sweep, and $\alpha=0.07$. For the task-specific result, we conducted a sweep using the instruction-following dataset to determine an upper bound on performance. These sweeps are shown in Figure 21. The 1B sweep is included as well, revealing that the model is essentially performing at chance. For the 4B model, source layer 18, destination layer 8 was best; for the 12B model, source layer 29, destination layer 16 was best.

D.2 Contextualization

For Figure 11 of the main paper, we used hyperparameters chosen based on minimizing perplexity of the pretrained model. We also swept hyperparameters of the Gemma3 1B IT model using a particular condition of the [11] dataset: both the 5-distractor condition of the gender questions and the polysemy questions. These sweeps appear in the upper right of Figures 22-24, corresponding to the 1B, 4B, and 12B models. In the lower row of the Figure, left to right we show recirculation accuracy with hyperparameters chosen based on pretrained model perplexity, instruction tuned model perplexity, accuracy for gender 5-distractor condition, and accuracy for polysemy 5-distractor condition. Note that the 1B model is not much above chance except for no-distractor polysemy questions.

D.3 Multiple-choice and single-token response tasks

We tested the Gemma3 4B PT model. To determine the optimal hyperparameters we conducted a (source, destination) sweep using the 1531 MMLU development-set problems. These problems are distinct from the examples used for evaluation. Figure 25 a shows the sweep, fixing $\alpha=0.07$. Figure 25 b shows a scan over $\alpha$, fixing source and destination layers to be the pair that yields the lowest perplexity in the sweep of Figure 25 a. The resulting hyperparameters that were used in the various single-token response datasets were source 16, destination 5, $\alpha=0.09$.

D.4 Standard benchmark datasets: GSM8k

We trained Gemma3 4B model with identical hyperparameters as the perplexity experiments (see Section 4.6), except that we masked the prompt part of the question, and only trained the model to predict the ground-truth response present in the dataset. We only report results using our best-performing conditional $\boldsymbol{\alpha}, \boldsymbol{\beta}$ scheme as highlighted in Figure 13.

D.5 Adaptive recirculation

For all training experiments, we use 250 documents from each of the PG19, C4, and arXiv training sets. Documents are partitioned into windows of 1024 tokens and only completely full windows were included in the training set. The transformer with recirculation and the MLP module is trained with Back Propagation Through Time (BPTT). For the perplexity studies of Figure 13, we use the Gemma3 1B PT model with the previously selected source and destination layers (Table 3).

Our architecture is adapted from the next-latent prediction MLP in [42], where we use a 2 hidden layer GELU-based MLP with the same hidden size as the model dimension, with layer-norm at the input of the MLP. The input is twice the model hidden dimension as we concatenate the source and the destination embeddings to be fed to the MLP. Output size is dependent on the formulation used; e.g., it would be twice the model hidden dimension for learned conditional vectors scheme, which predicts two scalars ($\alpha$, $\beta$) per dimension.

We use sigmoid activation at the output to ensure that these coefficients lie in [0, 1]. Further, we initialize the parameters of the network such that it starts with $\alpha=0.1$ and $\beta=0.9$ at initialization, based on the range of values we found to be suitable from our grid-search results.

We train the model for 100 steps with a batch size of 32 using AdamW ([97]), a learning rate of 3e-4, and a weight decay of 1e-4. For all simulations other than LLM fine-tuning, we freeze the parameters of Gemma3. For LLM fine-tuning, we disable weight decay and use a small learning rate of 1e-5. Similarly, for the unconditional prediction schemes (same for all inputs), we disable weight decay and increase the learning rate to 1e-1.

Evaluation was performed with the validation or test set of nine datasets: ArXiv, PubMed, PG19, BookSum, Lambada, Gov Report, BillSum, OpenWebText, and Big Patent. Note that nonoverlapping subsets of ArXiv and PG19 were used for training and evaluation.

Figure 26 b compares recirculation with fixed coefficients ($\alpha=0.15, \beta=0.85$) and the scheme in which an MLP is trained to produce vector-valued $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$ coefficients for each token (Figure 26 a). We refer to this scheme as adaptive recirculation for short, as well as conditional $\boldsymbol{\alpha}, \boldsymbol{\beta}$ in the text. The comparison indicates that adaptive recirculation increases the percentage reduction in perplexity for every dataset, and by a factor of three or more for many of the datasets.

When training our conditional vector $\boldsymbol{\alpha}, \boldsymbol{\beta}$ scheme on downstream tasks with Gemma3 4B (Table 2 and Figure 12), we use the same hyperparameters as our perplexity experiments. However, we mask the prompt and only train on the response, which refers to only a single token in the single-token benchmarks (Table 2). Because the ARC datasets have fewer examples than the MMLU datasets (MMLU train set: 99842, MMLU test set: 14042, ARC easy: 2251, ARC challenge: 1119), we used multi-epoch training for the ARC datasets as we train on a total of 3200 examples (100 steps with a batch-size of 32).

References

Section Summary: This section compiles a lengthy list of academic citations, mostly recent conference papers and arXiv preprints on transformer models, language-model reasoning, and related AI techniques. It also includes a handful of older works from cognitive science on topics like mental models and decision-making under uncertainty. The references point to venues such as NeurIPS, ICLR, and ACL, along with links to full texts.

[1] Michael C. Mozer et al. (2026). The Topological Trouble With Transformers. arXiv:2604.17121 [cs.LG]. https://arxiv.org/abs/2604.17121. arXiv:2604.17121.

[2] Belinda Z. Li et al. (2025). (How) Do Language Models Track State?. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=8SXosAVIFH.

[3] Mateusz Piotrowski et al. (2025). Constrained belief updates explain geometric structures in transformer representations. arXiv:2502.01954 [cs.LG]. https://arxiv.org/abs/2502.01954. arXiv:2502.01954.

[4] Nikhil Prakash et al. (2026). Language Models Use Lookbacks to Track Beliefs. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=6gO6KTRMpG.

[5] Adam Shai et al. (2024). Transformers Represent Belief State Geometry in their Residual Stream. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=YIB7REL8UC.

[6] Philippe Laban et al. (2025). LLMs Get Lost In Multi-Turn Conversation. arXiv:2505.06120 [cs.CL]. https://arxiv.org/abs/2505.06120. arXiv:2505.06120.

[7] Daniel P. Sawyer et al. (2025). Exploring exploration with foundation agents in interactive environments. In NeurIPS 2025 Workshop on Embodied World Models for Decision Making. https://openreview.net/forum?id=ay2d66HWO2.

[8] Tim R. Davidson et al. (2025). The Collaboration Gap. arXiv:2511.02687 [cs.AI]. https://arxiv.org/abs/2511.02687. arXiv:2511.02687.

[9] Arpandeep Khatua et al. (2026). CooperBench: Why Coding Agents Cannot be Your Teammates Yet. arXiv:2601.13295 [cs.LG]. https://arxiv.org/abs/2601.13295. arXiv:2601.13295.

[10] Peiyang Song et al. (2026). Large Language Model Reasoning Failures. Transactions on Machine Learning Research. https://openreview.net/forum?id=vnX1WHMNmz.

[11] Lepori et al. (2025). Racing Thoughts: Explaining Contextualization Errors in Large Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 3020–3036. doi:10.18653/v1/2025.naacl-long.155. https://aclanthology.org/2025.naacl-long.155/.

[12] Vul et al. (2014). One and Done? Optimal Decisions From Very Few Samples. Cognitive Science. 38(4). pp. 599–637. doi:10.1111/cogs.12101.

[13] Tversky, Amos and Kahneman, Daniel (1971). Belief in the Law of Small Numbers. Psychological Bulletin. 76(2). pp. 105–110. doi:10.1037/h0031322.

[14] Johnson-Laird, P.N. (1983). Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Harvard University Press. https://books.google.com/books?id=FS3zSKAfLGMC.

[15] Ghandeharioun et al. (2024). Patchscopes: a unifying framework for inspecting hidden representations of language models. In Proceedings of the 41st International Conference on Machine Learning.

[16] Yaniv Nikankin et al. (2025). Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs. arXiv:2506.09047 [cs.LG]. https://arxiv.org/abs/2506.09047. arXiv:2506.09047.

[17] Elhage et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.

[18] Giyeong Oh et al. (2026). Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=LWmfHjJnrx.

[19] M. Zhang et al. (2026). Amplification-free residual networks. arXiv:26xx.xxxxx [cs.LG]. https://arxiv.org/abs/26xx.xxxxx. arXiv:26xx.xxxxx.

[20] Mostafa Dehghani et al. (2019). Universal Transformers. In International Conference on Learning Representations. https://openreview.net/forum?id=HyzdRiR9Y7.

[21] Giannou et al. (2023). Looped transformers as programmable computers. In International Conference on Machine Learning. pp. 11398–11442.

[22] Nikunj Saunshi et al. (2025). Reasoning with Latent Thoughts: On the Power of Looped Transformers. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=din0lGfZFd.

[23] Liu Yang et al. (2024). Looped Transformers are Better at Learning Learning Algorithms. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=HHbRxoDTxE.

[24] Aleksandra I. Nowak et al. (2024). Towards Optimal Adapter Placement for Efficient Transfer Learning. arXiv:2410.15858 [cs.LG]. https://arxiv.org/abs/2410.15858. arXiv:2410.15858.

[25] David Raposo et al. (2024). Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. arXiv:2404.02258 [cs.LG]. https://arxiv.org/abs/2404.02258. arXiv:2404.02258.

[26] Ibrahim Alabdulmohsin and Xiaohua Zhai (2025). Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=cLbGkINOLP.

[27] Sangmin Bae et al. (2025). Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=QuqsEIVWIG.

[28] Yilong Chen et al. (2025). Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking. arXiv:2502.13842 [cs.CL]. arXiv:2502.13842.

[29] Jonas Geiping et al. (2025). Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=S3GhJooWIC.

[30] Ivan Rodkin et al. (2025). Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling. arXiv:2508.16745 [cs.LG]. https://arxiv.org/abs/2508.16745. arXiv:2508.16745.

[31] Yu et al. (2025). Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 11257–11272. doi:10.18653/v1/2025.emnlp-main.567. https://aclanthology.org/2025.emnlp-main.567/.

[32] Rui-Jie Zhu et al. (2025). Scaling Latent Reasoning via Looped Language Models. arXiv:2510.25741 [cs.LG]. https://arxiv.org/abs/2510.25741. arXiv:2510.25741.

[33] Boyi Zeng et al. (2026). PonderLM: Pretraining Language Models to Ponder in Continuous Space. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=UrM4MNRYZm.

[34] Ahmadreza Jeddi et al. (2026). LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=RzYXb5YWBs.

[35] Sunny Sanyal (2026). Looped-GPT: Looping During Pre-training improves Generalization. Blog. https://sanyalsunny111.github.io/posts/2026-01-15-post1-looped-gpt/.

[36] Yeskendir Koishekenov et al. (2025). Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts. arXiv:2510.07358 [cs.LG]. https://arxiv.org/abs/2510.07358. arXiv:2510.07358.

[37] Sean McLeish et al. (2025). Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence. arXiv:2511.07384 [cs.CL]. https://arxiv.org/abs/2511.07384. arXiv:2511.07384.

[38] Ziyue Li et al. (2025). Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs. arXiv:2507.07996 [cs.LG]. https://arxiv.org/abs/2507.07996. arXiv:2507.07996.

[39] Lizhang Chen et al. (2026). Training-Free Looped Transformers. arXiv:2605.23872 [cs.CL]. https://arxiv.org/abs/2605.23872. arXiv:2605.23872.

[40] Ng, David Noel (2026). LLM Neuroanatomy: How I Topped the LLM Leaderboard Without Changing a Single Weight. https://dnhkng.github.io/posts/rys/.

[41] Edward S. Hu et al. (2025). The Belief State Transformer. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=ThRMTCgpvo.

[42] Jayden Teoh et al. (2025). Next-Latent Prediction Transformers Learn Compact World Models. arXiv:2511.05963 [cs.LG]. https://arxiv.org/abs/2511.05963. arXiv:2511.05963.

[43] Hai Huang et al. (2026). Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA. arXiv:2602.22617 [cs.LG]. https://arxiv.org/abs/2602.22617. arXiv:2602.22617.

[44] Yuxi Liu et al. (2026). The Serial Scaling Hypothesis. arXiv:2507.12549 [cs.LG]. https://arxiv.org/abs/2507.12549. arXiv:2507.12549.

[45] Merrill, William and Sabharwal, Ashish (2023). The Parallelism Tradeoff: Limitations of Log-Precision Transformers. Transactions of the Association for Computational Linguistics. 11. pp. 531–545. doi:10.1162/tacl_a_00562. https://aclanthology.org/2023.tacl-1.31/.

[46] Strobl et al. (2024). What Formal Languages Can Transformers Express? A Survey. Transactions of the Association for Computational Linguistics. 12. pp. 543-561. doi:10.1162/tacl_a_00663. https://doi.org/10.1162/tacl_a_00663.

[47] William Merrill and Ashish Sabharwal (2025). A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers. arXiv:2503.03961 [cs.LG]. https://arxiv.org/abs/2503.03961. arXiv:2503.03961.

[48] Biran et al. (2024). Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 14113–14130. doi:10.18653/v1/2024.emnlp-main.781. https://aclanthology.org/2024.emnlp-main.781/.

[49] Constantin Venhoff et al. (2025). Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=qeL8fi8GS7.

[50] Angela Fan et al. (2021). Addressing Some Limitations of Transformers with Feedback Memory. arXiv:2002.09402 [cs.CL]. https://arxiv.org/abs/2002.09402. arXiv:2002.09402.

[51] Bulatov et al. (2022). Recurrent Memory Transformer. In Advances in Neural Information Processing Systems. pp. 11079–11091. https://proceedings.neurips.cc/paper_files/paper/2022/file/47e288629a6996a17ce50b90a056a0e1-Paper-Conference.pdf.

[52] Hutchins et al. (2022). Block-Recurrent Transformers. In Advances in Neural Information Processing Systems. pp. 33248–33261. https://proceedings.neurips.cc/paper_files/paper/2022/file/d6e0bbb9fc3f4c10950052ec2359355c-Paper-Conference.pdf.

[53] Alexis Chevalier et al. (2023). Adapting Language Models to Compress Contexts. arXiv:2305.14788 [cs.CL]. https://arxiv.org/abs/2305.14788. arXiv:2305.14788.

[54] Allan Jabri et al. (2023). Scalable Adaptive Computation for Iterative Generation. arXiv:2212.11972 [cs.LG]. https://arxiv.org/abs/2212.11972. arXiv:2212.11972.

[55] Yinpeng Chen et al. (2025). MELODI: Exploring Memory Compression for Long Contexts. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=TvGPP8i18S.

[56] Nasim Borazjanizadeh and James McClelland (2026). Modeling Language as a Sequence of Thoughts. arXiv:2512.25026 [cs.CL]. https://arxiv.org/abs/2512.25026. arXiv:2512.25026.

[57] Katharopoulos et al. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. In Proceedings of the 37th International Conference on Machine Learning. pp. 5156–5165.

[58] Imanol Schlag et al. (2021). Linear Transformers Are Secretly Fast Weight Programmers. arXiv preprint arXiv:2102.11174.

[59] Albert Gu and Tri Dao (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. In First Conference on Language Modeling. https://openreview.net/forum?id=tEYskw1VY2.

[60] Zeyuan Allen-Zhu (2025). Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. arXiv:2512.17351 [cs.CL]. https://arxiv.org/abs/2512.17351. arXiv:2512.17351.

[61] Songlin Yang et al. (2025). PaTH Attention: Position Encoding via Accumulating Householder Transformations. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=ZBlHEeSvKd.

[62] Bo Peng et al. (2025). RWKV-7 "Goose" with Expressive Dynamic State Evolution. In Second Conference on Language Modeling. https://openreview.net/forum?id=ayB1PACN5j.

[63] Yu Sun et al. (2025). Learning to (Learn at Test Time): RNNs with Expressive Hidden States. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=wXfuOj9C7L.

[64] Julien Siems et al. (2025). DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=SoRiaijTGr.

[65] William Merrill et al. (2025). The Illusion of State in State-Space Models. arXiv:2404.08819 [cs.LG]. https://arxiv.org/abs/2404.08819. arXiv:2404.08819.

[66] Zhiyuan Li et al. (2024). Chain of Thought Empowers Transformers to Solve Inherently Serial Problems. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=3EWTEy9MTM.

[67] William Merrill and Ashish Sabharwal (2024). The Expressive Power of Transformers with Chain of Thought. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=NjNGlPh8Wh.

[68] Shibo Hao et al. (2025). Training Large Language Models to Reason in a Continuous Latent Space. In Second Conference on Language Modeling. https://openreview.net/forum?id=Itxz7S4Ip3.

[69] Alexia Jolicoeur-Martineau (2025). Less is More: Recursive Reasoning with Tiny Networks. arXiv:2510.04871 [cs.LG]. https://arxiv.org/abs/2510.04871. arXiv:2510.04871.

[70] Turner et al. (2023). Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.

[71] Zou et al. (2023). Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.

[72] Gao et al. (2025). Scaling and evaluating sparse autoencoders. In International Conference on Learning Representations. pp. 26721–26754.

[73] Marks, Samuel and Tegmark, Max (2023). The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824.

[74] Arditi et al. (2024). Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems. 37. pp. 136037–136083.

[75] Rimsky et al. (2024). Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 15504–15522.

[76] Gemma Team et al. (2024). Gemma: Open Models Based on Gemini Research and Technology. arXiv:2403.08295 [cs.CL]. https://arxiv.org/abs/2403.08295. arXiv:2403.08295.

[77] Alexandre Galashov et al. (2025). Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production. arXiv:2510.13879 [cs.CL]. https://arxiv.org/abs/2510.13879. arXiv:2510.13879.

[78] Edward Beeching et al. (2023). Open LLM Leaderboard (2023-2024). https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard.

[79] Gao et al. (2024). The Language Model Evaluation Harness. Zenodo. https://zenodo.org/records/12608602. doi:10.5281/zenodo.12608602.

[80] Clark et al. (2018). Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457.

[81] Dan Hendrycks et al. (2021). Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR).

[82] Sakaguchi et al. (2021). Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM. 64(9). pp. 99–106.

[83] Clark et al. (2019). BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In NAACL.

[84] Yonatan Bisk et al. (2020). PIQA: Reasoning about Physical Commonsense in Natural Language. In Thirty-Fourth AAAI Conference on Artificial Intelligence.

[85] Zellers et al. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

[86] Paperno et al. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long papers). pp. 1525–1534.

[87] Kojima et al. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems. 35. pp. 22199–22213.

[88] Jason Wei et al. (2023). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL]. arXiv:2201.11903.

[89] Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168.

[90] Yang Yue et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. pp. 57654–57689. https://openreview.net/forum?id=4OsgYD7em5.

[91] Norman, Donald A. (1999). Affordance, conventions, and design. Interactions. 6(3). pp. 38–43. doi:10.1145/301153.301168. https://doi.org/10.1145/301153.301168.

[92] Liu et al. (2020). Understanding the Difficulty of Training Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 5747–5763. doi:10.18653/v1/2020.emnlp-main.463. https://aclanthology.org/2020.emnlp-main.463/.

[93] Xiong et al. (2020). On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning.

[94] Team et al. (2026). Gemma 4 technical report. arXiv preprint arXiv:2607.02770.

[95] Kim et al. (2025). Peri-ln: Revisiting normalization layer in the transformer architecture. arXiv preprint arXiv:2502.02732.

[96] Sun et al. (2026). The curse of depth in large language models. Advances in Neural Information Processing Systems. 38. pp. 163104–163136.

[97] Ilya Loshchilov and Frank Hutter (2019). Decoupled Weight Decay Regularization. In International Conference on Learning Representations. https://openreview.net/forum?id=Bkg6RiCqY7.