Separating Representation from Reconstruction Enables Scalable Text Encoders

Megi Dervishi $^{1}$ $^{2}$ $^{*}$
$^{1}$ FAIR Meta
$^{2}$ Paris Dauphine University

Mathurin Videau $^{1}$ $^{*}$
$^{1}$ FAIR Meta

Yann LeCun $^{3}$
$^{3}$ New York University

Abstract

While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the representations of BERT encoders become increasingly unexploitable by frozen probes, despite improved perplexity. The misalignment originates in BERT's flat design, which couples representation learning to the token reconstruction loss. We propose CrossBERT, a two-part architecture that separates the learning of high-quality encoded representations from the rigid grounding of token reconstruction. This design further enables high masking ratios ($\ge 50%$) and gradient collection over all tokens via a Complementary Masking Strategy, respectively increasing throughput by $1.5$ to $2\times$ and sample efficiency by $2\times$. Overall, CrossBERT demonstrates monotonic scaling and superior performance on MTEB(eng, v2) and frozen GLUE benchmarks.

$^{*}$ Equal contribution. Correspondence to: Megi Dervishi [email protected], Mathurin Videau [email protected].

Executive Summary: Executive Summary

Text encoders such as BERT power retrieval, recommendation, and retrieval-augmented generation systems, yet their architectures have changed little since 2018 while decoders have scaled rapidly. The core problem is that standard BERT-style models produce representations that become progressively less useful when kept frozen, even as perplexity improves. This limits their value in settings where heavy fine-tuning is costly or impractical.

The work set out to diagnose why flat BERT encoders fail to scale under frozen evaluation and to test whether separating representation learning from token reconstruction would restore monotonic gains. Researchers introduced CrossBERT, a bipartite architecture in which a deep encoder produces contextual features and a shallow cross-attention predictor handles masked-token reconstruction. They evaluated this design through systematic frozen probing on GLUE and MTEB benchmarks across compute budgets from 2 × 10^18 to 1 × 10^21 FLOPs, using only the MS-MARCO corpus for contrastive stages.

At comparable scale, CrossBERT matched or exceeded BERT on full fine-tuning while delivering markedly stronger frozen performance: linear probes on GLUE rose by roughly six points, and frozen MTEB scores retained nearly all of the full-tuning result instead of dropping more than ten points. The bipartite design tolerated 50 % masking without loss, enabling 1.5–2× training throughput and a Complementary Masking Strategy that doubled sample efficiency by exposing every token in each batch. Standard BERT representations degraded sharply at larger scales; CrossBERT continued to improve, eventually allowing frozen adaptation to surpass full fine-tuning of the baseline.

These results matter because high-quality frozen embeddings reduce the need for expensive downstream adaptation, cut training costs, and improve reliability in retrieval and embedding applications. The findings also show that architectural separation, rather than objective choice alone, drives the gains.

The authors recommend testing alternative reconstruction objectives within the same bipartite structure and comparing it to T5-style encoder–decoder pre-training. Further scaling beyond a few billion parameters and validation on larger, more diverse corpora would strengthen confidence before widespread adoption. The analysis rests on controlled experiments up to modest sizes, so extrapolation to frontier scales carries uncertainty.

1. Introduction

Section Summary: Recent advances in text encoders have largely relied on scaling data rather than improving core architectures like BERT, with heavy dependence on downstream fine-tuning that obscures weaknesses in the initial pre-training. Drawing from progress in vision models, where frozen evaluations encourage better representation learning, this work identifies how standard BERT-style models degrade on such benchmarks as they scale. To address this, the authors propose CrossBERT, a bipartite encoder design that uses a lightweight cross-attention predictor to separate feature extraction from token reconstruction, yielding stronger scaling behavior and more versatile frozen embeddings even under data constraints.

Encoders are critical for a variety of modern applications, ranging from large-scale data curation and retrieval-augmented generation to recommendation systems. However, encoder architectures have remained largely unchallenged since BERT, most improvements stem from scaling training datasets. The research community has focused on elaborate post-training pipelines utilizing pre-trained models merely as initialization ([1]). This heavy reliance on downstream finetuning does not show the shortcomings of the pre-trained backbone, hindering its development. [2] recently demonstrated the cost of neglecting pre-training: frontier pre-trained backbones like ModernBERT [3] and NeoBERT [4] are vastly overtrained relative to their size.

Concurrently, the vision encoder community has continued to improve pre-trained backbones via scaling strategies [5, 6, 7, 8] and novel training recipes [9, 10, 11, 12, 13, 14, 15, 16]. Since these encoders are often utilized frozen, i.e. without modifications, evaluation standards have naturally prioritized frozen benchmarks. Therefore encouraging pre-training innovations. Effective representation learning is defined by the interpretability and versatility of frozen embeddings.

Motivated by recent architectural insights in the vision community, we revisit the design and evaluation of text encoders. Specifically, we conduct a comprehensive scaling analysis, evaluating the representations of frozen pretrained backbones alongside standard finetuning protocols. This shift in perspective reveals a counterintuitive phenomenon: as modern BERT models scale, their features become increasingly unexploitable by frozen probes. To resolve this bottleneck, we introduce CrossBERT. Inspired by [16, 9], CrossBERT decouples representation learning from token reconstruction by appending a lightweight cross-attention predictor to the final backbone layers. This architectural shift ensures that the backbone focuses entirely on feature extraction, while token reconstruction is isolated within the predictor. We demonstrate that this novel design preserves representation quality as the model scales. Furthermore, even when evaluated in a data-constrained contrastive setup using only MS-MARCO, CrossBERT mitigates saturation and exhibits a significantly superior scaling trends compared to standard baselines.

Throughout the paper, our baseline BERT is a modern implementation (see Section 5.1). BERT and CrossBERT differ only on the architecture structure: flat vs bipartite. In Section 2 we explain our conjecture for the failure mode of the BERT architecture and how CrossBERT solves it. In Section 3 we detail the architecture of CrossBERT and present its advantages compared to the usual BERT models. In Section 4 we summarize the new frozen evaluation methods that we use to measure solely the pre-training performance. Finally we present our experimental results and discussion in Section 5 and conclude with future work in Section 8.

Contributions

• C1. The CrossBERT Architecture. We introduce a bipartite encoder that ensures consistent performance scaling on frozen evaluations. To the best of our knowledge, this is the first Masked Autoencoder for text.
• C2. High-Efficiency Training via High Masking Ratio. We demonstrate CrossBERT's robustness at masking ratios
  gt;50%$, accelerating training throughput by $\approx 1.5\text{--}2\times$. Furthermore, this tolerance enables a Complementary Masking Strategy (CMS), which processes the inverse mask in parallel. Effectively doubling the sample efficiency by collecting gradients from every token in the sequence.
• C3. Scaling Laws under Frozen Evaluation. We conduct a scaling analysis (2 x 10^18 to 1 x 10^21 FLOPs) to quantify intrinsic representation quality. This exposes a performance gap in standard BERTs and validates CrossBERT's superior extractability, an advantage that we show extends to MTEB tasks.

2. CrossBERT: Intuition

Section Summary: The section argues that BERT's representations degrade with increased compute because its masked language modeling objective, combined with a single flat architecture, forces the model to tie its internal features too closely to the narrow task of guessing individual missing tokens rather than building broadly useful abstractions. Probing experiments support this view, showing that earlier layers sometimes outperform the final output. To fix the issue, the authors introduce CrossBERT, a two-part design that uses one component to create rich representations and a separate component to handle token reconstruction, inspired by similar separation techniques in computer vision.

Problem.

BERT representations worsen as we scale compute as can be seen in Figure 1.

Explanation.

We conjecture that the reason behind such degradation stems from the Masked Language Modeling (MLM) objective [17] applied on the flat design of the BERT architecture.

MLM trains an encoder to reconstruct a corrupted input sequence. Given a sequence of tokens $X = {x_1, \dots, x_N}$, a subset of indices $\mathcal{M}$ is selected for masking. The tokens at these positions are replaced by a special token $\texttt{\textless{}MASK\textgreater{}}$, yielding a corrupted sequence $\tilde{X}$. The model processes $\tilde{X}$ to generate contextualized representations, and the objective is to minimize the negative log-likelihood of the original tokens $x_m$ at the masked positions:

$ \mathcal{L}{\text{MLM}} = - \sum{m \in \mathcal{M}} \log P(x_m \mid \tilde{X})\tag{1} $

where $P(x_m \mid \tilde{X})$ denotes the probability assigned to the true token $x_m$ by the prediction head. Hence the objective only measures the token reconstruction ability of the model but not the actual quality of its representations.

Since the "flat" design of the BERT architecture does not explicitly separate representation creation from token reconstruction, the representations remain overly "grounded" in the local signal required to predict missing tokens instead of being versatile high-level abstractions. Some evidence of this phenomenon is displayed in Table 1, where the BERT encoder is probed at different depths. Notably, retrieving representations from earlier layers, rather than the final output, yields slightly improved performance, showcasing the over-specialization of these last layers.

Solution.

Inspired by Masked Auto-Encoders (MAE) approaches in vision ([16, 9]), we propose CrossBERT : a bipartite architecture that separates the heavy lifting of representation creation (Encoder) from the specific task of token reconstruction (Predictor), as illustrated in Figure 2.

3. CrossBERT: Architecture & Advantages

Section Summary: CrossBERT splits each input sequence into visible tokens fed to a deep encoder and masked tokens handled by a shallow predictor that only cross-attends to the encoder outputs, using rotary position embeddings to maintain order. This separation allows the model to tolerate much higher masking ratios than standard BERT without losing representation quality, which in turn enables a complementary masking strategy that processes every token in the sequence exactly once per batch. The result is substantially improved sample efficiency, lower training cost, and a reusable predictor module that serves as an effective interface for later tasks.

3.1 Architecture

A sketch of the architecture is shown in the right panel of Figure 2. The input sequence is partitioned into a visible set (processed by the encoder) and a masked set (processed by the predictor). To preserve sequence order, we encode the position of each token through RoPE, ensuring the encoder and predictor are aware of which positions are missing. The predictor is a few transformer blocks that only cross-attend to the encoder representations. By removing self-attention between masked tokens, we force the predictor to act strictly as a "readout" interface that must satisfy its objective solely by querying the encoder’s embeddings. Additionaly we implement modern architectural optimizations [3, 4] such as RMSNorm.

To set the size of the encoder and predictor we align with MAE [16]. The predictor shares the encoder's hidden dimension, but is significantly shallower (approximately one-forth of the encoder's depth). Ablation on the predictor's shape and size can be found in Appendix A. We gain two insights from this ablation. First, the specific aspect ratio (width vs. depth) of the predictor is not significant. Second, while increasing the predictor's capacity can yield some performance gains (with diminishing returns), it comes at the cost of slower training. Since our objective is to obtain rich frozen features, we decide to allocate most of the compute to the encoder.

::: {caption="Table 1: Layer-wise Frozen GLUE Score Analysis on BERT. The scores are obtained by fitting a linear probe on the frozen features as explained in Section 4.1"}

:::

3.2 Emerging Advantages

This design choice leads to several advantages: predictor's transfer learning; robustness to higher masking ratios; the ability to use the Complementary Masking Strategy (CMS); better data efficiency and lower training costs.

Transferability.

The predictor learns to extract information only from the encoder representation during pre-training. Hence, the predictor functions as a learned pooling mechanism. It can be effectively re-used as a warm-started module for downstream finetuning, serving as an efficient bridge between the frozen encoder features and the target task.

Robustness to higher masking ratios.

BERT architectures typically suffer performance degradation when masking ratios exceed $20\text{--}40%$ [18] (see Appendix C). We challenge this limitation by profiling the frozen representation quality of CrossBERT across multiple masking ratios. As shown in Table 2, CrossBERT exhibits remarkable stability: increasing masking from $20%$ to $50%$ incurs a negligible drop in GLUE performance ($-0.7%$). This resilience confirms that our bipartite architecture successfully insulates representation learning from the difficulty of the reconstruction task. Increasing to higher masking ratios directly translates into reduced training costs, which opens the door to a Complementary Masking Strategy (CMS).

: Table 2: Average Frozen GLUE score of CrossBERT across different masking ratios keeping the same data budget.

Masking ratio (%)	20	40	50	65
CrossBERT	73.8	73.7	73.1	72.4

CMS.

Complementary Masking Strategy is depicted in the left panel of Figure 2. Every batch is augmented with its inverse mask, i.e. creating two complementary views of the original token sequence. This allows the model to predict and learn from every token in the sequence. The model ensures that information from the original sequence does not "leak" into the complementary view within the same batch by applying a unique attention mask on each view. Hence, both views can be processed simultaneously. For BERT, this approach does not mix well with the 20% – 40 % masking ratio requirement as the inverse view would land on the 60% – 80% ratio. Because CrossBERT remains effective beyond $50%$ masking, it can learn from both the original and the complementary sequence efficiently.

Sample efficiency.

MLM is known for being sample-inefficient; the model learns only from a fraction of tokens per pass since gradients are only computed on masked tokens. However by utilizing CMS, CrossBERT processes the visible tokens and its inverse, effectively seeing all the tokens in the sequence and predicting all the tokens from the predictor in one go. Empirically, our results (Appendix B) confirm that CMS does not damage the performance of the baseline compared to standard masking. Instead, CMS effectively doubles sample efficiency, requiring half the training data to reach the same performances.

Computational efficiency.

Even if BERT were to employ CMS, the computational cost would be prohibitively expensive: two forward-backward passes on the full sequence. On the other hand, the encoder of CrossBERT drops masked tokens. Hence the cost of training with CMS is just one forward-backward pass of the encoder and predictor on the full sequence.

\begin{tabular}{r|lll|ccccccccc}
\toprule
\multicolumn{12}{c}{\textbf{GLUE}} \\
\midrule
\textbf{Model} & \textbf{Params} & \textbf{FLOPS} & \textbf{tps} & \textbf{MNLI} & \textbf{QNLI} & \textbf{QQP} & \textbf{RTE} & \textbf{SST} & \textbf{MRPC} & \textbf{CoLA} & \textbf{STS} & \textbf{Avg.} \\
 \midrule
 BERT & 239M & 6.8e19 & 123k & 86.5 & 90.0 & 88.5 & 85.3 & 95.1 & 91.9 & 63.0 & 91.2 & 86.4\\
CrossBERT & 279M & 4.1e19 & 207k & 86.4 & 90.3 & 88.4 & 85.1 & 94.8 & 91.4 & 65.5 & 91.5 & 86.6 \\
Electra & 258M & 7.5e19 & 116k & 88.7 & 93.3 & 89.1 & 83.4 & 94.6 & 92.0 & 71.0 & 90.0 & 87.8\\
\midrule
\textit{ModernBERT} & 352M & 4.9e21 & - & 90.8 & 95.2 & 92.7 & 92.1 & 97.1 & 91.7 & 71.4 & 92.8 & 90.5\\
\textit{NeoBERT} & 198M & 2.9e21 & - & 88.9 & 93.9 & 90.7 & 91.0 & 95.8 & 93.4 & 64.8 & 92.1 & 88.8\\
\textit{OptiBERT} & 239M & 7.0e19 & - & 86.6 & 92.1 & 90.3 & 83.2 & 92.6 & 91.0 & 59.6 & 90.8 & 85.8\\
\textit{DeBERTaV3} & 304M & >2e20 & - &91.9 & 96.0 & 93.0 & 92.7 & 96.9 & 91.9 & 75.3 & 93.0 & \textbf{91.4}\\
\midrule
\multicolumn{12}{c}{\textbf{GLUE Linear probe}} \\
\midrule
BERT & 239M & 6.8e19 & 123k & 58.6 & 74.8 & 77.7 & 59.9 & 84.9 & 76.7 & 27.4 & 79.5 & 67.4 \\
CrossBERT & 279M & 4.1e19 & 207k & 61.6 & 81.8 & 81.4 & 64.3 & 90.6 & 77.9 & 47.9 & 84.8 & 73.8 \\
Electra & 258M & 7.5e19 & 116k & 69.2 & 83.6 & 82.7 & 66.8 & 86.9 & 79.6 & 58.8 & 87.5 & 76.9 \\
\midrule
\textit{ModernBERT} & 352M & 4.9e21 & - & 63.7 & 80.0 & 80.7 & 61.0 & 86.7 & 74.0 & 40.7 & 83.0 & 71.2 \\
\textit{NeoBERT} & 198M & 2.9e21 & - & 47.9 & 69.8 & 72.9 & 56.0 & 78.0 & 69.1 & 10.1 & 66.1 & 58.7 \\
\textit{DeBERTaV3} & 304M & >2e20 & - &76.6 & 86.9 & 85.8 & 75.8 & 86.5 & 81.9 & 67.1 & 90.1 & \textbf{81.3} \\
\midrule
\multicolumn{12}{c}{\textbf{MTEB(eng, v1) Full Contrastive Finetuning on MS-MARCO}} \\
\midrule
  {} & & & & \textbf{Class.} & \textbf{Clust.} & \textbf{PairClass.} & \textbf{Rerank.} & \textbf{Retriev.} & \textbf{STS} & \textbf{Summ.} & \textbf{Avg.} & \textbf{Overall}\\
\midrule
BERT & 239M & 6.8e19 & 123k & 62.7 & 35.1 & 80.7 & 51.5 & 43.0 & 75.8 & 29.8 & 54.1 & 53.9 \\
CrossBERT & 279M & 4.1e19 & 207k & 67.6 & 31.5 & 80.5 & 52.2 & 42.5 & 75.3 & 31.6 & \textbf{54.5} & \textbf{54.1} \\
Electra & 258M & 7.5e19 & 116k & 62.1 & 27.7 & 77.9 & 48.8 & 34.3 & 72.4 & 29.1 & 50.3 & 49.0\\
\midrule
ModernBERT$^*$ & 352M & 4.9e21 & - & 62.4 & 38.7 & 65.5 & 50.1 & 23.1 & 68.3 & 27.8 & 46.9 & - \\
NeoBERT$^*$ & 198M & 2.9e21 & - & 61.6 & 40.8 & 76.2 & 51.2 & 31.6 & 74.8 & 30.7 & 51.3 & - \\
OptiBERT$^\times$ & 239M & 7.0e19 & - & 67.5 & 36.1 & 75.8 & 48.1 & 23.3 & 79.7 & 30.0 & 51.5 & - \\
DeBERTaV3$^*$ & 304M & >2e20 & - & 45.9 & 16.4 & 45.0 & 40.8 & 4.0 & 40.1 & 29.9 & 27.1 & - \\  
\midrule
\multicolumn{12}{l}{$^*$ Full finetuning on much larger dataset including MSMARCO, StackOverFlowDupQuestion, Fever, STS12, and STSBenchmark and AllNLI.}\\
\cmidrule(lr){1-1}
\multicolumn{12}{l}{$^\times$ Full finetuning on AllNLI only.}\\
\midrule
\multicolumn{12}{c}{\textbf{MTEB(eng, v1) Contrastive Finetuning over frozen features on MS-MARCO}}\\
\midrule
BERT & 239M & 6.8e19 & 123k & 60.1 & 29.7 & 64.6 & 43.7 & 19.2 & 62.7 & 30.5 & 44.3 & 42.1 \\
CrossBERT & 279M & 4.1e19 & 207k & 65.7 & 37.1 & 78.0 & 51.4 & 40.7 & 72.3 & 30.7 & \textbf{53.7} & \textbf{53.6} \\
Electra & 258M & 7.5e19 & 116k & 41.3 & 12.8 & 35.4 & 32.4 & 0.0 & 32.8 & 26.9 & 26.2 & 22.4\\
\bottomrule
\end{tabular}

4. Method

Section Summary: To prove that CrossBERT produces stronger pretrained representations than standard BERT, the authors evaluate models by freezing the encoder weights entirely and testing only how well those fixed features perform on downstream tasks, rather than allowing full finetuning that could mask weaknesses. They adapt two benchmarks—GLUE for classification via simple linear and nearest-neighbor probes, and the larger MTEB suite for embedding quality via contrastive training of a lightweight adapter on top of the frozen encoder—to isolate representation quality. The approach also examines how performance and optimal settings scale predictably with total pretraining compute across model sizes.

In this section we aim to describe how we prove that CrossBERT works better than the current BERT recipe.

Historically, encoder evaluations have relied heavily on full finetuning for downstream tasks ranging from classification to information retrieval. Doing so, makes it hard to judge whether the final downstream performance is a result of finetuning or original pretrained representation quality. Therefore, we evaluate performance by freezing the encoder, which allows us to directly measure the pretrained representation quality.

We focus on two evaluation benchmarks. While GLUE [19] served as the gold standard for early pre-training, the field, especially within the contrastive finetuning landscape, has since adopted the MTEB benchmark [20] to better assess embedding quality. We now describe how to adapt these established benchmarks to a frozen evaluation protocol.

4.1 GLUE frozen evaluation

Given the historical importance of classification in encoder evaluation, we use classification tasks to probe feature quality throughout the training process. Specifically, the output of the last layer is averaged across all tokens to generate a single representation of the sequence. A classifier is then optimized on these frozen features. To ensure efficiency, these probes are fitted in parallel, necessitating only a single forward pass over the training set. Two distinct types of classifiers are used:

Linear probing.

The linear probes are optimized via Ridge Regression, which minimizes the standard least-squares error augmented by a $L_2$ penalty term. The loss function is defined as:

$ \mathcal{L}_{\text{Ridge}}(\mathbf{W}) = | \mathbf{Y} - \mathbf{X}\mathbf{W} |_F^2 + \lambda | \mathbf{W} |_F^2\tag{2} $

where $\mathbf{X}$ represents the frozen encoder representations, $\mathbf{Y}$ the target labels, and $\lambda$ the regularization coefficient controlling the penalty strength. The optimal weights $\mathbf{W}^*$ are computed directly via the closed-form analytical solution:

$ \mathbf{W}^* = (\mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I})^{-1} \mathbf{X}^\top \mathbf{Y}\tag{3} $

This approach allows for rapid, deterministic fitting across multiple regularization strengths without the need for iterative optimization. Moreover, this procedure is done on GPU leveraging torch.linalg.solve making it even faster. In practice, we sweep multiple logspaced $\lambda$ as we notice that some dataset, and especially small ones, are very sensitive to this hyperparameter.

kNN probing.

As a non-parametric complement to linear evaluation, k-Nearest Neighbors (kNN) assess the intrinsic geometry of the representation space. For a given query $z$, the prediction is determined by a majority vote among the set of its $k$ closest neighbors $\mathcal{N}_k(z)$, identified by minimizing the distance metric $d$ (either $L_2$ or Cosine):

$ \begin{aligned} \mathcal{N}k(z) &= \underset{\mathcal{S} \subset \mathcal{D}, |\mathcal{S}|=k}{\operatorname{arg, min}} \sum{x_j \in \mathcal{S}} d(z, x_j) \ \hat{y} &= \operatorname*{arg, max}{c \in \mathcal{C}} \sum{x_i \in \mathcal{N}_k(z)} \mathbb{1}(y_i = c) \end{aligned}\tag{4} $

where $\mathcal{D}$ is the set of all evaluation points and $\mathcal{C}$ the set of all classes. The implementation runs on GPU and is heavily based on the released code of CAPI [12].

In practice, these evaluations are computationally negligible. For a 250M parameter model, the entire linear and kNN probing process on GLUE takes less than 5 minutes on a single H100. The cost is dominated by the forward pass over the dataset.

4.2 MTEB frozen evaluation

We evaluate the richness and adaptability of our frozen representation using the Massive Text Embedding Benchmark [20], which spans seven distinct task downstream categories: Classification, Clustering, Pair Classification, Semantic Textual Similarity (STS), Reranking, Retrieval, and Summarization.

Contrastive probing.

The above downstream tasks require sentence/document representations and a coherent, well-structured representation space, which is not tackled by the pre-training objective. Contrastive finetuning aims to address these gaps. While existing literature typically performs this via contrastive finetuning on an unfrozen backbone, we explore two configurations: a standard unfrozen backbone and a frozen-backbone approach where only an attention based pooler is optimized on top of fixed features.

The token-level encoded representations are pooled into a single sentence/document representation via a learnable lightweight adapter, consisting of a few Transformer blocks with cross-attention. Let the pooled representations of a passage (resp. query) be $p$ (resp. $q$). Given a query $q$, the contrastive loss, aims to pull closer a related passage $p$ and the query $q$ in representation space while pushing away semantically similar unrelated passages $p^-$ (hard-negatives). The contrastive loss, as defined in [10], is

$ \mathcal{L}{c} = -\frac{1}{N} \sum{i=1}^{N} \log \frac{e^{ \phi(q_i, p_i) }}{ e^{ \phi(q_i, p_i) } + \sum_{n \in \mathcal{S}i} e^{ \phi(q_i, p{in}^-) } }\tag{5} $

where $\phi(q_i, p_i)$ is the cosine similarity between a query $q_i$ and a passage $p_i$, and $p_{in}^-$ denotes the hard negatives for each query.

Since CrossBERT already features an adaptable component (the predictor), we re-purpose it to warm-start the adapter for the contrastive learning phase, while keeping the encoder frozen. As detailed in our ablation study (Appendix D), this strategy accelerates convergence and improves MTEB scores by $\approx 2$ %.

4.3 Evaluations at different scales

Model performance and optimal hyperparameters follow predictable power-law trends relative to the total compute budget $C$ [21, 22]. We define $C$ as:

$ C = F_N D\tag{6} $

where $D$ is the number of pre-training tokens and $F_N$ are the FLOPs per token for a forward-backward pass. For a standard BERT transformer with sequence length $S$, layers $L$, and non-embedding parameters $N$ [2] we have:

$ F_N = 6 N + 12 d L S\tag{7} $

However, CrossBERT modifies $F_N$ based on the masking ratio $r$, linearly interpolating between the encoder ($F^{\text{enc}}_N$) and the cross-attention predictor ($F^{\text{pred}}_N$):

$ F_N = F^{\text{enc}}_N (1-r) + F^{\text{pred}}_N r\tag{8} $

This demonstrates that as the masking ratio increases, the total computational cost linearly interpolates between the encoder and the predictor. To scale up we need to increase model sizes $F_N$ and dataset sizes $D$ in tandem.

5. Experiments

Section Summary: In the experiments, researchers trained three related models—standard BERT, Electra, and a new bipartite architecture called CrossBERT—on a large shared text corpus while testing different training objectives, data masking rates, and hyperparameter scaling rules fitted from smaller sweeps. CrossBERT ran substantially faster during training and required far less total compute than comparable models, yet it matched or exceeded baselines on GLUE language-understanding tasks and achieved the highest score on the MTEB retrieval benchmark after contrastive fine-tuning. Additional linear-probing and scaling tests across model sizes confirmed that the bipartite design preserved strong performance even when the backbone remained frozen.

5.1 Setup

Models.

We train three models that share the same corpus, tokenizer, and encoder backbone: BERT, Electra, and CrossBERT. BERT and Electra are flat, whereas CrossBERT is bipartite (Section 3). BERT and CrossBERT use the MLM objective of Section 2, while Electra replaces it with replaced-token detection (RTD) [23]: a small auxiliary generator substitutes a fraction of the input tokens with plausible alternatives, and the encoder predicts at each position whether the token is original or replaced. For context, we also report the published results of ModernBERT [3], NeoBERT [4], DeBERTaV3 [24], and OptiBERT [2], all of which are trained with substantially more data and compute.

Data.

We choose a subset of DCLM [25] as our training corpus ($\approx 4\text{T}$ tokens). Across all experiments, the masking ratio is set to $20%$ for BERT and $50%$ for CrossBERT. Additionally, all CrossBERT models leverage CMS (see Section 3.2). All models rely on the RoBERTa tokenizer [26].

Codebase.

Our codebase is based on PyTorch [27] and Lingua [28], opting for Fully Sharded Data Parallelism (FSDP) and torch.compile for maximum throughput and reduced memory footprint. Handling Dynamic Shapes: Compilation requires static graphs, which conflicts with random MLM masking. We resolve this by enforcing a fixed count of masked tokens per GPU (Per GPU Batch size $\times$ Mask Ratio) and permuting each mask in the sequence via random permutations (torch.randperm). This ensures static tensor shapes without compromising masking randomness.

Hyperparameters.

Following [22], we fit power laws for both Batch Size (BSZ) in total tokens and Learning Rate (LR) sweeping model sizes ranging from (50M to 700M) with a LR (resp. BSZ) logspaced from $10^{-4}$ to $10^{-2}$ (resp. $10^4$ to $5\times10^6$). After keeping only the top 3 performing models for the fitting, we obtain the following results for CrossBERT:

$ \begin{aligned} \text{BSZ}{\text{CrossBERT}}(C) &= 10^4 \times C^{0.092} \ \text{LR}{\text{CrossBERT}}(C) &= 564.6 \times C^{-0.279} \end{aligned} $

For the BERT setup, we simply reuse the hyperparameters found by [2]

$ \begin{aligned} \text{BSZ}{\text{BERT}}(C) &= 17.38 \times C^{0.24} \ \text{LR}{\text{BERT}}(C) &= 69.18 \times C^{-0.24} \end{aligned} $

Appendix H shows the hyperparameter heatmap for the CrossBERT sweeping.

Eval on GLUE.

For Linear Probing, we fit linear heads on frozen features along logarithmic data regimes (${1, \dots, 10^4}$ samples) and L2-regularization strengths ($\lambda \in [1, 10^4]$). For each task we report the best score across all $\lambda$ for the largest available sample size. For kNN Probing, non-parametric evaluation with kNN are conducted using diverse configurations by sweeping across distance metrics ($L_2$, Cosine) and neighborhood sizes ($k \in {1, 3, 10, 30}$). We report the maximum score across all $k$ and distances.

Eval on MTEB.

We use the MS-MARCO training set (500k queries) with hard negatives. No instruction templates are applied. We finetune contrastively for one epoch, with a batch size of 512 and a learning rate of 5 x 10^-5 (cosine decay to 5 x 10^-7). To be comparable with current published models, we use both an unfrozen and frozen backbone.

Single-scale runs.

All three models share the same encoder backbone (28 layers, hidden dimension 768); CrossBERT additionally couples this encoder to a 6-layer cross-attention predictor. The models are trained on 50B tokens with a data-to-model ratio of $35:1$ that deliberately exceeds the compute-optimal $\approx 15:1$ to avoid undertraining. The learning rate is fixed at 6 x 10^-4 and the global batch size at $\approx 393\text{k}$.

Scaling sweep.

[2] showed that the data-to-model ratio $F_N / D$ heavily impacts performance and should be kept around 10:1 to 100:1. In all our scaling experiments we fix the ratio at $20:1$ and generate a suite of models spanning a total compute range from 2 x 10^18 to 1 x 10^21 FLOPs. To optimize performance at every scale, we determine batch size and learning rate by sweeping values across smaller models ($50\text{M}$ to $700\text{M}$) and extrapolating the optimal settings via a power law fitting). In total we train 8 BERT and 9 CrossBERT models, see Appendix J.

5.2 Results: Single-scale Runs

Table 3 reports the single-scale runs of our three models alongside the current literature.

Training efficiency and throughput.

CrossBERT demonstrates superior computational efficiency. In terms of training throughput, it achieves 207k tokens/sec compared to the baseline's 123k tokens/sec, representing a $1.68\times$ speedup in wall-clock time on H100 GPUs. Furthermore, when compared to the existing literature, CrossBERT remains highly competitive despite utilizing approximately $100\times$ less total compute than models like ModernBERT and NeoBERT. Against OptiBERT, a model of similar scale, CrossBERT requires $\approx 40%$ fewer FLOPs while delivering higher performance on both GLUE and MTEB(eng, v1).

Full-finetuning performances.

Despite this massive reduction in compute, CrossBERT does not compromise on quality. On the GLUE benchmark, it matches our robust baseline ($86.6$ vs. $86.4$). More notably, on the MTEB(eng, v1) benchmark (Full Contrastive Finetuning), CrossBERT achieves the highest average score of 54.5, outperforming both NeoBERT ($51.3$) and ModernBERT ($46.9$) which used much more finetuning data.

Robustness of frozen representations.

The most significant advantage of CrossBERT lies in the versatility of its frozen features. We observe three critical behaviors:

• Minimal degradation: When switching from full finetuning to frozen adaptation, the standard BERT baseline suffers a substantial drop of over 10 points on MTEB(eng, v1) ($53.9 \to 42.1$). In contrast, CrossBERT retains 99% of its performance, scoring 53.6 in the frozen setting. This indicates that the pretrained features are easily adaptable.
• Retrieval capability: This robustness is most visible in the Retrieval task, where CrossBERT more than doubles the score of the baseline ($40.7$ vs. $19.2$), proving that the model captures semantic similarity without needing deep task-specific adaptation.
• Linear separability: On the GLUE Linear Probe, CrossBERT surpasses the BERT baseline by a large margin ($+6.4$ points). This confirms that our approach prevents the over-specialization often seen in standard MLMs, producing high-level abstractions that are linearly separable and directly usable for downstream tasks. While a performance gap of $\approx 13$ points remains between the linear probe and full finetuning, this is partly attributable to the simplicity of averaging encoder outputs; utilizing more sophisticated probing mechanisms, such as attention-based pooling, would likely reduce this gap further. Electra scores even higher than CrossBERT on this probe, but this advantage does not carry over to the sentence-embedding tasks examined next.

MLM versus RTD on a flat backbone.

On token-level probes, Electra outperforms BERT and CrossBERT, reaching $87.8$ on GLUE full finetuning and $76.9$ on the GLUE linear probe. However, on sentence-level embedding tasks it collapses. The unfrozen backbone benchmarks at $49.0$ on MTEB(eng, v1) whereas the frozen one drops to $22.4$ with a Retrieval score of $0.0$. This collapse mirrors prior findings that RTD distorts sentence-embedding geometry [29, 3]. Two conclusions follow. First, neither swapping the objective on a flat backbone (Electra) nor keeping MLM on a flat backbone (BERT) yields versatile frozen representations; the bipartite separation of representation from reconstruction is what produces them. Second, GLUE-style classification probes alone cannot diagnose representation quality, since MTEB exposes failures that GLUE hides.

5.3 Results: Scaling Sweep

Figure 1 aggregates the GLUE and MTEB(eng, v2) scores of BERT and CrossBERT, while Figure 4 and Figure 3 break them down per task. Our sweep reveals four insights into the scaling behavior of masked language models.

Degradation of frozen BERT representations.

The most striking trend is the sharp divergence in scaling laws between the two models. While standard BERT benefits from scaling when it is unfrozen and fully finetuned, its frozen performance suffers a brutal drop as model size increases. Specifically, for the two largest configurations (

gt;500$ M parameters), BERT's frozen performance drops below that of the smallest model in the sweep ($10\times$ smaller). Since the data-to-model ratio (20:1) is held above the compute-optimal ratio (15:1), undertraining cannot explain the drop. This confirms a severe misalignment between the standard MLM objective and semantic embedding tasks: as the model scales, it becomes increasingly specialized for token reconstruction at the expense of versatile, high-level abstractions.

Task-specific sensitivity on GLUE.

This degradation manifests non-uniformly across tasks (Figure 4). On the GLUE benchmark, the sudden decline of performance does not necessarily appear at the same point. For example MNLI accuracy drops only for the last model while performances for the rest seem to drop earlier.

MTEB(eng, v2) superior scaling and saturation profiles.

In contrast, CrossBERT exhibits robust, monotonic scaling across all metrics. Most notably, CrossBERT fundamentally alters the relationship between pre-training and adaptation:

• The adaptation gap & equalization on MTEB(eng, v2): Standard BERT requires full finetuning to bridge a massive performance deficit ($\approx15$ points) between its frozen and unfrozen states. While this heavy downstream adaptation can eventually equalize performance, i.e. masking pre-training deficiencies by bringing BERT closer to CrossBERT, it comes at a significant compute cost. In contrast, CrossBERT's frozen representations are naturally aligned, sitting at worse around one point below the fully fine-tuned optimum.
• Structural retrieval capability: We observe a fundamental distinction in retrieval tasks. Standard MLM effectively flatlines near zero at all scales, indicating a structural inability to learn dense retrieval without supervision. In contrast, CrossBERT builds these capabilities naturally, scaling linearly with compute.
• Frozen outperforming unfrozen finetuning: At larger scales, CrossBERT with frozen adaptation begins to outperform even the unfrozen baselines. This indicates that CrossBERT scales more effectively than standard BERT, avoiding early saturation. It produces representations that are naturally richer, rendering the heavy process of unfrozen finetuning unnecessary, and eventually inferior, to a lightweight adaptation of the frozen features.

Training stability.

Self-supervised pre-training can suffer from unstable optimization, so we monitor training across the full scaling range. Two observations indicate that CrossBERT trains stably. First, the pre-training validation loss decreases smoothly and tracks the expected scaling law at every compute budget (Appendix F). Second, downstream probing performance rises steadily over the course of training rather than oscillating or collapsing (Figure 6). We observe no divergence across scales. In contrast higher scale BERT trainings suffer from significant instability and loss spikes (see Appendix I).

6. Related Works

Section Summary: Recent language models such as BERT rely on flat architectures that blend representation learning with text reconstruction, and scaled versions of this design continue to show diminishing improvements. To work around the resulting limitations, many approaches add heavy post-training or specialized token-level objectives, though a more aligned pre-training setup could reduce the need for such fixes. Inspired by vision models that use lightweight, separate decoders for reconstruction, CrossBERT applies an asymmetric design with a simple cross-attention predictor so the encoder can focus on semantic abstraction instead of token prediction.

Standard Encoders.

Since BERT [17] and RoBERTa [26], encoders have relied on flat architectures where representation and reconstruction are mixed. Recent updates like ModernBERT [3] and NeoBERT [4] scale this design but inherit its structural inefficiency. As a result, brute-force scaling of this approach yields diminishing returns for representation quality [2].

Contrastive Learning.

To compensate for this misalignment, the field relies on heavy post-training [30, 1, 31] or token-level objectives like MEXMA [32]. These methods effectively treat the symptoms, but we posit that a better aligned pre-training can reduce the amount of adaptation needed to obtain the final model.

Architecture Design.

In computer vision, Masked Autoencoders [16] established the efficacy of asymmetric designs, where a lightweight decoder reconstructs pixels from highly masked inputs. [9] further demonstrated that reducing interaction between representation and reconstruction, using cross attention only, enhances produced features. In NLP, while T5 [33] also utilizes a bipartite structure, it relies on a heavy decoder optimized for autoregressive text generation employing both self and cross attention. In contrast, CrossBERT adopts asymmetry strictly for representation learning: rather than generating text, we employ a lightweight, cross attention only, predictor solely to offload the reconstruction burden, ensuring the encoder optimizes for semantic abstraction rather than token prediction.

7. Limitations

Section Summary: The study's findings come from experiments rather than a full theoretical explanation of why linking representation and reconstruction tasks harms model features as size increases. All tests used models up to just a few billion parameters, leaving open the possibility of different patterns at the much larger scales common in other systems. The authors also note strong results for token-level tasks with one method but weak sentence embeddings, without exploring the underlying reasons for that gap.

Our analysis is empirical. We identify the failure mode of flat MLM through controlled experiments (Table 1 and Figure 1), but we do not give a theoretical account of why coupling representation and reconstruction degrades frozen features as models grow. Our conclusions also rest on models no larger than a few billion parameters; every trend we observe is monotonic across this range, yet we cannot formally exclude qualitatively different behavior at the much larger scales typical of decoders. Finally, we find that RTD (Electra) yields strong token-level probes but poor sentence embeddings, and a mechanistic explanation of this collapse lies outside the scope of this work.

8. Conclusion & Future Work

Section Summary: The research shifts focus from fully retraining text encoders to evaluating their fixed, off-the-shelf representations, exposing how standard BERT models become less useful as they grow larger because they overfit to word-prediction tasks. CrossBERT addresses this by keeping representation learning separate from token prediction, yielding frozen features that can outperform even fully tuned conventional models at scale. Future work could test alternative pre-training tasks or compare the approach with models like T5.

We revisited the design of text encoders by shifting the evaluation focus from full finetuning to frozen representation quality. This change in perspective revealed that flat BERT-like architectures trained with MLM suffer from a fundamental misalignment: as compute scales, representations become increasingly unexploitable for downstream tasks, overspecializing on reconstruction at the expense of versatility. CrossBERT resolves this by insulating representation learning from the token prediction task. Beyond improving training efficiency, our results show an interesting trend: at large scales, CrossBERT produces frozen features that outperform fully finetuned baselines. This result challenges the expensive downstream contrastive adaptation of standard encoders, demonstrating that the right pre-training incentives can produce significantly richer representations needing less heavy adaptation to be effective.

Several directions follow naturally from these findings. The bipartite design is agnostic to the reconstruction objective, so MLM could be swapped for alternative pretext tasks better suited to representation learning. A systematic comparison with T5-style pre-training, where the predictor is replaced by an auto-regressive decoder would be an interesting avenue to explore.

Acknowledgements

We thank Badr Youbi Idrissi and Jo ao Maria Janeiro for the helpful discussions and feedback that shaped this work.

Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Appendix

Section Summary: The appendix presents a series of ablation studies and supporting experiments that examine design choices in predictor size, masking strategies, and contrastive finetuning for models such as BERT and CrossBERT. It includes tables showing how these choices affect downstream GLUE and MTEB task performance, along with figures that track training dynamics, learning curves, and hyperparameter sweeps. Additional analysis highlights training stability issues that arise when scaling standard architectures to billion-parameter sizes, while noting CrossBERT’s greater robustness under the same conditions.

A. Predictor shape design ablation

Table 4 analyzes the trade-off between predictor size and downstream performance. At the lower bound, the 8M parameter predictor acts as a representational bottleneck, noticeably impairing model quality. However, increasing capacity to just 25M yields a substantial boost. Beyond this point, we observe diminishing returns; scaling further to 50M or 85M incurs higher computational costs for only marginal performance gains. While we ultimately selected the 42M configuration (one-fourth of the encoder depth, grey row) to maintain dimensional alignment with the backbone, the data suggests that the 25M variant remains a highly competitive alternative, promising faster pretraining speed.

::: {caption="Table 4: Impact of varying the shape and size of the predictor. The encoder has 236.8M parameters (768 dim, 28 layers). All model are trained on same exact setup as Table 3"}

:::

B. Complementary Masking ablation

: Table 5: Effect of CMS on GLUE Linear probe. All results are obtained under the same settings of Table 3

Model	CMS	MNLI	QNLI	QQP	RTE	SST2	MRPC	CoLA	STSB	Avg.
CrossBERT 279M	✗	61.4	81.4	81.3	62.8	88.9	74.2	48.7	85.8	73.1
CrossBERT 279M	✓	61.6	81.8	81.4	64.2	90.6	77.9	47.8	84.8	73.8

C. BERT Masking ablation

: Table 6: GLUE scores for BERT models trained with different masking rates.

BERT mask	CoLA	MNLI	MRPC	QNLI	QQP	RTE	SST2	STSB	Avg.
20%	27.4	58.6	76.7	74.8	77.7	59.9	84.9	79.5	67.4
30%	18.8	57.0	73.0	74.1	77.9	54.2	80.7	77.1	64.1
40%	11.0	53.6	69.1	73.5	75.0	57.0	75.1	73.4	61.0
65%	22.6	50.0	67.2	70.0	73.4	55.2	75.1	65.7	59.9

D. Predictor MTEBv2 contrastive finetuning ablation

: Table 7: Contrastive finetuning under frozen backbones on MSMarco for one epochs as described in Section 5.1

Model	Warm-start	Class.	Clust.	PairClass.	Rerank.	Retriev.	STS	Summ.	Avg.	Overall
CrossBERT 279M	✗	68.7	38.3	76.4	43.4	37.8	68.9	28.7	51.8	53.7
CrossBERT 279M	✓	70.5	40.8	77.9	44.1	43.9	71.7	26.7	53.7	56.7

E. MTEB(eng, v2) Results of the Single-scale runs.

\begin{tabular}{r|lll|ccccccccc}
\toprule
\multicolumn{12}{c}{\textbf{MTEB(eng, v2) Full Contrastive Finetuning on MS-MARCO}} \\
\midrule
  {} & & & & \textbf{Class.} & \textbf{Clust.} & \textbf{PairClass.} & \textbf{Rerank.} & \textbf{Retriev.} & \textbf{STS} & \textbf{Summ.} & \textbf{Avg.} & \textbf{Overall}\\
\midrule
BERT & 239M & 6.8e19 & 123k & 67.7 & 39.3 & 80.8 & 43.9 & 45.7 & 75.3 & 28.7 & 54.5 & \textbf{57.3} \\
CrossBERT & 279M & 4.1e19 & 207k & 72.2 & 35.5 & 80.5 & 44.8 & 44.2 & 74.6 & 32.0 & \textbf{54.8} & 57.0 \\
Electra & 258M & 7.5e19 & 116k & 62.1 & 27.7 & 77.9 & 48.8 & 34.4 & 72.4 & 29.1 & 50.3 & 49.0\\
\midrule
\multicolumn{12}{c}{\textbf{MTEB(eng, v2) Contrastive Finetuning over frozen features on MS-MARCO}}\\
\midrule
BERT & 239M & 6.8e19 & 123k & 64.9 & 37.3 & 64.5 & 39.4 & 20.5 & 61.6 & 31.8 & 45.7 & 45.9 \\
CrossBERT & 279M & 4.1e19 & 207k & 70.4 & 40.8 & 77.9 & 44.1 & 43.9 & 71.7 & 26.7 & \textbf{53.6} & \textbf{56.7}\\
Electra & 258M & 7.5e19 & 116k & 43.9 & 29.5 & 35.4 & 33.0 & 0.3 & 31.9 & 19.4 & 27.6 & 26.1\\
\bottomrule
\end{tabular}

F. Learning curve of scaling laws

:::: cols="2"

Figure 5: Learning curves for different BERT(left) and CrossBERT(right) setup specified in model list Appendix J. Each dot represents the validation loss on wikipedia and dclm [25]. ::::

G. Downstream performance evolution during training

H. Learning rate and batch size sweeps

I. Large scale training stability

Because scaling BERT resulted in a severe degradation of downstream performance, we further investigated this collapse across various training setups. To determine if this instability is specific to BERT, we additionally scaled Electra to 1B parameters. As shown in Table 9, both BERT and Electra exhibit a similar performance collapse at the 1B scale. For BERT, altering standard hyperparameters—including data and model seeds, learning rate, and batch size—failed to prevent the degradation. While specific hyperparameter tuning allowed Electra to avoid collapsing, this extreme sensitivity highlights the inherent brittleness of training standard 'flat' architectures at scale.

Conversely, CrossBERT is highly robust, achieving strong out-of-the-box performance without requiring such exhaustive hyperparameter sweeps. When scaling further to 2B parameters, standard configurations initially failed to converge. However, we found that simply reducing the initialization standard deviation from the conventional 0.02 (standard across ViT and BERT implementations) to 0.015 successfully stabilized the training. Furthermore, while Electra historically demonstrates superior performance at smaller scales, CrossBERT 1B significantly outperforms Electra 1B on GLUE, confirming the superior scaling trajectory of our proposed architecture.

::: {caption="Table 9: Zoom on 1B scale model with 1e21 compute on Frozen Glue"}

:::

J. List of Models with hyperparameters

\begin{tabular}{c|cccccc|ccccccccc}
  \toprule
  \textbf{ID} & \textbf{Params (M)} & \textbf{FLOPs} & \textbf{Dim} & \textbf{Layers} & \textbf{Heads} & \textbf{Head Dim} & \textbf{Steps} & \textbf{Tokens (B)} & \textbf{BSZ} & \textbf{GPUs} & \textbf{Grad Acc}& \textbf{LR} & \textbf{Min LR} & $\boldsymbol\beta_1$ & $\boldsymbol\beta_2$ \\
  \midrule
  0 & 76.73 & 2.17e+18 & 576 & 12 & 9 & 64 & 15019 & 6.58 & 438272 & 4 & 1 & 2.75e-03 & 1.0e-6 & 0.90 & 0.95 \\
  1 & 103.28 & 4.64e+18 & 640 & 14 & 10 & 64 & 18268 & 9.63 & 527360 & 1 & 5 & 2.29e-03 & 1.0e-6 & 0.90 & 0.95 \\
  2 & 151.85 & 1.14e+19 & 768 & 16 & 12 & 64 & 24783 & 15.10 & 609280 & 1 & 7 & 1.85e-03 & 1.0e-6 & 0.90 & 0.95 \\
  3 & 226.70 & 2.83e+19 & 896 & 18 & 14 & 64 & 29287 & 23.78 & 812032 & 1 & 13 & 1.48e-03 & 1.0e-6 & 0.90 & 0.95 \\
  4 & 334.06 & 6.73e+19 & 1024 & 22 & 16 & 64 & 36291 & 36.68 & 1010688 & 3 & 7 & 1.21e-03 & 1.0e-6 & 0.90 & 0.95 \\
  5 & 503.81 & 1.65e+20 & 1152 & 28 & 9 & 128 & 46771 & 57.47 & 1228800 & 6 & 5 & 9.72e-04 & 1.0e-6 & 0.90 & 0.95 \\
  6 & 795.27 & 4.24e+20 & 1408 & 30 & 11 & 128 & 58575 & 92.13 & 1572864 & 16 & 3 & 7.75e-04 & 1.0e-6 & 0.90 & 0.95 \\
  7 & 1208.24 & 1.00e+21 & 1664 & 33 & 13 & 128 & 72072 & 141.70 & 1966080 & 40 & 3 & 6.30e-04 & 1.0e-6 & 0.90 & 0.95 \\
  \bottomrule
\end{tabular}

\begin{tabular}{c|ccccccccc|ccccccccc}
\toprule
\textbf{ID} & \thead{Params \\ (M)} & \textbf{FLOPs} & \thead{Enc \\ Dim} & \thead{Enc \\ Layers} & \thead{Enc \\ Heads} & \thead{Enc Head \\ Dim} & \thead{Pred \\ Dim} & \thead{Pred \\ Layers} & \thead{Pred \\ Heads} & \textbf{Steps} & \thead{Tokens \\ (B)} & \textbf{BSZ} & \textbf{GPUs} & \thead{Grad \\ Acc} & \textbf{LR} & \thead{Min \\ LR} & $\boldsymbol\beta_1$ & $\boldsymbol\beta_2$ \\
\midrule
0 & 61.80 & 1.92e+18 & 640 & 15 & 10 & 64 & 640 & 3 & 10 & 13745 & 6.19 & 450560 & 1 & 5 & 4.51e-03 & 1.0e-6 & 0.90 & 0.95 \\
1 & 93.62 & 4.91e+18 & 768 & 17 & 12 & 64 & 768 & 4 & 12 & 19749 & 9.91 & 501760 & 1 & 7 & 3.47e-03 & 1.0e-6 & 0.90 & 0.95 \\
2 & 138.58 & 1.15e+19 & 896 & 19 & 14 & 64 & 896 & 4 & 14 & 26978 & 15.19 & 563200 & 1 & 11 & 2.73e-03 & 1.0e-6 & 0.90 & 0.95 \\
3 & 205.57 & 2.72e+19 & 1024 & 23 & 16 & 64 & 1024 & 5 & 16 & 39850 & 23.34 & 585728 & 1 & 13 & 2.15e-03 & 1.0e-6 & 0.90 & 0.95 \\
4 & 315.61 & 6.83e+19 & 1152 & 29 & 9 & 128 & 1152 & 7 & 9 & 53691 & 36.95 & 688128 & 3 & 7 & 1.67e-03 & 1.0e-6 & 0.90 & 0.95 \\
5 & 494.24 & 1.70e+20 & 1408 & 31 & 11 & 128 & 1408 & 7 & 11 & 79141 & 58.35 & 737280 & 6 & 5 & 1.29e-03 & 1.0e-6 & 0.90 & 0.95 \\
6 & 757.47 & 4.07e+20 & 1664 & 34 & 13 & 128 & 1664 & 8 & 13 & 114660 & 90.17 & 786432 & 16 & 3 & 1.01e-03 & 1.0e-6 & 0.90 & 0.95 \\
7 & 1176.29 & 9.99e+20 & 1920 & 41 & 15 & 128 & 1920 & 10 & 15 & 176106 & 141.38 & 802816 & 56 & 1 & 9.00e-04 & 1.0e-6 & 0.90 & 0.95 \\
8 & 5411.7 & 5.0e21 & 2560 & 52 & 20 & 128 & 2560 & 13 & 20 & 304688 & 319.49 & 1048576 & 128 & 1 & 6.00e-04 & 1.0e-6 & 0.90 & 0.95 \\
\bottomrule
\end{tabular}

References

Section Summary: The references section catalogs dozens of academic papers and preprints that underpin the work, centering on transformer-based models for language and vision tasks. It draws from foundational studies on BERT-style pre-training, contrastive learning, masked modeling, and self-supervised approaches, alongside more recent efforts on efficient encoders, scaling laws, and benchmark evaluations. Sources range from conference proceedings like EMNLP and CVPR to arXiv preprints dated mainly between 2019 and 2025.

[1] Wang et al. (2022). Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.

[2] Dervishi et al. (2025). Training compute-optimal transformer encoder models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 35602–35617. doi:10.18653/v1/2025.emnlp-main.1804. https://aclanthology.org/2025.emnlp-main.1804/.

[3] Warner et al. (2025). Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2526–2547.

[4] Breton et al. (2025). NeoBERT: A Next-Generation BERT. arXiv preprint arXiv:2502.19587.

[5] Bolya et al. (2025). Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504.13181.

[6] Sun et al. (2023). Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389.

[7] Oquab et al. (2023). Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193.

[8] Siméoni et al. (2025). Dinov3. arXiv preprint arXiv:2508.10104.

[9] Fu et al. (2024). Rethinking patch dependence for masked autoencoders. arXiv preprint arXiv:2401.14391.

[10] Chen et al. (2020). A simple framework for contrastive learning of visual representations. In International conference on machine learning. pp. 1597–1607.

[11] Caron et al. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660.

[12] Darcet et al. (2025). Cluster and predict latent patches for improved masked image modeling. arXiv preprint arXiv:2502.08769.

[13] Assran et al. (2023). Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15619–15629.

[14] Bardes et al. (2021). Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906.

[15] Garrido et al. (2024). Learning and leveraging world models in visual representation learning. arXiv preprint arXiv:2403.00504.

[16] He et al. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16000–16009.

[17] Devlin et al. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). pp. 4171–4186.

[18] Wettig et al. (2023). Should you mask 15% in masked language modeling?. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. pp. 2985–3000.

[19] Wang et al. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP. pp. 353–355.

[20] Muennighoff et al. (2023). Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. pp. 2014–2037.

[21] Kaplan et al. (2020). Scaling laws for neural language models. arXiv:2001.08361.

[22] Bi et al. (2024). Deepseek llm: Scaling open-source language models with longtermism. arXiv:2401.02954.

[23] Kevin Clark et al. (2020). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. https://arxiv.org/abs/2003.10555. arXiv:2003.10555.

[24] Pengcheng He et al. (2023). DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. https://arxiv.org/abs/2111.09543. arXiv:2111.09543.

[25] Li et al. (2024). Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems. 37. pp. 14200–14282.

[26] Liu et al. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

[27] Paszke et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems. 32.

[28] Mathurin Videau et al. (2024). Meta Lingua: A minimal PyTorch LLM training library. https://github.com/facebookresearch/lingua.

[29] Rep et al. (2024). Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity. In Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 9159–9169. doi:10.18653/v1/2024.findings-emnlp.535. https://aclanthology.org/2024.findings-emnlp.535/.

[30] Gao et al. (2021). Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821.

[31] Li et al. (2023). Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.

[32] Janeiro et al. (2025). MEXMA: Token-level objectives improve sentence representations. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 23960–23995.

[33] Raffel et al. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research. 21(140). pp. 1–67.