Gemma 4 Technical Report

Gemma Team
Google DeepMind$^1$

Abstract

We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.

$^1$ See Contributions and Acknowledgments section for full author list. Please send correspondence to [email protected].

Executive Summary: Gemma 4 is Google DeepMind’s latest family of open-weight language models. It adds native support for text, images, and audio to the Gemma line while targeting both high reasoning performance and deployment on a wide range of on-device hardware.

The work responds to two practical pressures: closed frontier models continue to pull ahead on complex reasoning and multimodal tasks, yet many organizations need capable alternatives they can run locally, audit, and fine-tune without incurring high inference costs or latency. The report therefore sets out to demonstrate that a single model family can close much of that gap at modest size.

The authors trained five model variants (2.3 B to 31 B parameters, plus one Mixture-of-Experts model) on large-scale multimodal data through January 2025. They introduced a “thinking mode” that produces an explicit reasoning trace before the final answer, replaced separate vision and audio encoders in the 12 B model with lightweight projection layers, and applied several memory-saving techniques for long contexts. All models were further optimized with quantization-aware training and speculative decoding heads. Evaluation combined standard academic benchmarks, vision and audio tasks, long-context tests, and blind human preference ratings on the Chatbot Arena platform.

Three findings stand out. First, Gemma 4 31 B is now the strongest dense open model on Arena and matches or exceeds several much larger Mixture-of-Experts systems. Second, the smaller variants (especially the 4.5 B model) equal or surpass the prior Gemma 3 27 B model on vision and audio benchmarks despite using far fewer parameters. Third, the efficiency measures reduce the global key-value cache by up to 37 percent and deliver 10–70 percent lower memory footprints after quantization, with only marginal quality loss.

These results matter for any organization that must balance capability against cost, latency, or data-sovereignty constraints. They show that frontier-level multimodal reasoning is now available under an Apache 2.0 license and can run efficiently on edge hardware, lowering barriers to experimentation and integration.

Leaders should decide where Gemma 4 fits in their model portfolio and pilot it on representative workloads, paying particular attention to the new thinking mode and the encoder-free 12 B variant. Where safety or compliance risk is material, organizations should apply additional output filters and conduct domain-specific red-teaming before broader rollout. Google DeepMind has already released safety evaluations and usage guidelines; these should be reviewed and, if needed, supplemented with internal monitoring.

The main limitations are the January 2025 data cutoff, which will gradually reduce relevance on very recent events, and the fact that some efficiency claims rest on specific hardware and quantization libraries. Overall, the evidence for performance and efficiency gains is consistent across automated benchmarks and human ratings, giving reasonable confidence for near-term deployment decisions.

1. Introduction

Section Summary: Gemma 4 is the newest addition to the Gemma family of open-weight language models, designed to handle text, images, and audio together while delivering strong performance on complex reasoning tasks. The lineup includes several model sizes, from small efficient versions to larger ones, along with technical improvements that make them faster, more memory-friendly, and better at extended conversations or step-by-step thinking. The models are released under an Apache 2.0 license after extensive testing against other leading systems.

The rapid evolution of large language models has driven the need for open-weight models with strong multimodal understanding, reasoning, and computational efficiency. Building upon the foundations of its predecessors ([1, 2, 3]), we introduce Gemma 4, the most capable and efficient generation in the Gemma model family to date. Gemma 4 offers natively multimodal architectures, capable of seamlessly processing text, images, and audio while achieving frontier-level performance on highly complex reasoning tasks. The Gemma 4 family is built to serve a variety of on-device hardware. The model suite includes both dense architectures (2.3B, 4.5B, 12B, and 31B parameters) and a Mixture-of-Experts ([4], MoE) variant with 3.8B activated and 26B total parameters. We introduce several architectural and methodological innovations:

• Thinking mode for advanced reasoning: We introduce a thinking mode ([5]) to Gemma 4 models. By outputting a reasoning trace before the response, models demonstrate improved capabilities in reasoning-heavy domains such as mathematics and coding.
• Long-context efficiency: Extended contexts lead to a memory explosion in the KV cache. We conserve a 5:1 ratio of local sliding window to global self-attention (4:1 for the 2.3B model) and use $p$-RoPE ([6]) as positional encoding. Combined with KV cache sharing ([7]) and the reuse of keys as values in global layers ([8]), these optimizations reduce the global KV cache footprint by up to 37.5%.
• Compute efficiency: We release an autoregressive multi-token prediction (MTP) drafter head ([9]) designed for speculative decoding ([10]) to improve the decoding speed of our models.
• Memory efficiency: We provide quantized versions of our models trained with quantization-aware training ([11], QAT) to reduce their parameter memory footprint and latency with minimal impact on quality.
• Encoder-free architecture: Gemma 4 models have frozen vision and audio encoders. We introduce a unified encoder-free architecture for the 12B model, which projects raw 40ms audio chunks and image patches into the LLM embedding space, alleviating the need for separate encoders and reducing memory fragmentation.

In this technical report, we outline the different model architectures across model sizes as well as the pre-training and post-training recipe of Gemma 4. Through comprehensive benchmarks and human evaluations such as Arena ([12]), we demonstrate that Gemma 4 operates at a level comparable to larger, frontier open-source models across text, image, and audio modalities. We release the Gemma 4 models under an Apache 2.0 license, empowering developers and researchers everywhere to build upon, customize, and extend these capabilities.

\begin{tabular}{@l r r r r r@}
  \toprule
  Model & \makecell{Audio\\ Encoder} & \makecell{Vision\\ Encoder} & \makecell{Embedder} & \makecell{Einsums} & \makecell{Drafter} \\
  \midrule
  \textbf{E2B} & 305M & 150M & \makecell{400M + 2, 340M} & 1, 870M & 76M \\
  \textbf{E4B} & 305M & 150M & \makecell{670M + 2, 820M} & 3, 940M & 77M \\
  \textbf{12B} & - & - & 1, 000M & 10, 890M & 400M \\
  \textbf{26B-A4B*} & - & 550M & 740M & 24, 500M / 2, 800M (active) & 430M \\
  \textbf{31B} & - & 550M & 1, 410M & 29, 290M & 500M \\
  \bottomrule
  \end{tabular}

2. Model Architecture

Section Summary: Gemma 4 models use a decoder-only Transformer backbone that includes both dense variants of varying sizes and one mixture-of-experts model, along with specialized normalization layers. Smaller versions incorporate separate vision and audio encoders, whereas the 12B model replaces these with lightweight projection layers that feed image patches and raw audio chunks directly into the core network. Efficiency improvements include alternating local and global attention patterns, reduced key-value caching, and tailored positional encodings to support longer contexts.

Gemma 4 models follow a decoder-only Transformer architecture ([13]). Our models have pre-norm and post-norm with RMSNorm ([14]), and QKNorm ([15]).

Dense and MoE: The Gemma 4 family of models comprises dense architectures, with effective 2.3B (E2B), effective 4.5B (E4B), 12B and 31B parameters, as well as an MoE model with 3.8B activated parameters for 26B total parameters (26B-A4B). E2B and E4B use per-layer embeddings as in Gemma 3n ([16]), making them 2.3B and 4.5B effective out of 5B and 8B total parameters respectively.

\begin{tabular}{@l c c c c c@}
  \toprule
   {} & & & \multicolumn{3}{c}{Shards} \\
  \cmidrule{4-6}
  Model & TPU & #Chips & Data & Seq & Replica \\
  \midrule
  \textbf{E2B} & v6e & 4, 096 & 16 & 8 & 32 \\
  \textbf{E4B} & v6e & 6, 144 & 16 & 16 & 24 \\
  \textbf{12B} & v5p & 12, 288 & 16 & 16 & 48 \\
  \textbf{26B-A4B*} & v6e & 6, 144 & 16 & 16 & 24 \\
  \textbf{31B} & v6e & 10, 240 & 16 & 16 & 40 \\
  \bottomrule
  \end{tabular}

Long-context efficiency: Our local to global attention ratio patterns follow [3], that is, 4-to-1 local attention blocks for E2B and 5-to-1 for the rest. We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B), i.e., $\text{values} = \text{keys}$. We encode position with $p$-RoPE with $p=0.25$ on global attention layers and with RoPE on local attention layers, effectively reducing the global KV cache by 37.5%. The RoPE frequencies are set to 1M and 10k on global and local attention layers, respectively. Finally, we share the KV cache with ratios of 20/35 and 18/42 for the E2B and E4B model.

2.1 Vision modality

E2B and E4B Gemma models come with a 150M vision encoder, while larger models use a 550M encoder (except for the unified 12B). Both are Vision Transformers ([17], ViT) with a patch size of 16, whose architectural differences are detailed in Table 10 in Appendix. Our vision encoders support variable aspect ratios (see Figure 2 and Algorithm 1) and incorporate both axial 2D-RoPE ([18]) with non-causal attention and 2D absolute positional embeddings. We restrict the maximum number of tokens, $N_\text{max}$ to the values $70, 140, 280, 560$ and $1120$ (see Algorithm 1 for implementation details).

2.2 Audio modality

E2B and E4B Gemma models use a 305M audio encoder that processes audio in 40ms chunks with Mel filterbank inputs. The encoder architecture is based on the Universal Speech Model ([19], USM), consisting of two downsampling convolution layers followed by twelve Conformer layers ([20]). While the architecture remains similar to that of Gemma 3n, we reduce the number of parameters by 55% (from 680M to 305M). We do not use vector quantization; the LLM ingests the continuous representations produced by the audio encoder. As with the vision encoder, we keep weights frozen during pre-training.

2.3 Encoder-free architecture

Gemma 4 12B is trained from scratch based on a new, unified, and encoder-free model paradigm, replacing the separate vision and audio encoders with lightweight projection modules. For the vision modality, Gemma 4 12B takes in 48 $\times$ 48 $\times$ 3 RGB patches, but replaces the 550M vision encoder by a single large matmul (35M parameters). Spatial awareness is maintained by adding 2D coordinate-based positional embeddings directly to the patch representations before a final LayerNorm layer ([21]).

For audio, the 305M USM-based conformer encoder is entirely discarded. Raw audio is segmented into 40ms chunks at 16kHz, resulting in 640-dimensional vectors per chunk. These are projected directly into the LLM embedding space. Since audio is a temporal sequence, it does not require additional positional encoding.

\begin{tabular}{@l c c c@}
  \toprule
  Model & bf16 & Quantized & KV Cache \\
  \midrule
  \textbf{E2B} & 4.6 & ${0.8}^{\dagger}$ & +0.05 \\
  \textbf{E4B} & 9.0 & ${2.3}^{\dagger}$ & +0.14 \\
  \textbf{12B} & 24.0 & ${7.65}^{\ddagger}$ & +0.28 \\
  \textbf{26B-A4B*} & 52.0 / 7.6 & ${16.2 / 2.8}^{\ddagger}$ & +0.28 \\
  \textbf{31B} & 64.0 & ${19.2}^{\ddagger}$ & +1.10 \\
  \bottomrule
  \end{tabular}

2.4 Pre-training

We follow a similar pre-training as Gemma 3.

Training data. Our pre-training dataset is a large-scale, diverse collection of data from a wide range of domains and modalities, including web documents, code, images, and audio (for E2B, E4B and 12B), with a cutoff date of January 2025.

Tokenizer. We use the same tokenizer as [22] that is, a SentencePiece tokenizer ([23]) with split digits, preserved whitespace, and byte-level encodings. The vocabulary has 262k entries.

Filtering. We filter data to decontaminate benchmarks, and to reduce the risk of unwanted or unsafe utterances and the risk of recitation.

: Table 4: Leading open-weight models on Arena Text ([12]) (as of June 19, 2026). Models are evaluated through blind side-by-side evaluations by human raters, and attributed scores based on the Elo rating system. The top closed model (gray) is included for scale. Gemma models rival much larger models, and Gemma 4 31B is the leading dense open model on the leaderboard.

Rank	Model	Elo	95% CI	Open	Type	#params/#activated
1	Claude Fable 5	1508	$\pm$ 9	no	-	- / -
...
15	GLM 5.1	1475	$\pm$ 6	yes	MoE	744B / 40B
25	GLM 5.2 (Max)	1471	$\pm$ 10	yes	MoE	744B / 40B
29	MiMo V2.5 Pro	1466	$\pm$ 5	yes	MoE	1T / 42B
34	Kimi K2.6	1460	$\pm$ 5	yes	MoE	1T / 32B
36	DeepSeek V4 Pro Thinking	1458	$\pm$ 5	yes	MoE	1.6T / 49B
37	GLM 5	1457	$\pm$ 5	yes	MoE	744B / 40B
38	DeepSeek V4 Pro	1456	$\pm$ 5	yes	MoE	1.6T / 49B
43	Gemma 4 31B	1451	$\pm$ 8	yes	Dense	31B
44	Kimi K2.5 Thinking	1450	$\pm$ 4	yes	MoE	1T / 32B
57	Qwen 3.5 397B-A17B	1444	$\pm$ 4	yes	MoE	397B / 17B
61	Gemma 4 26B-A4B	1438	$\pm$ 8	yes	MoE	26B / 4B
63	DeepSeek V4 Flash Thinking	1436	$\pm$ 5	yes	MoE	284B / 13B
...
157	Gemma 3 27B	1366	$\pm$ 4	yes	Dense	27B

2.5 Quantization-Aware Training

We provide quantized models and encoders in different formats along with the raw checkpoints. Based on the most popular open source quantization inference engines (e.g. llama.cpp) as well as efficient hardware support, we focus on two sets of weight representations:

• mobile quantization: per-channel low bitwidth weight (mix of int2 and int4) and activation quantization (int8).
• Q4_0 quantization: blockwise quantization, often referred to as Q4_0.

In Table 3, we report the memory filled by raw and quantized models with and without a KV cache for a sequence of 32k tokens. Furthermore, to enable stable inference in fp16, we introduce a scalar scale at each block in order to bound the activation ranges to fit fp16.

::: {caption="Table 5: Performance comparison of Gemma 3 27B and Gemma 4 models on diverse benchmarks. All models are in thinking mode unless explicitly stated."}

:::

We also apply QAT to the image and audio encoders. On the 150M image encoder, quantizing activations and weights to 8-bit precision (W8A8) yields a 2 $\times$ reduction in total forward-pass memory footprint (from 400 MB to 200 MB, including on-device compilation overhead) and a 44% reduction in on-device latency relative to Gemma 3n on newer hardware. On the audio encoder, we further reduce activation precision to 8 bits and weight precision to ${2, 4, 8}$ bits, varying by layer cluster. Overall, we achieve a 78% reduction in on-disk footprint, from 390 MB in Gemma 3n to 87 MB in this version.

2.6 Multi-Token Prediction Drafter

We train a small autoregressive MTP drafter head with our models, used for speculative decoding. In our MTP procedure, the model's last layer activations from the previous step and token embeddings are fed into the MTP head. The MTP head generates future tokens sequentially using a separate embedder and a 4-layer Transformer block that cross-attends to the KVs of the main model (Figure 1), thus eliminating the need for MTP prefill and supporting any draft length. The Transformer block has model dimension 256 for E2B and E4B, 1024 for 26B-A4B and 31B, three local, and one global attention layers.

Efficient MTP Decoding.

For the E2B and E4B drafters, we reduce the decoding overhead by replacing the projection operation to the entire vocabulary by a top-k operation on clusters of tokens. As a result, final matrix multiplication is reduced from $d\times 262, 000$ to $d\times 4096$ while preserving a similar acceptance rate.

2.7 Compute Infrastructure

We train our models with TPUv5p and TPUv6e as outlined in Table 2. Each model configuration is optimized to minimize training step time. For our larger models, we leverage Slice-Granularity Elasticity ([22]), which allows continuous training with fewer “slices” of TPU chips when there is a localized failure. This reconfiguration reduces the delay caused by interruptions from many minutes to a few seconds.

The optimizer state is sharded using an implementation of ZeRO-3 ([24]). For multi-pod training, we perform a data replica reduction over the data center network, using the Pathways approach of [25]. We use the single controller programming paradigm of JAX ([26]) and Pathways, along with the GSPMD partitioner ([27]) and the MegaScale XLA compiler ([28]).

\begin{tabular}{@l ccccc c c @}
\toprule
  {} & \multicolumn{5}{c}{Gemma 4} && Gemma 3 \\
\cmidrule{2-6}\cmidrule{8-8}
 {} & 31B & \makecell{26B-A4B} & 12B & E4B & E2B && 27B \\
\midrule
MMMU Pro & 76.9 & 73.8 & 69.1 & 52.6 & 44.2 && 49.7 \\
MATH-Vision & 85.6 & 82.4 & 79.7 & 59.5 & 52.4 && 46.0 \\
MedXPertQA MM & 61.3 & 58.1 & 48.7 & 28.7 & 23.5 && - \\
InfographicVQA & 92.0 & 89.3 & 88.4 & 70.0 & 63.9 && 70.6 \\
OmniDocBench 1.5 $\downarrow$ & 0.131 & 0.149 & 0.164 & 0.181 & 0.290 && 0.365 \\
\bottomrule
\end{tabular}

3. Instruction Tuning

Section Summary: Instruction tuning converts pre-trained models into more capable versions by applying targeted post-training, with the key addition of a thinking mode that lets the model generate an internal reasoning trace before responding. The team carefully filters training data to remove personal details, toxic content, duplicates, and errors while adding examples that promote hedging, source attribution, and appropriate refusals, which improves factual accuracy. They also standardize formatting differences, such as using distinct end-of-sequence tokens, so the models can properly handle instruction following, thinking activation, and tool use.

Pre-trained models are turned into instruction-tuned models with a similar post-training approach as in Gemma 3. A significant difference is the addition of a thinking mode, where the model can output a reasoning trace before answering.

Data filtering. We carefully optimize the data used in post-training to maximize model performance. We filter examples that show certain personal information, unsafe or toxic model outputs, mistaken self-identification data, and duplicated examples. Including subsets of data that encourage better in-context attribution, hedging, and refusals to minimize hallucinations also improves performance on factuality metrics, without degrading model performance on other metrics.

PT versus IT formatting. All models share the same tokenizer, with some control tokens dedicated to IT formatting. A key difference is that PT models output an <eos> token at the end of generation, while IT models output <turn|> at the end of the generation. An example is given for IT in Table 11. Fine-tuning either model type thus requires adding their respective end tokens. We detail how to activate thinking and how models handle function calling in Table 11.

4. Evaluation of final models

Section Summary: The section evaluates the final Gemma 4 models through both human judgments on an online arena platform and a range of automated benchmarks covering general knowledge, vision, audio transcription and translation, and long-context tasks. Human ratings place the 31B model at the top among open dense models while the smaller mixture-of-experts variant matches the performance of much larger systems. Static benchmarks show consistent gains over the prior Gemma 3 27B release, with the new models achieving equal or better results despite having far fewer parameters.

In this section, we evaluate the IT models over a series of automated benchmarks and human evaluations across a variety of domains, as well as static benchmarks such as MMLU Pro.

4.1 Human evaluation

We report the performance of our 31B and 26B-A4B models on Arena ([12]) in blind side-by-side evaluations by human raters against other state-of-the-art models. We report Elo scores in Table 4. Gemma 4 31B is the top open model in the dense category, and both Gemma 4 31B and 26B-A4B show performance equal to much larger open models.

::: {caption="Table 7: Audio performance for Gemma 4 and Gemma 3n models. Top: CoVoST (S2TT prompt: transcribe then translate). Bottom: FLEURS ASR (transcription). Compared to Gemma 3n of corresponding sizes, Gemma 4 achieves a 12% (E2B) / 10% (E4B) relative improvement on translation and a 17% (E2B) / 12% (E4B) relative improvement on transcription, despite a 78% reduction in on-disk audio encoder footprint (from 390 MB to 87 MB after quantization)."}

:::

4.2 Static benchmarks

In Table 5, we show the performance of our final models across a variety of benchmarks compared to Gemma 3 27B. Gemma 4 31B is closest in size and significantly better across the board, while E2B roughly matches Gemma 3 27B performance with 10x less parameters. Table 6 shows the performance of Gemma 4 models on vision benchmarks, with E4B equaling or outperforming Gemma 3 27B on all evals. Table 7 and Table 8 display the multilingual audio transcription and translation performance of E2B & E4B and of 12B respectively. Table 9 shows a leap on long-context capabilities between Gemma 3 27B and Gemma 4 models, with E4B outperforming Gemma 3 27B.

5. Responsibility, Safety, Security

Section Summary: Gemma 4 models are built with a strong focus on safety and responsibility, undergoing the same detailed evaluations as Gemini models to reduce risks of harmful outputs like hate speech, explicit content, or dangerous instructions. The training process includes data filtering to remove sensitive information, alignment with Google's safety policies, and extensive testing that shows clear improvements over earlier versions. Developers are provided with guidelines and tools to address issues such as bias, privacy, and misuse, supporting the safe release of open models only when benefits are judged to outweigh foreseeable risks.

As open models become central to enterprise infrastructure, provenance and security are paramount. Gemma 4 undergoes the same rigorous safety evaluations as Gemini models. Responsibility, safety, and security are of utmost importance in the development workflow, ensuring that these language models are designed from the ground up for responsible AI development.

5.1 Governance & Assessment

Our approach to assessing the benefits and risks of Gemma 4 reflects the foundation established in prior models, updated to account for its expanded multimodal capabilities. We maintain the belief that openness in AI can spread the benefits of these technologies across society, but this must be continuously evaluated against the risk of malicious uses that can cause individual and institutional harm ([29]).

Gemma 4 models were developed in partnership with internal safety and responsible AI teams. Releasing these models required careful scrutiny of the evolving risks associated with LLMs and an understanding of how models are deployed in the wild. While an open model shares innovation across the AI ecosystem, we remain committed to providing educational resources to users and monitoring downstream model usage.

::: {caption="Table 8: Audio performance of Gemma 4 12B model on supported languages, demonstrating that competitive audio-text performance can be achieved without a dedicated audio encoder."}

:::

::: {caption="Table 9: Long context performance of Gemma 3 and Gemma 4 models (without thinking)."}

:::

5.2 Safety Policies and Train-Time Mitigations

A key pillar of Gemma's safety approach is aligning our fine-tuned models with Google's AI principles and safety policies. These policies aim to prevent our generative models from producing harmful content, specifically:

• Content related to child sexual abuse material (CSAM) and exploitation;
• Dangerous content, e.g., promoting suicide, or instructing in activities that could cause real-world harm;
• Sexually explicit content;
• Hate speech, e.g., dehumanizing members of protected groups;
• Harassment, e.g., encouraging violence against people.

To mitigate these risks, Gemma 4 models underwent careful input data pre-processing and scrutiny. The training data was specifically filtered for the removal of certain personal information and other sensitive data to guard against privacy violations. Post-training evaluations and train-time mitigations were also implemented to align the model with our safety policies.

5.3 Safety Evaluations

We conduct rigorous automated and human evaluations to understand the potential harms our models might cause. For all areas of safety testing, we saw major improvements in every category of content safety relative to previous Gemma models. Overall, Gemma 4 models significantly outperform Gemma 3 and 3n models in improving safety, while keeping unjustified refusals low.

Importantly, all testing was conducted without safety filters to accurately evaluate the model's inherent capabilities and behaviors. For both text-to-text and image-to-text modalities, and across all model sizes, the models produced minimal policy violations. We balance development speed with targeted safety testing, upholding the commitments laid out in our Frontier Safety Framework ([30]).

5.4 Ethical Considerations and Risk Mitigation

The development of LLMs introduces specific ethical considerations. In making Gemma 4, we focused heavily on:

• Bias and Fairness: LLMs trained on large-scale text and image data can reflect embedded socio-cultural biases. We encourage developers to perform continuous monitoring (using evaluation metrics and human review) and explore de-biasing techniques during model fine-tuning.
• Misinformation and Misuse: LLMs can be misused to generate false or misleading text. We provide technical limitations, developer education, and guidelines for responsible use within the Responsible Generative AI Toolkit to mitigate malicious applications.
• Privacy Considerations: While our training datasets were filtered to remove certain personal information and other sensitive data, developers are strongly encouraged to adhere to local privacy regulations and implement privacy-preserving techniques in their applications.

5.5 Our Approach to Responsible Open Models

Designing safe, secure, and responsible applications requires a system-level approach that mitigates risks associated with specific use cases and environments. We provide guidelines, mechanisms, and safeguards for content safety, and encourage developers to implement appropriate configurations based on their product policies. We will continue to adopt safety mitigations proportionate to potential risks, sharing these models with the community only when confident that the benefits significantly outweigh foreseeable risks.

6. Discussion and Conclusion

Section Summary: The report introduces Gemma 4, a new family of open AI models that handle text, images, and audio together in a single efficient system designed to run across a range of hardware. These models include a “thinking mode” that generates step-by-step reasoning before answering, along with technical improvements that reduce memory use and speed up processing for long or complex inputs. Overall, the models show clear performance gains over earlier versions and reach levels comparable to much larger systems, supporting both practical edge applications and open research.

In this technical report, we presented Gemma 4, an open-weight model family featuring multimodal dense and MoE architectures designed for varied hardware environments. Gemma 4 models come with a thinking mode in which they generate reasoning traces prior to responding, improving overall performance. We introduced a unified, encoder-free architecture that processes raw audio and image patches. We also alleviated long-context memory limitations via better local-to-global attention ratios, positional encoding, and KV cache sharing. We increased the overall compute efficiency via QAT and memory efficiency via MTP drafters. Gemma 4 models demonstrate a leap in performance compared to Gemma 3 across benchmarks, and human evaluations demonstrate that Gemma 4 performs comparably to significantly larger open models, providing a scalable foundation for edge deployment and reasoning while supporting open research.

Core contributors

Sherif El Abd Vaibhav Aggarwal Robin Algayres Alek Andreev Olivier Bachem Ian Ballantyne Cormac Brick Victor Cărbune Michelle Casbon Mayank Chaturvedi Victor Cotruta Alice Coucke Phil Culliton Robert Dadashi Lucas Dixon Mohamed Elhawaty Utku Evci Clément Farabet Johan Ferret Filippo Galgani Sertan Girgin Jean-Bastien Grill Maarten Grootendorst Jiaxian Guo Cassidy Hardin Yanzhang He Steven M. Hernandez Omri Homburger Léonard Hussenot Juyeong Ji Armand Joulin Aishwarya Kamath Parnian Kassraie Olivier Lacombe Preethi Lahoti Gaël Liu Gus Martins Luciano Martins Tatiana Matejovicova Ramona Merhej Nikola Momchev Sneha Mondal Ryan Mullins Sindhu Raghuram Panyam Shreya Pathak Sarah Perrin

André Susano Pinto Etienne Pot Angéline Pouget Alexandre Ramé Sabela Ramos Douglas Reid David Rim Morgane Rivière Karsten Roth Louis Rouillard

Omar Sanseviero

Pier Giuseppe Sessa Shane Settle Danila Sinopalnikov Sara Smoot Piotr Stanczyk Andreas Steiner Lawrence Stewart Ilya Tolstikhin Michael Tschannen Anton Tsitsulin Nino Vieillard Renjie Wu Pingmei Xu Haichuan Yang Edouard Yvinec Li Zhang Joe Zou

Contributors

Nicolas Aagnes Abdelrahman Abdelhamed Shivani Agrawal Shubham Agrawal Ibrahim Alabdulmohsin Jean Baptiste Alayrac Uri Alon Chandramouli Amarnath Ankesh Anand Chrysovalantis Anastasiou Setareh Ariafar François-Xavier Aubet Kyriakos Axiotis Federico Barbero Joelle Barral Alexei Bendebury Urs Bergmann Stanley Bileschi Kat Black Mathieu Blondel Sebastian Borgeaud Arthur Bražinskas Ryan Burnell Robert Busa-Fekete Mu Cai Glenn Cameron Charlotte Caucheteux Garima Chadha Jetha Chan Aditya Chawla Blake Jianhang Chen Jesse Chen Lin Chen Xu Chen Derek Cheng Tzu-hsiang Chien Nikolai Chinaev Yi Chou Zhaohui Chu Benjamin Coleman Pooja Consul Sam Conway-Rahman Scott Crowell Dylan Cutler Vivek Dani Samira Daruki Anil Das Daniel Deutsch Nishanth Dikkala Li Ding Qiuhan Ding Shenil Dodhia Konstantin Donhauser Tulsee Doshi Anca Dragan Alex Druinsky Sahil Dua Zoltan Egyed Danielle Eisenbud Daniel Eppens Cindy Fan Bahare Fatemi Yassir Fathullah Vlad Feinberg Milen Ferev Takumi Fujimoto Isaac Galatzer-Levy João Gante Simon Geisler Soham Ghosal Antonious M. Girgis Alec Go Alhaad Gokhale Alex Grills Yiming Gu Pramod Gupta Guru Guruganesh Raia Hadsell Hamza Harkous Jitendra Harlalka Demis Hassabis Anja Hauth Joe Heyward Arian Hosseini Chih-Yang Hsia I-Hung Hsu Xiaopeng Huang Yangsibo Huang Kevin Hui Adrian Hutter Te I Fotis Iliopoulos Advait Jain Ganesh Jawahar Ziwei Ji Qilin Jin Melvin Johnson Kandarp Joshi Arun Kandoor Wang-Cheng Kang Koray Kavukcuoglu Mehran Kazemi Kathleen Kenealy Amr Khalifa Phoebe Kirk Suraj Kothawade Vitaly Kovalev Neel Kovelamudi Adam Kraft Ravin Kumar Harish Kuppam Justin Lannin Chen-Yu Lee Seungji Lee Dmitry Lepikhin Dongdong Li Qiujia Li Valentin Liévin Ethan Lin Ziqian Lin Casper Liu Tianlin Liu Tianqi Liu Xin Liu Mayank Lunayach Min Ma Gagan Madan Andrii Maksai Eric Malmi Michal Matuszak Daniel McDuff Gaurav Menghani Daniil Mirylenka Karolis Misiunas Vedant Misra Andreea Mitran Kareem Mohamed Maksim Mukha Eric Noland James O'Donnell Kate Olszewska Bernett Orlando Wanqiong Pan Rina Panigrahy Unnati Parekh Chunjong Park Eric Paskie Liqian Peng Bryce Petrini Slav Petrov Jonas Pfeiffer Bilal Piot Martyna Plomecka Siim Poder Octavio Ponce Arijit Pramanik David Racz Anish Rajan Michelle Ramanovich Anand Rao Marvin Ritter Vitor Rodrigues Evan Rosen Mikołaj Rybiński Noveen Sachdeva Michaël E. Sander Rohit Sathyanarayana Sagar Savla Samuel Schmidgall Tal Schuster Benoit Seguin Andrew Sellergren Aliaksei Severyn Izhak Shafran Dhruv Shah Yuan Shangguan Ashish Shenoy Pradeep Shenoy Rakesh Shivanna Pauline Sho Lucas Spangher Wojciech Stokowiec Tim Strother Yao Su Yinghao Sun Mukund Sundararajan Andrea Tacchetti Mor Hazan Taege Pouya Tafti Chetan Tekur Rahul Thapa Madeleine Traverse Lenart Treven Tao Tu Chien Te Tung Petar Veličković Malini Pooni Venkat Sagar Gubbi Venkatesh Vidya Venkiteswaran Francesco Visin Alex Vitvitskyi Kiran Vodrahalli Weiyi Wang Xin Wang Tris Warkentin Jan Wassenberg John Wieting Lechao Xiao Hao Xu Yuhui Xu Fuzhao Xue

Arun Yadav Jun Yan Antoine Yang Lin Yang Ming-Hsuan Yang Ziyu Ying Jae Hyeon Yoo Sajjad Zafar Fred Zhang Jiageng Zhang Jianyi Zhang Xiaofan Zhang Chao Zhao David Zhou Chen Zou

Appendix

Section Summary: The appendix provides supporting technical details on the model's design and evaluation, including an example of the structured conversation format used for interactive tasks with thinking steps and tool calls. It describes the architecture of the vision encoder, the specific method for resizing images while preserving aspect ratios before processing, and related algorithms and figures that explain how visual inputs are converted into tokens for the language model. Performance results on various vision benchmarks at lower image resolutions are also included.

Conversation format. We give an example of a conversation including thinking, function definition and function calling in Table 11.

Vision. We detail the vision encoder architecture in Table 10. We then illustrate how images are resized before being fed to the vision encoder in Figure 2, and detail the resizing algorithm in Algorithm 1. We display the vision benchmark scores of Gemma 4 models at low resolution ($N_{max} = 280$) in Table 12.

\begin{tabular}{@ c c c c c @}
\toprule
\textbf{Total Params} & $d_{model}$ & $d_{MLP}$ & $N_{heads}$ & $N_{layers}$ \\
\midrule
\textbf{550M} & 1152 & 4304 & 16 & 27 \\
\textbf{150M} & 768 & 3072 & 12 & 16 \\
\bottomrule
\end{tabular}

**Require:** Image $\mathbf{I} \in \mathbb{R}^{H \times W \times C}$, patch size p, max tokens N(max), pooling kernel size k
m ← k · p // Pooled patch size
T ← N(max) · m²
f ← √(T / (H · W)) // Ideal scaling factor
H(ideal) ← f · H
W(ideal) ← f · W
H(target) ← ⌊ H(ideal) / m ⌋ · m // Round down
W(target) ← ⌊ W(ideal) / m ⌋ · m
$\mathbf{I}_{\text{resized}} \gets \text{BicubicResize}(\mathbf{I}, H_{\text{target}}, W_{\text{target}})$
**return** $\mathbf{I}_{\text{resized}}$

::: {caption="Table 11: Formatting for Gemma IT models. Explicitly add the [BOS] token after tokenization, or use the add_bos=True option in the tokenizer. Do not tokenize the text '[BOS]'. Add <|think|> in a leading system turn to activate the thinking mode. Check the official documentation for the function declaration and function calling syntax, as well as more advanced examples."}

:::

::: {caption="Table 12: Gemma 4 models performance on vision benchmarks at resolution $N_m$ax = 280 (thinking)."}

:::

References

Section Summary: The references section compiles a bibliography of around thirty sources, primarily research papers, technical reports, and online resources. These cover the development of large language models such as the Gemma and Gemini families, along with foundational and applied work on transformer techniques, model training infrastructure, faster inference methods, and issues of AI safety and ethics. The citations range from early neural network studies to recent advances in efficient architectures and evaluation platforms.

[1] Gemma Team. Gemma: Open models based on gemini research and technology, 2024a.

[2] Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024b.

[3] Gemma Team. Gemma 3: Technical report. arXiv preprint arXiv:2503.19786, 2025a.

[4] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3:79–87, 1991.

[5] OpenAI. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.

[6] F. Barbero, A. Vitvitskyi, C. Perivolaropoulos, R. Pascanu, and P. Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025.

[7] N. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019.

[8] A. Kayyam, A. M. Gopal, and M. A. Lewis. Do transformers need three projections? systematic study of qkv variants. arXiv preprint arXiv:2606.04032, 2026.

[9] Y. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024.

[10] Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.

[11] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In CVPR, 2018.

[12] W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024.

[13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. 2017.

[14] B. Zhang and R. Sennrich. Root mean square layer normalization. 2019.

[15] A. Henry, P. R. Dachapally, S. S. Pawar, and Y. Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4246–4253, 2020.

[16] Gemma Team. Gemma 3n. https://deepmind.google/models/gemma/gemma-3n/, 2025b.

[17] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.

[18] B. Heo, S. Park, D. Han, and S. Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pages 289–305. Springer, 2024.

[19] Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang, et al. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037, 2023.

[20] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020.

[21] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.

[22] Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.

[23] T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. 2018.

[24] J. Ren, S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He. Zero-offload: Democratizing billion-scale model training. In USENIX, 2021.

[25] P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim, R. Pang, S. Roy, B. Saeta, P. Schuh, R. Sepassi, L. E. Shafey, C. A. Thekkath, and Y. Wu. Pathways: Asynchronous distributed dataflow for ml, 2022.

[26] A. Roberts, H. W. Chung, G. Mishra, A. Levskaya, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, et al. Scaling up models and data with t5x and seqio. JMLR, 2023.

[27] Y. Xu, H. Lee, D. Chen, B. A. Hechtman, Y. Huang, R. Joshi, M. Krikun, D. Lepikhin, A. Ly, M. Maggioni, R. Pang, N. Shazeer, S. Wang, T. Wang, Y. Wu, and Z. Chen. GSPMD: general and scalable parallelization for ML computation graphs. 2021.

[28] XLA. Xla: Optimizing compiler for tensorflow, 2019.

[29] L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, L. Rimell, L. A. Hendricks, W. Isaac, S. Legassick, G. Irving, and I. Gabriel. Ethical and social risks of harm from language models, 2021.

[30] Google DeepMind. Introducing the frontier safety framework. https://deepmind.google/blog/introducing-the-frontier-safety-framework/, 2024.