Google DeepMind Releases Gemma 4: Encoder-Free Multimodal Models from 2.3B to 30.7B Parameters
Google DeepMind released Gemma 4, a family of open-weight multimodal models ranging from 2.3B to 30.7B parameters. The flagship 12B Unified model eliminates separate encoders, processing text, images, audio, and video directly through a single decoder-only transformer with up to 256K token context window.
Gemma 4 12B Unified — Quick Specs
Google DeepMind Releases Gemma 4: Encoder-Free Multimodal Models from 2.3B to 30.7B Parameters
Google DeepMind released Gemma 4, a family of five Apache 2.0-licensed multimodal models ranging from 2.3B to 30.7B parameters. The flagship 12B Unified model introduces an encoder-free architecture that processes text, images, audio, and video directly through a single decoder-only transformer.
Model Lineup and Architecture
Gemma 4 consists of five models:
- E2B: 2.3B effective parameters (5.1B with embeddings), 128K context
- E4B: 4.5B effective parameters (8B with embeddings), 128K context
- 12B Unified: 11.95B parameters, 256K context
- 26B A4B (MoE): 25.2B total parameters with 3.8B active, 256K context
- 31B Dense: 30.7B parameters, 256K context
All models support text and image input. E2B, E4B, and 12B Unified include native audio and video capabilities.
The 12B Unified model eliminates the dedicated vision and audio encoders used in other Gemma 4 models. Instead, it projects raw image patches and audio waveforms directly into the LLM's embedding space through lightweight linear layers, reducing multimodal latency and enabling end-to-end fine-tuning.
Technical Specifications
All models use a hybrid attention mechanism that alternates between local sliding window attention (512-1024 tokens) and full global attention. Global layers feature unified Keys and Values with Proportional RoPE (p-RoPE) to optimize memory for long contexts.
The MoE model (26B A4B) activates only 3.8B of its 25.2B parameters during inference, using 8 active experts from a pool of 128 total experts plus 1 shared expert.
Benchmark Performance
According to Google DeepMind, Gemma 4 31B achieved:
- MMLU Pro: 85.2%
- AIME 2026 (no tools): 89.2%
- LiveCodeBench v6: 80.0%
- Codeforces ELO: 2150
- GPQA Diamond: 84.3%
- Vision MMMU Pro: 76.9%
The 12B Unified model scored 77.2% on MMLU Pro, 77.5% on AIME 2026, and 72.0% on LiveCodeBench v6.
Capabilities and Release Details
All models include configurable reasoning modes, native function calling support, variable aspect ratio image processing, and multilingual support across 140+ languages. The E2B, E4B, and 12B models handle automatic speech recognition and speech-to-translated-text translation.
Models support up to 256K token context windows (12B, 26B A4B, 31B) with interleaved multimodal input. The smaller E2B and E4B models use Per-Layer Embeddings (PLE) to maximize parameter efficiency for on-device deployment.
All models are available on Hugging Face under Apache 2.0 license in both pre-trained and instruction-tuned variants.
What This Means
Gemma 4's encoder-free architecture in the 12B model represents a significant shift in multimodal model design, potentially reducing deployment complexity and latency compared to traditional encoder-decoder approaches. The family's range from mobile-optimized 2.3B models to the 30.7B dense variant provides options across the performance-efficiency spectrum, though real-world performance on production workloads remains to be validated independently. The MoE architecture's ability to deliver near-31B performance while activating only 3.8B parameters could make it attractive for inference-constrained deployments.
Related Articles
Mistral AI Releases Shieldstral-1.0-3B, a 3B-Parameter Policy-Adaptive Safety Classifier
Mistral AI has released Shieldstral-1.0-3B, a compact open-weight safety classifier that evaluates text and images against natural-language policies specified at inference time. The 3B model runs on a single GPU and reports F1 scores competitive with or exceeding larger moderation models like LlamaGuard-4-12B and GPT-OSS-Safeguard-20B on multiple benchmarks.
Mistral Releases Shieldstral, a 3B Open-Weights Safety Classifier That Matches Models 7x Its Size
Mistral has released Shieldstral, a 3B open-weights safety classifier that reframes content moderation as a policy-adaptive question-answering task. The model claims to match or outperform guard models up to 7x its size on text safety and multimodal benchmarks, and runs on a single 16GB GPU.
LG AI Research Releases K-EXAONE 2.0, a 750B-Parameter Open-Weight MoE Model with 262K Context
LG AI Research has released K-EXAONE 2.0, a 750-billion-parameter mixture-of-experts language model with 37B active parameters, a 262,144-token context window, and support for 10 languages. The model is open-weighted under Apache 2.0 and claims competitive results against Qwen3.5, GLM-5.1, and DeepSeek-V4 Pro on reasoning, coding, and long-context benchmarks.
Alibaba Unveils Qwen3.8-Max, a 2.4T-Parameter Open-Weight Model for Coding and Agentic Work
Alibaba announced Qwen3.8-Max, a 2.4T-parameter flagship model targeting coding and long-horizon agentic work, with open weights promised for next week alongside Qwen3.8-27B. The model posted strong third-party benchmark results, ranking #4 in Frontend Code Arena and matching Claude Opus 4.7 on the Vals Index at roughly 2.3x lower cost.
Comments
Loading...