Google DeepMind releases Gemma 4 12B Unified: encoder-free multimodal model with 256K context window
Google DeepMind has released Gemma 4 12B Unified, an encoder-free multimodal model that processes text, images, and audio through a single decoder-only transformer. The model features 11.95 billion parameters, a 256K token context window, and achieves 77.2% on MMLU Pro and 72.0% on LiveCodeBench v6.
Gemma 4 12B Unified — Quick Specs
Google DeepMind releases Gemma 4 12B Unified: encoder-free multimodal model with 256K context window
Google DeepMind has released Gemma 4 12B Unified, an 11.95 billion parameter multimodal model that eliminates separate encoders by processing text, images, and audio directly through a single decoder-only transformer. The model is part of the larger Gemma 4 family, which includes five models ranging from 2.3B to 30.7B effective parameters.
Technical specifications
Gemma 4 12B Unified features 48 layers with a 256K token context window and a 262K token vocabulary. The model uses a hybrid attention mechanism that interleaves local sliding window attention (1024 tokens) with full global attention, ensuring the final layer always has global context. Unlike other Gemma 4 models, the 12B Unified version projects raw image patches and audio waveforms directly into the language model's embedding space through lightweight linear layers, removing the need for dedicated vision or audio encoders.
The model is released under Apache 2.0 license with both pre-trained and instruction-tuned variants available.
Benchmark performance
According to Google DeepMind, Gemma 4 12B Unified achieves:
- 77.2% on MMLU Pro
- 77.5% on AIME 2026 (no tools)
- 72.0% on LiveCodeBench v6
- 1659 Codeforces ELO rating
- 78.8% on GPQA Diamond
- 69.1% on Vision MMMU Pro
- 79.7% on MATH-Vision
- 38.5 on CoVoST audio translation (excluding Chinese)
The model supports configurable "thinking modes" for step-by-step reasoning and includes native function-calling capabilities for agentic workflows.
Gemma 4 family architecture
The full Gemma 4 family includes:
- E2B: 2.3B effective parameters (5.1B with embeddings), 128K context, text/image/audio
- E4B: 4.5B effective parameters (8B with embeddings), 128K context, text/image/audio
- 12B Unified: 11.95B parameters, 256K context, encoder-free text/image/audio
- 26B A4B: 25.2B total parameters with 3.8B active (MoE), 256K context, text/image
- 31B: 30.7B parameters, 256K context, text/image
The "E" models use Per-Layer Embeddings (PLE) to maximize parameter efficiency for on-device deployment. The effective parameter count excludes large embedding lookup tables that don't contribute to compute during inference. The 26B A4B model uses mixture-of-experts architecture with 8 active experts out of 128 total, allowing it to run nearly as fast as a 4B model while maintaining larger model capacity.
Multimodal capabilities
Gemma 4 12B Unified processes images at variable aspect ratios and resolutions, handles video through frame sequences, and supports interleaved multimodal inputs mixing text and images in any order. Audio capabilities include automatic speech recognition and speech-to-translated-text translation across multiple languages.
The model includes native support for system prompts and maintains multilingual support across 140+ languages during pre-training, with out-of-the-box support for 35+ languages.
What this means
The encoder-free architecture in Gemma 4 12B Unified represents a shift toward simpler multimodal models that can be fine-tuned end-to-end in a single pass. By eliminating separate encoders, the model reduces multimodal processing latency and deployment complexity. The 256K context window and strong coding benchmarks (72% on LiveCodeBench v6) position it for long-context reasoning tasks and agentic workflows, while the 12B parameter count targets consumer-grade GPUs rather than requiring high-end infrastructure. The Apache 2.0 license allows commercial use without restrictions.
Related Articles
Mistral AI Releases Shieldstral-1.0-3B, a 3B-Parameter Policy-Adaptive Safety Classifier
Mistral AI has released Shieldstral-1.0-3B, a compact open-weight safety classifier that evaluates text and images against natural-language policies specified at inference time. The 3B model runs on a single GPU and reports F1 scores competitive with or exceeding larger moderation models like LlamaGuard-4-12B and GPT-OSS-Safeguard-20B on multiple benchmarks.
Mistral Releases Shieldstral, a 3B Open-Weights Safety Classifier That Matches Models 7x Its Size
Mistral has released Shieldstral, a 3B open-weights safety classifier that reframes content moderation as a policy-adaptive question-answering task. The model claims to match or outperform guard models up to 7x its size on text safety and multimodal benchmarks, and runs on a single 16GB GPU.
OpenAI Halts Parts of Astra Model Development After It Hit 'Critical' Cybersecurity Threshold
OpenAI disclosed that its in-development Astra model showed cyberattack capabilities strong enough that it cannot rule out a 'Critical' risk classification. The company has paused related internal activity and added security controls under its Preparedness Framework.
Mistral's 3B-Parameter Shieldstral Matches 20B Safety Model on Text Benchmarks
Mistral's new Shieldstral, a 3-billion-parameter open-weight safety classifier, posts an 84.9% F1 score on text benchmarks—tying OpenAI's GPT-OSS-Safeguard-20B, a model roughly seven times larger. The model lets operators define safety rules at runtime using plain-language yes/no questions instead of fixed taxonomies.
Comments
Loading...