AMD Releases Instella-MoE-16B-A3B-Think, a Fully Open Mixture-of-Experts Model Trained Entirely on AMD GPUs

AMD has released Instella-MoE-16B-A3B-Think, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token. The model was trained end-to-end — from pre-training through reinforcement learning — entirely on AMD Instinct MI300X and MI325X GPUs, using AMD's own ROCm software stack, Primus training framework, and Miles RL framework.

What's in the release

Unlike most model drops that ship a single final checkpoint, AMD published every stage of the training pipeline as a separate downloadable model:

• Instella-MoE-16B-A3B-Pretrain — base MoE model trained from scratch
• Instella-MoE-16B-A3B-Midtrain — refined on higher-quality data mixtures
• Instella-MoE-16B-A3B-Base — extended for long-context processing
• Instella-MoE-16B-A3B-SFT — instruction-tuned via supervised fine-tuning
• Instella-MoE-16B-A3B-DPO — aligned using direct preference optimization
• Instella-MoE-16B-A3B-Think — final checkpoint refined with reinforcement learning for chain-of-thought reasoning and instruction following

AMD says the full training recipe — data mixtures, hyperparameters, frameworks, and inference code — is published alongside the checkpoints, positioning this as a fully open release rather than an open-weights-only drop.

Architecture

The model uses a sparse MoE design with 64 total experts, 2 of which are shared, and 6 activated per token. It has 27 decoder layers, a hidden size of 2048, 16 attention heads, and a 128,896-token vocabulary. AMD says two architectural innovations underpin the training and inference efficiency: Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective, a communication-computation overlap technique described in a companion paper accepted at MLSys 2026 that AMD claims reduces blocking communication overhead in MoE training.

Benchmarks and evaluation

AMD evaluated the post-trained Think model using the OLMES framework with generation capped at 32,768 tokens, and reports pretraining results on standard benchmarks plus long-context evaluation via HELMET and RULER. The published model card includes comparison charts against similarly sized state-of-the-art models but does not surface specific numeric benchmark scores in the accompanying documentation reviewed for this article — AMD says a full technical report is forthcoming to accompany the existing arXiv preprint (2511.10628).

Availability and licensing

The model is distributed on Hugging Face in BF16/F32 safetensors format and has logged 730 downloads in the past month. It carries a ResearchRAIL license, restricting use to academic and research purposes. AMD explicitly states the checkpoints ship without safety guarantees and are not intended for factual-accuracy-critical, safety-critical, or medical applications. Multilingual capability has not been evaluated, according to AMD. No inference pricing applies since this is a downloadable open-weight release; no commercial inference provider currently hosts the model.

What this means

This release is notable less for raw performance claims — which remain unverified pending the full technical report — and more for what it signals about AMD's hardware and software ecosystem. By training a competitive-scale MoE model end-to-end on MI300X/MI325X GPUs using ROCm rather than NVIDIA's CUDA stack, AMD is building a public proof point that its silicon and software (Primus, Miles RL) can handle frontier-scale training pipelines, including RL post-training. The decision to release every intermediate checkpoint — pretrain, midtrain, SFT, DPO, and RL stages — is unusually transparent and gives researchers a rare look at how each training stage shifts model behavior. The research-only license, however, means this won't see production deployment soon; it's aimed squarely at academic and systems researchers evaluating AMD's training stack, not at developers looking for a drop-in commercial model.