Allen Institute releases EMO, 14B parameter MoE model with selective 12.5% expert use

Allen Institute for AI (AI2) released EMO, a mixture-of-experts model that can maintain near full-model performance while using just 12.5% of its total experts for specific tasks. The model contains 14 billion total parameters with 1 billion active per forward pass, trained on 1 trillion tokens.

Model architecture and capabilities

EMO uses 8 active experts per token selected from a pool of 128 total experts. According to AI2, when limited to a 16-expert subset (12.5% of total experts), the model shows only a 3% absolute performance drop across benchmarks. At 32 experts (25% of total), the degradation is approximately 1%.

The model differs from standard MoE architectures in how experts specialize. Rather than organizing around low-level lexical patterns like prepositions or punctuation, EMO's experts form coherent groups around higher-level domains and capabilities.

Training methodology

EMO uses document boundaries as a supervisory signal during training. All tokens within a single document are constrained to route through the same subset of experts, rather than allowing each token to independently select experts. This document-level routing encourages groups of experts to specialize in consistent domains.

The model implements global load balancing across many documents rather than local balancing within micro-batches. This prevents the model from collapsing onto a small number of experts while still allowing document-level expert consistency.

Document pool sizes are randomly sampled during training rather than fixed, allowing the model to support different expert subset sizes at inference time.

Benchmark performance

On general-purpose benchmarks, EMO matches the performance of a standard MoE model with equivalent architecture trained on the same data, according to AI2. The performance advantage appears when using expert subsets: a standard MoE with the same architecture degrades sharply when limited to small expert subsets, while EMO maintains robustness.

Task-specific expert subsets are constructed by ranking experts based on routing usage on small validation datasets, then keeping only the most-used experts.

What this means

EMO demonstrates that mixture-of-experts models can be trained to support modular deployment without sacrificing general-purpose performance. The ability to use 12.5% of experts while maintaining near full-model performance addresses a key limitation in current MoE architectures, where all experts typically need to be loaded even for narrow tasks.

The approach avoids requiring predefined domain labels across the pretraining corpus, instead letting domain specialization emerge from document-level routing patterns. This could enable more flexible deployment options for large sparse models, particularly for users who need specific capabilities without the computational cost of hosting the full parameter set.

Code, models, and technical paper are available through AI2's GitHub and Hugging Face collections.