mixture-of-experts

ModelBest published an engineering write-up on Lookahead Expert Scheduling (LES), the inference system behind MiniCPM-V 4.7 35B-A3B. The company reports over 20 tokens per second on an iPhone 18 Pro Max, and has open-sourced the code in its llama.cpp-omni repository.

MarkTechPost has published a config-verified comparison of how 10 open mixture-of-experts models route tokens. The title names DeepSeek, Qwen, Mixtral, Llama 4 and gpt-oss among the models compared.

Huawei's Ascend team says it has adapted Moonshot AI's Kimi K3.1 Pro for LoRA fine-tuning and validated training on a 128-card Atlas 950 SuperPoD. The model has about 2.8 trillion total parameters and a context window of up to one million tokens. Ascend claims a 15% average training performance gain from a caching mechanism, but has not published absolute throughput.

Tencent has released the official version of Hy4, with a 1.0M-token context window and visual understanding for the first time. It is listed on OpenRouter at $0.417 input / $1.251 output per 1M tokens, shown as 50% off.

inclusionAI has released Ling-3.1-flash-1019, a post-training upgrade of Ling-3.1-flash with 560B total parameters, 25B active per token, and a 1M-token context window. The company reports a score of 44 on the Artificial Analysis Intelligence Index v4.3.2, up from 41 for its predecessor.

iFLYTEK detailed SparkAudio 1.0 at its 2026 Global 1024 Developer Festival. The speech foundation model combines a 0.65-billion-parameter audio encoder with a 30B-A3B mixture-of-experts language model, trained on 13 million hours of audio. iFLYTEK also launched Robot Super Brain Platform 3.0 for embodied robots.

MiniMax has published MiniMax-M3.1-Flash on Hugging Face: a native multimodal mixture-of-experts model with about 428B total parameters, 23B activated per token, and a context window of up to 1M tokens. It builds on MiniMax-M3 with sparse attention, quantization-aware training and DSpark speculative decoding.

Recursive released Flint, a free, open-weight mixture-of-experts coding model with 40 billion total and 2.5 billion active parameters, under Apache 2.0. The company claims its Eureka Machine automated research system contributed to nearly every stage of development, and that Flint runs at about 131 tokens per second on an M4 Pro MacBook Pro.

NVIDIA has published Nemotron 3.5 Super VL in NVFP4 quantization on Hugging Face: a 120B-parameter, 12B-active hybrid Mamba-2/Transformer MoE with a 1M-token context window and text, image and video input. NVIDIA reports the quantized checkpoint stays close to BF16 on most benchmarks. It fits on one DGX Spark, one H200, two H100s or one GB200.

Tencent Hunyuan released Hy-Memory-30B-A3B, a 30B-parameter mixture-of-experts model with about 3B active parameters, built to run the write path in external memory frameworks. Tencent reports a 55.49 cross-harness average, ahead of Claude Opus 5 (54.62) and DeepSeek V4.1 Flash (53.64) on its own fixed 20% benchmark subsamples.

DeepSeek has released V4.1 Flash 1017, an October 17, 2026 snapshot built on its new Causal Encoder-Decoder architecture. It uses a 552B-parameter sparse MoE backbone with a 1M-token context window, native image understanding, and pricing of $0.15 input / $0.60 output per 1M tokens on OpenRouter.

Reflection AI has released Beam 1.0, an open-weight sparse mixture-of-experts model with 501 billion total and 23 billion active parameters, built for agentic coding. Weights ship under Apache 2.0 on Hugging Face, and the model is available through Amazon SageMaker JumpStart in two US regions.

Moondream 3 Preview is a vision-language model with 9B total and 2B active parameters (mixture-of-experts), a 32K-token context window and an optional reasoning mode. OpenRouter lists it at $0.30 per 1M input tokens and $2.50 per 1M output tokens, served by Prime Intellect.

Moondream has released Moondream 3.1, a mixture-of-experts vision-language model with 9B total and 2B active parameters and a 32K-token context window. It is listed on OpenRouter at $0.30 per 1M input tokens and $2.50 per 1M output tokens, with an optional reasoning mode.

Ollama v0.40.5 adds NVIDIA Nemotron 3.5 Super, a 120B-parameter hybrid Mamba-2/Transformer MoE vision-language model with 12B active parameters and context up to 1M tokens. It also adds a smaller sibling, Nemotron 3.5 Bolt, and fixes a Responses API bug with custom tool calls.

Nvidia has released Nemotron 3.5 Super VL, an open multimodal Mixture-of-Experts model with 120 billion total and 12 billion active parameters, built for continuously running enterprise agents. Two compact Bolt models for local PCs and edge devices launch alongside it, though the 4 GB variant ships later this month.

Liquid AI has published GGUF builds of LFM2.5-24B-A2B, a 24B-parameter model on the lfm2moe architecture, in sizes from 13.6 GB (Q4_0) to 48 GB (BF16). The Q4_0 file is a Quantization-Aware Distillation checkpoint trained to recover accuracy lost to 4-bit quantization.

Fish Audio has released Drama 3, a text-to-speech model with 35B total parameters and 3B active per token, built for multi-character, long-form audio. It supports up to 16 speakers per text, voice cloning, and streaming. Fish Audio reports first audio in under 100 ms on a single H200.

Liquid AI released LFM2.5-24B-A2B, an open-weight reasoning mixture-of-experts model with 24B total and 2B active parameters and a 131K-token context window. The company claims it is competitive with Qwen3.6-35B-A3B on agentic and tool-use tasks while decoding at 184 tokens/s on an Apple M5 Max.

OpenBMB has released MiniCPM-V 4.7 35B-A3B, the first mixture-of-experts model in the MiniCPM-V series, with open weights on Hugging Face. The company claims it runs at 13+ tokens/s using 3 GB of memory on an iPhone 17 Pro Max, and that it beats Qwen3.6 35B-A3B on overall score in less inference time in Instruct mode.

Reflection introduced Beam, a 501B-parameter mixture-of-experts model with 23B active parameters. Mistral said it is finishing Mistral Large 4, a 1T-parameter multimodal model with 49B active parameters. Both companies plan open-weight releases in October, and both are positioning the models against Chinese open-weight leaders.

Amazon Bedrock now offers GLM 5.3, Z.ai's 753B-parameter mixture-of-experts model, through managed APIs with cross-Region inference, prompt caching and service tiers. Access is limited to eligible enterprise customers. Pricing and context window were not disclosed in AWS's announcement.

Reflection AI has unveiled Beam, a text-only mixture-of-experts model with 501 billion total parameters, 23 billion active, and a 1 million token context window. The company claims it matches Z.ai's GLM-5.2 on advanced reasoning benchmarks while using 3-4x less inference compute. Weights and the full technical report are due later this month.

inclusionAI has released Ling 3.1 Flash, a hybrid reasoning mixture-of-experts model with 560B total and 25B active parameters and a 262K-token context window. It is listed as free on OpenRouter through NovitaAI. No benchmark scores have been published on the listing.

Ai2 released Olmo-core 3, an open training framework for large mixture-of-experts models. The company reports about 2.7× the throughput of its earlier FSDP-based implementation and benchmarks up to 1.2 trillion total parameters on 512 NVIDIA B300 GPUs. It is the infrastructure for Ai2's next MoE-based Olmo, not a model release.

Upstage has released Solar Mini 4, a compact mixture-of-experts model with 35B total parameters, 3B active parameters, and a 524K token context window. The model targets agentic workloads and is priced at $0.05 per 1M input tokens and $0.20 per 1M output tokens, a promotional 50% discount off standard rates.

Xiaomi's MiMo team has released MiMo-V2.6-Pro-RL, a 1.02-trillion-parameter sparse mixture-of-experts model with 42B active parameters, 1M-token context, and native text/image/video/audio processing. The model was trained via a single mixed reinforcement learning run spanning coding, agentic, visual, and cybersecurity tasks, with benchmark scores that Xiaomi claims approach or match Claude Opus 5 and GPT-5.6 on several agentic and coding tests.

Alibaba's Qwen team released Qwen3.8 Max (0902), a 2.4-trillion-parameter mixture-of-experts model with a 1M-token context window that accepts text, image, and video input. The snapshot is post-trained for coding, agentic workflows, and long-horizon task execution, priced at $2/$6 per 1M input/output tokens.

InclusionAI has released Ling 3.0 Flash Fin, a finance-specialized mixture-of-experts model built on Ling 3.0 Flash. The model activates 5.1B of its 124B total parameters and targets long-horizon investment planning tasks while retaining general reasoning, coding, and math capabilities.

Tencent has released Hy4 preview, a mixture-of-experts model with 770B total parameters and 49B active parameters, targeting coding agents and multi-step tool-use workflows. The model ships with a 1 million token context window and is priced at $0.834 per 1M input tokens and $2.501 per 1M output tokens.

Hugging Face's transformers v5.16.1 release adds support for GLM-5.3-Flash, a 320B total-parameter (18B active) multimodal mixture-of-experts model. Zhipu AI claims it outperforms GLM-5.2 while approaching Claude Opus 4.8 on coding and agentic benchmarks at one-tenth the cost.

DeepSeek has released V4 Flash Vision Exp, an experimental vision-enabled variant of DeepSeek V4 Flash 0731 that adds image understanding while matching the base model's text performance. The sparse mixture-of-experts model uses 13B active parameters out of 284B total and supports a 1M token context window.

Tencent has released Hy-MT2-30B-A3B, a mixture-of-experts translation model with 30B total parameters and 3B active parameters, supporting 33 language pairs and five Chinese dialect and minority-language pairs. The model is available through Tencent Cloud at $0.074 per 1M input tokens and $0.295 per 1M output tokens.

Hugging Face repositories for Qwen3.8-2.4T-A95B, its FP8 variant, and NVIDIA's Nemotron-3.5-Lightning-30B-A3B have surfaced, but neither company has published accompanying benchmarks, technical reports, or pricing.

DeepSeek has shipped the general availability release of DeepSeek V4 Pro, codenamed 0813, featuring a 1,049,000-token context window. The mixture-of-experts model is priced at $0.43 per million input tokens and $0.87 per million output tokens, and is live now on OpenRouter.

LG AI Research has released K-EXAONE 2.0, a 750-billion-parameter mixture-of-experts language model with 37B active parameters, a 262,144-token context window, and support for 10 languages. The model is open-weighted under Apache 2.0 and claims competitive results against Qwen3.5, GLM-5.1, and DeepSeek-V4 Pro on reasoning, coding, and long-context benchmarks.

Thinking Machines Lab has released Inkling Small, an open-weight multimodal mixture-of-experts model with 12B active parameters out of 276B total and a 524K token context window. The model targets reasoning, coding, agentic workflows, and multilingual use cases at $0.58 per 1M input tokens and $1.44 per 1M output tokens.

DeepSeek has released V4 Flash 0731, a sparse mixture-of-experts model with 13B active parameters out of 284B total and a 1049K token context window. The model targets coding, reasoning, and agent workflows, priced at $0.14 per million input tokens and $0.28 per million output tokens via OpenRouter.

Moonshot AI has released Kimi K3, an open-weight 2.8-trillion-parameter mixture-of-experts model with 104B activated parameters, a 1,048,576-token context window, and native multimodal support. The company describes it as the world's first open 3T-class model, built on a new Kimi Delta Attention architecture.

Thinking Machines Lab released Inkling, a mixture-of-experts model with 975B total parameters and 41B active parameters, trained on 45 trillion tokens across text, images, audio, and video. The Apache 2.0-licensed model supports up to 1M context and debuts alongside Inkling-Small (276B-A12B), marking what observers call the strongest US-based open-weight release to date.

Moonshot AI released Kimi K3 on July 16, 2026, a 2.8 trillion parameter mixture-of-experts model that ranks #2 on the Vals AI index and #3 on Artificial Analysis's Intelligence Index. The company will release the model's weights on July 27, making it the strongest open-weight model to date, surpassing all previous open releases including DeepSeek R1.

Meituan has released LongCat 2.0, a sparse mixture-of-experts language model with 48 billion active parameters out of 1.6 trillion total. The model features a 1,049,000 token context window and costs $0.30 per 1M input tokens and $1.20 per 1M output tokens.

Thinking Machines Lab released Inkling, a 975-billion-parameter mixture-of-experts model that uses 41 billion active parameters per task. The open-weight model was trained on 45 trillion tokens across text, image, audio, and video, marking the first public release from Mira Murati's AI startup.

Amazon SageMaker AI now supports serverless fine-tuning for NVIDIA Nemotron 3 Nano (30B parameters, 3B active) and Nemotron 3 Super (120B parameters, 12B active). The integration includes supervised fine-tuning, reinforcement learning with verifiable rewards (RLVR), and reinforcement learning from AI feedback (RLAIF).

Poolside has released Laguna XS 2.1, a 33B total parameter Mixture-of-Experts model with 3B activated parameters per token and a 262,144-token context window. The model achieves 70.9% on SWE-bench Verified and 63.1% on SWE-bench Multilingual, representing a 5.4% improvement over its predecessor on multilingual coding tasks.

Nex AGI has released Nex-N2-Mini, an open-source agentic mixture-of-experts model with a 262K-token context window. The model accepts text and image inputs and is priced at $0.025 per 1M input tokens and $0.10 per 1M output tokens.

Amazon Web Services has added three MiniMax models to Amazon Bedrock: M2, M2.1, and M2.5. The newest model, M2.5, uses a mixture-of-experts architecture with 230 billion total parameters and 10 billion active per token, trained specifically for agent-native execution and coding tasks.

Mistral AI has released Leanstral 1.5, an open-source 119B parameter mixture-of-experts model designed specifically for Lean 4 proof assistance. The model features 128 experts with 4 active per token (6.5B activated parameters), a 256k token context window, and multimodal input capabilities.

DeepReinforce has released Ornith-1.0, an MIT-licensed model designed for agentic coding tasks with variants ranging from 9B to 397B parameters. Built on top of Apache 2.0-licensed Gemma 4 and Qwen 3.5 base models, the company claims it achieves state-of-the-art performance among open-source models of comparable size on coding benchmarks.

DeepSeek has released two Mixture-of-Experts models: V4-Pro with 1.6 trillion parameters (49B activated) and V4-Flash with 284B parameters (13B activated), both supporting 1 million token context windows. V4-Pro requires only 27% of inference FLOPs and 10% of KV cache compared to V3.2 at 1M token context, trained on over 32 trillion tokens.