sparse attention

Alibaba's Qwen team has released Qwen3.8-Flash-Next, an open-weight model with 125B total parameters (6B activated) that previews architectural changes planned for Qwen4, including a new sparse attention mechanism and n-gram embeddings. The model natively supports 262,144 tokens of context, extensible to 1 million.

MiniMax has released M3, a multimodal foundation model supporting text, image, and video inputs with a 1-million-token context window. The model costs $0.30 per million input tokens and $1.20 per million output tokens, available through OpenRouter.