benchmarkMicrosoft

Microsoft's MAI Code 1.1 Flash Loses on Price and Performance to DeepSeek-V4-Flash

TL;DR

Microsoft's MAI Code 1.1 Flash beats its predecessor and mini-models from Anthropic and OpenAI on SWE-bench Verified, but DeepSeek-V4-Flash-0731 outperforms it on Terminal Bench 2.1 (82.7% vs 62.9%) while costing roughly a third as much per token.

2 min read
0

What happened

Microsoft released MAI Code 1.1 Flash, an updated code model powering GitHub Copilot. According to Microsoft, the model writes better code, is 25 percent more token-efficient, and costs a quarter of its June predecessor, MAI-Code-1-Flash. Microsoft says developers accepted 4 percent more of its output than the previous version, and that training used "hundreds of thousands of reinforcement-learning environments in GitHub Copilot."

The benchmarks

On SWE-bench Verified, MAI-Code-1.1-Flash scores 72.6%, ahead of MAI-Code-1-Flash (71.6%), Claude Haiku 4.5 (69.8%), and GPT-5.4 mini (69.2%). DeepSeek did not publish a comparable SWE-bench Verified score for its V4-Flash-0731 release.

On Terminal Bench 2.1, the picture reverses sharply. MAI-Code-1.1-Flash posts 62.9%, ahead of its predecessor (51.7%) and Haiku 4.5 (49.4%), and roughly in line with GPT-5.4 mini (60.7%). But DeepSeek-V4-Flash-0731 scores 82.7% on the same benchmark — nearly 20 points higher than Microsoft's model.

Pricing gap

DeepSeek-V4-Flash charges $0.14 per 1M input tokens ($0.0028 with caching) and $0.28 per 1M output tokens. MAI Code 1.1 Flash charges $0.20 per 1M input tokens ($0.02 with caching) and $1.20 per 1M output tokens — more than four times DeepSeek's output price. Claude Haiku 4.5 is the most expensive of the three, at $1.00 input and $5.00 output per 1M tokens.

Cost per token alone doesn't capture full usage efficiency, but the gap favoring DeepSeek is large enough that token-efficiency differences are unlikely to close it.

Notable omission

Microsoft's official announcement for MAI Code 1.1 Flash does not include direct benchmark comparisons against competitors. It cites internal metrics instead — "code survival rose 4% and return visits increased 9%" relative to the prior version — while the head-to-head benchmark table against Haiku 4.5, GPT-5.4 mini, and DeepSeek-V4-Flash-0731 appears only in the model card.

What this means

Microsoft has been vocal about supporting open AI development, but its own roadmap points toward proprietary, closed-weight models like MAI Code 1.1 Flash that underperform freely available alternatives such as DeepSeek-V4-Flash-0731 on some benchmarks while costing more per output token. This follows Microsoft's recent Copilot changes, where it replaced some OpenAI and Anthropic models with cheaper in-house MAI alternatives, trading capability for margin.

The pattern suggests strategy rather than accident: Microsoft controls the default model choice inside its own ecosystem, and most Copilot users never actively select a different model. Even a model that trails on independent benchmarks can capture significant usage share simply by being the default — regardless of whether cheaper, better-performing open alternatives exist elsewhere.

Related Articles

product update

Microsoft Confirms Copilot 'Super App' Merging Chat, Code, and Agents Ships This Year

Microsoft CEO Satya Nadella confirmed during a Wednesday earnings call that the company is merging Copilot chat, GitHub Copilot coding features, Cowork, and Autopilot agents into a single 'super app' launching this year. The move mirrors OpenAI's recent ChatGPT Work app, which combines ChatGPT and Codex.

benchmark

Composio Benchmark: Claude Code Fastest Agent Framework, But Costs Nearly 3x More Than OpenCode

Composio benchmarked DeepSeek V4 Flash across four agent frameworks—Claude Code, Codex, OpenCode, and Oh My Pi—on 30 real-world tasks. Claude Code finished fastest at 122 seconds per task but cost $0.195, nearly three times OpenCode's $0.073, while Oh My Pi had the highest success rate at 17/30 but took 272 seconds per task.

product update

Microsoft Confirms Copilot 'Super App' Merging Chat, Code, and Agents Launching This Year

Microsoft CEO Satya Nadella confirmed during an earnings call that a Copilot 'super app' merging chat, code, Cowork, and Autopilots will launch this year for both consumer and commercial users. The announcement follows OpenAI's own super app rollout, which the company has admitted is 'kind of a mess.'

model release

Microsoft AI Shifts Strategy to Cheap Specialist Models Over Frontier Chasing

Microsoft AI CEO Mustafa Suleyman says the company is prioritizing token efficiency and compact, single-purpose models over general-purpose frontier systems. New models MAI-Cyber-1-Flash and MAI-Image-2.5-Flash claim strong cost-performance gains, but rely on an orchestration layer that still routes hard tasks to OpenAI's reasoning models.

Comments

Loading...