Gemini 3.5 Flash ranks 6th in Android coding benchmark at 3x cost of Gemini 3.1 Pro
Google's latest Android Bench results show Gemini 3.5 Flash ranking 6th with a 63.7% success rate, despite averaging $147.10 per benchmark run compared to Gemini 3.1 Pro Preview's $47.90. The newer model used 355.9 tokens per run versus 73.3 for its predecessor, while GPT 5.5 leads the benchmark at 74% success rate.
Gemini 3.5 Flash ranks 6th in Android coding benchmark at 3x cost of Gemini 3.1 Pro
Google's Gemini 3.5 Flash placed sixth in the company's Android Bench coding benchmark, scoring 63.7% success rate while costing significantly more per run than competing models, including Google's own Gemini 3.1 Pro Preview.
Benchmark results
The Android Bench measures model performance across 10 runs of Android coding tasks, scoring each model by the percentage of cases it successfully solves. According to Google's latest results:
Top 5 performers:
- GPT 5.5: 74% success rate, $134.20 per run, 64.7 tokens average
- GPT 5.4: 72.4% success rate, $91.70 per run, 64.2 tokens
- Gemini 3.1 Pro Preview: 72.4% success rate, $47.90 per run, 73.3 tokens
- Claude Opus 4.7: 68.7% success rate, $124.30 per run, 90.0 tokens
- Claude Opus 4.6: 66.6% success rate, $84.40 per run, 69.5 tokens
Gemini 3.5 Flash:
- Score: 63.7% success rate
- Average latency: 14.2 seconds
- Average tokens: 355.9 per run
- Average cost: $147.10 per run
The model performed 9 percentage points worse than Gemini 3.1 Pro Preview while costing more than triple ($147.10 vs $47.90) and consuming 4.9x more tokens (355.9 vs 73.3).
Cost and efficiency comparison
Gemini 3.5 Flash was positioned as a cheaper, faster alternative to Gemini 3.1 Pro. In Android coding benchmarks, it shows higher latency and significantly increased resource consumption. GPT 5.5 achieved similar per-run costs ($134.20) while using 5.5x fewer tokens than Gemini 3.5 Flash.
Open-weight models occupy the lower rankings, with DeepSeek V4 Pro offering the lowest cost at $13.70 per run with a 55.4% success rate.
Rounding out the top 10
- GLM 5.1: 59.7% ($46.70)
- Kimi K2.6: 58.6% ($42.50)
- Claude Sonnet 4.6: 58.4% ($40.40)
- DeepSeek V4 Pro: 55.4% ($13.70)
- Claude Sonnet 4.5: 53.7% ($61.00)
Google has not yet published benchmark scores for Claude Opus 4.8 or Fable 5. The company removed GPT 5.3 Codex from the rankings since the previous update.
What this means
This benchmark reveals a performance-cost mismatch for Gemini 3.5 Flash in specialized Android development tasks, contradicting its general positioning as an efficient alternative. The results suggest model performance varies significantly by use case—while Gemini 3.5 Flash may excel at general tasks, Android coding appears to be a weak spot. For developers choosing coding models, the data shows GPT 5.5 and Gemini 3.1 Pro Preview deliver better value in this specific domain. The token consumption disparity (355.9 vs 73.3) indicates potential optimization issues in how Gemini 3.5 Flash approaches Android development problems.
Related Articles
Artificial Analysis Index v5 drops Kimi K3.1 Pro from 4th to 9th; Gemini 4 Carbon takes #1 at 59
Artificial Analysis released Intelligence Index v5 on Tuesday, with 60% of the test built on private, held-out questions. Moonshot AI's Kimi K3.1 Pro fell from 4th to 9th, and Google's Gemini 4 Carbon debuted at #1 with a score of 59.
Replit's ViBench 2.0: Top models pass every test plan in only 49-65% of full software builds
Replit released ViBench 2.0, a benchmark in which an agent builds a full software product feature by feature from an empty project. GPT-6.1 Sol leads with a 65% pass@1, followed by GPT-6 Astra at 60% and Opus 5.5 at 49%. No open-weight model passes a full build.
Artificial Analysis launches Intelligence Index v5; Claude Opus 5.5 (max) leads at 58
Artificial Analysis has launched Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades, including new coding and science benchmarks. For the first time, the majority of the Index is built on private datasets. Claude Opus 5.5 (max) leads at 58.
IBM and Kaggle Partner on Continuously Updated ScarfBench Leaderboard for Java Migration Agents
IBM and Kaggle are partnering on a continuously updated public leaderboard for ScarfBench, IBM's open benchmark for coding agents doing enterprise Java framework migrations. The benchmark covers migrations across Jakarta EE, Quarkus, and Spring, according to an IBM Research technical note published October 27, 2026.
Comments
Loading...