Gemini 3.5 Flash ranks 6th in Android coding benchmark at 3x cost of Gemini 3.1 Pro
Google's latest Android Bench results show Gemini 3.5 Flash ranking 6th with a 63.7% success rate, despite averaging $147.10 per benchmark run compared to Gemini 3.1 Pro Preview's $47.90. The newer model used 355.9 tokens per run versus 73.3 for its predecessor, while GPT 5.5 leads the benchmark at 74% success rate.
Gemini 3.5 Flash ranks 6th in Android coding benchmark at 3x cost of Gemini 3.1 Pro
Google's Gemini 3.5 Flash placed sixth in the company's Android Bench coding benchmark, scoring 63.7% success rate while costing significantly more per run than competing models, including Google's own Gemini 3.1 Pro Preview.
Benchmark results
The Android Bench measures model performance across 10 runs of Android coding tasks, scoring each model by the percentage of cases it successfully solves. According to Google's latest results:
Top 5 performers:
- GPT 5.5: 74% success rate, $134.20 per run, 64.7 tokens average
- GPT 5.4: 72.4% success rate, $91.70 per run, 64.2 tokens
- Gemini 3.1 Pro Preview: 72.4% success rate, $47.90 per run, 73.3 tokens
- Claude Opus 4.7: 68.7% success rate, $124.30 per run, 90.0 tokens
- Claude Opus 4.6: 66.6% success rate, $84.40 per run, 69.5 tokens
Gemini 3.5 Flash:
- Score: 63.7% success rate
- Average latency: 14.2 seconds
- Average tokens: 355.9 per run
- Average cost: $147.10 per run
The model performed 9 percentage points worse than Gemini 3.1 Pro Preview while costing more than triple ($147.10 vs $47.90) and consuming 4.9x more tokens (355.9 vs 73.3).
Cost and efficiency comparison
Gemini 3.5 Flash was positioned as a cheaper, faster alternative to Gemini 3.1 Pro. In Android coding benchmarks, it shows higher latency and significantly increased resource consumption. GPT 5.5 achieved similar per-run costs ($134.20) while using 5.5x fewer tokens than Gemini 3.5 Flash.
Open-weight models occupy the lower rankings, with DeepSeek V4 Pro offering the lowest cost at $13.70 per run with a 55.4% success rate.
Rounding out the top 10
- GLM 5.1: 59.7% ($46.70)
- Kimi K2.6: 58.6% ($42.50)
- Claude Sonnet 4.6: 58.4% ($40.40)
- DeepSeek V4 Pro: 55.4% ($13.70)
- Claude Sonnet 4.5: 53.7% ($61.00)
Google has not yet published benchmark scores for Claude Opus 4.8 or Fable 5. The company removed GPT 5.3 Codex from the rankings since the previous update.
What this means
This benchmark reveals a performance-cost mismatch for Gemini 3.5 Flash in specialized Android development tasks, contradicting its general positioning as an efficient alternative. The results suggest model performance varies significantly by use case—while Gemini 3.5 Flash may excel at general tasks, Android coding appears to be a weak spot. For developers choosing coding models, the data shows GPT 5.5 and Gemini 3.1 Pro Preview deliver better value in this specific domain. The token consumption disparity (355.9 vs 73.3) indicates potential optimization issues in how Gemini 3.5 Flash approaches Android development problems.
Related Articles
Simon Willison's Pelican Benchmark Shows GPT-6 Astra Outperforming GPT-5.6 Sol at Every Reasoning Level
Developer Simon Willison ran his signature 'pelican riding a bicycle' SVG test on newly-accessed GPT-6 Astra across five reasoning levels, comparing results against GPT-5.6 Sol, Terra, and Luna. Even Astra's lowest reasoning setting reportedly beat every Sol output, though Astra costs roughly twice as much per token.
OpenAI's GPT-6 Astra Splits Benchmarks but Beats Human Efficiency on ARC-AGI-3, Pushing Chollet's AGI Timeline Forward
OpenAI's GPT-6 Astra rates first place on Epoch AI's aggregate benchmark but ties its predecessor on Artificial Analysis. Its human-beating move efficiency on ARC-AGI-3 led ARC Prize co-founder François Chollet to call progress 2x faster than expected.
Nvidia Claims Groq 3 LPX Hits 3,400 Tokens/Sec, 4x Cerebras — But Needs 64 Chips to Do It
Nvidia's new Groq 3 LPX inference accelerator hit 3,400 tokens per second on Gemma 4 31B, a figure the company says is four times faster than Cerebras. Experts note the benchmark uses at least 64 LPX chips versus Cerebras' one or two accelerators, making the comparison far less clean than it appears.
Artificial Analysis Launches Search Index Benchmark for AI Agent Search APIs
Artificial Analysis has released the Search Index, a benchmark measuring how search API providers perform for AI agents across quality, cost, and speed. Parallel, Exa, and Firecrawl lead the initial rankings, with search access boosting model scores from 33 to as high as 75 points.
Comments
Loading...