AWS Benchmark: OpenAI's GPT-5.6 Luna Beats GPT-5.4 Mini on Cost-Per-Correct-Answer Despite Similar List Price

The headline number: cost per correct answer, not per token

An Amazon Web Services blog post argues that comparing large language models on price-per-million-tokens alone misrepresents real deployment costs, and backs the claim with benchmark data comparing three OpenAI models available on Amazon Bedrock — gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol — against two cost-optimized OpenAI API baselines, gpt-5.4-mini and gpt-5.4-nano.

Using an open-source harness (openai-on-aws/benchmarks-openai) that runs all five models through an identical OpenAI Responses API code path, AWS measured three things: single-call accuracy and cost on AIME competition mathematics, GPQA Diamond, and MMLU-Pro; multi-turn agent trajectory costs on live web-research tasks (DeepSearchQA); and pass rates on GDPval, a set of 48 real occupational deliverables graded against rubrics written by professionals averaging 14 years of experience. Sample sizes across benchmarks ranged from 48 to 198 items, which AWS says should be treated as directional rather than statistically definitive.

Accuracy gaps and price cuts change the math

On AIME, gpt-5.6-sol solved 75% of problems versus mini's 37%; it also led GPQA Diamond (68% vs. 43%) and MMLU-Pro (82% vs. 59%), according to the benchmark data. Because gpt-5.6-luna ran with reasoning disabled on Bedrock, it used fewer billed tokens than mini running at default settings, making it 25% cheaper per correct AIME answer even at its original list price.

AWS says a July 30, 2026 price reduction on Amazon Bedrock cut gpt-5.6-luna pricing by 80% and gpt-5.6-terra by 20%. Following that cut, the recorded cost per correct AIME answer was $0.0021 for luna versus $0.0139 for mini — roughly a 6.6x difference. The result files used in the benchmark list luna at $0.22/$1.32 per 1M input/output tokens and terra at $2.20/$13.20 per 1M tokens; AWS notes these should be confirmed against live Bedrock pricing before use, since pricing-page updates may lag announced parity changes.

Agent turn count as a hidden cost driver

Because the harness uses client-managed conversation history (store: false), every agent turn re-sends the system prompt and accumulated context, so billed input tokens can grow roughly quadratically with turn count. On a 50-question DeepSearchQA sample using live web-search and page-fetch tools, gpt-5.4-mini averaged 7.6 turns per question — the most of any model tested — accumulating 114,000 input tokens per question versus terra's 50,000 (2.3x more).

Despite terra's higher per-token price, its cost per passing answer was $0.31 versus mini's $0.40, with mean F1 scores of 0.50 versus 0.39. Luna reportedly combined fewer turns than mini with a cost of $0.05 per passing answer. Nano, despite a lower nominal token price, passed only 18% of questions, pushing its cost per passing answer to $0.07 — higher than luna's $0.05.

What this means

This is a vendor-published benchmark from AWS comparing OpenAI models specifically as deployed on Amazon Bedrock, run with reasoning disabled, against OpenAI API defaults — a comparison of deployment configurations rather than controlled model capability. The findings should be read as directional given small sample sizes (48-198 items) and the fact that AWS itself commissioned and ran the evaluation. That said, the core methodological point is sound and broadly applicable: token list price is a poor proxy for total cost when accuracy, retry behavior, and multi-turn context growth are ignored. Teams running agentic workloads with long tool-calling chains should independently benchmark turn count and token accumulation, not just per-call pricing, before choosing between frontier and cost-optimized model tiers.