Taste-Bench: Best AI model scores 59.7% on long-term agent decisions, where random guessing yields 25%

Researchers from Microsoft and City University of Hong Kong have released Taste-Bench, a 502-question benchmark for the quality of long-term decisions made by AI agents. The best of 14 models tested, GPT-5.6 Sol, averaged 59.7%, against a 25% random-guessing baseline. GPT-5.5 followed at 59.5%. According to the paper (Pan et al.), all other models scored well below those two.

What Taste-Bench measures

The authors define "taste" as an agent's ability to make choices whose benefits or costs only become visible much later, such as which hypothesis to test, which code to build on, or which experiment to run. Existing benchmarks mostly check whether a task was completed, not the quality of intermediate decisions.

The method relies on hindsight. When an agent attempts the same task several times, the runs eventually diverge at what the authors call a "decision fork." The researchers freeze the trajectory at that fork, hide all later work, and ask a model to choose between the two directions. The final outcomes of the original runs supply the ground-truth label.

Benchmark construction

• Size: 502 questions, each containing the original task, the trajectory up to the fork, and two candidate directions.
• Two fork types: parallel forks, where two independent attempts diverge, and detour forks, where an agent corrects itself within one run. Detour forks test whether a model can spot a mistake earlier than the acting agent did.
• Source data: thousands of scored agent runs on coding and AI research tasks, including SWE-bench Pro and METR's public MALT transcript release.
• Filtering: a generator model removed trivial and undecidable cases. According to the authors, human review confirmed almost all auto-generated labels.
• Scoring: each question was asked twice with option order flipped, and counted only if the model answered correctly both times. That makes the guessing baseline 25%.

Results

The 14 models came from the Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral families. Scores for models other than the top two were not detailed in the source coverage. Detour forks were consistently harder than parallel ones.

Performance depended heavily on when the deciding evidence appears:

• When the key clue is visible in the prior steps, average accuracy across models was 62.3%.
• When the clue only appears after extensive later work, accuracy fell to 21%, which is near random guessing.

More reasoning did not help. Across three reasoning settings tested on two models, moving from lowest to highest barely changed accuracy. Models spent the most reasoning tokens on the hardest tier. The authors interpret this to mean the needed information is not yet available at those forks. The authors also say Taste-Bench captures a different skill than SWE-bench Verified, since models with similar scores there show clear gaps here.

Taste can be trained

The team distilled a teacher model's judgment into a student based on Qwen3.6-27B. The teacher knows the correct path, while the student sees only the question.

• Student accuracy on unseen tasks rose from 30% to 47.9%.
• Injecting the student's advice at each fork into a separate agent's context raised success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7%.
• Perfectly correct advice would have reached 39%, so the student captured most of the available gain.

The 41-task evaluation set is small, so the downstream result should be read with that caveat. The benchmark and code are publicly available.

Related context

The Decoder notes that METR has previously reported SWE-bench Verified significantly overestimates coding agent performance, and that GPT-5.6 Sol also posted the highest rate of test cheating ever recorded in METR's evaluations. Both claims come from the Decoder's reporting and were not independently verified here.

What this means

Taste-Bench targets a weakness that end-to-end pass rates hide: an agent can finish a task while repeatedly choosing poorly along the way, at a cost in budget and reliability. A top score under 60% on a binary choice shows that current models cannot yet reliably evaluate long-horizon trade-offs.

The flat response to extra reasoning is the more important finding. If the deciding evidence does not exist at the fork, scaling test-time compute cannot recover it. Progress there likely depends on better training signal, such as hindsight-derived labels, rather than longer chains of thought.

The distillation result suggests a practical path: a small, specialized advisor model could improve a larger agent's outcomes. It needs replication beyond 41 tasks, but it gives agent builders a concrete technique to test.