Sakana AI releases JFinBench, a 321-question Japanese finance benchmark from 2026 certification exams
Sakana AI has released JFinBench, a Japanese financial benchmark of 321 questions drawn from the 2026 editions of six certification exams. Across 12 evaluated LLMs, Kimi K3.1 Pro falls about 10 points behind GPT-6 Astra, and Sakana AI says failures stem mainly from missing knowledge of Japan's systems rather than calculation errors.
Sakana AI has released JFinBench, a benchmark of 321 questions built from the 2026 editions of six Japanese financial certification exams. In the company's evaluation of 12 LLMs, Kimi K3.1 Pro, which is reported to match GPT-6 Astra on major benchmarks, falls about 10 points short of GPT-6 Astra on JFinBench. The announcement is in Japanese on Sakana AI's blog.
What JFinBench contains
The benchmark draws on the latest sittings of six exams, held between May and August 2026. Sakana AI used the top level of each exam, because easier questions make differences between models harder to see.
| Exam | Questions |
|---|---|
| Certified Public Accountant exam, short-answer section | 93 |
| Tax Accountant exam (all 11 subjects) | 40 |
| FP Skills Test Grade 1, written | 65 |
| Financial Window Service Skills Test Grade 1 | 66 |
| Nissho Bookkeeping Test Grade 1 | 24 |
| Zensho Bookkeeping Practice Test Grade 1 | 33 |
Of the 321 questions, 193 (60%) are multiple choice and 128 require written answers. Twelve questions require reading images, such as land plans, and include images in the input. Multiple-choice answers are extracted with regular expressions. Short answers, fill-in answers (with partial credit per blank) and essay answers (graded against rubrics built from model answers) are scored by an LLM judge. The overall score is the equal-weighted average of the six exam scores.
Models and evaluation setup
Sakana AI evaluated three proprietary models (Claude Fable 5, GPT-6 Astra, Gemini 3.8 Flash) and nine open-weight models: Kimi K3.1 Pro, Kimi K3, Kimi K2.6, DeepSeek V4.1 Flash 1017, Qwen3.8 Flash, MiMo-V2.6-Pro, MiMo-V2.6-Flash, GLM 5.3 Flash and Mistral Large 4.
The main setting uses no tools and a single API call. Reasoning effort was set to each model's maximum and minimum, and results are averages of three runs. Models without effort control were run with thinking enabled. Prompts state the exam date and the legal reference date, and grading follows the law in force at the time of the exam. Sakana AI also ran all models in a tool-enabled setting, where the model operates a shell in a Linux container with web access to look up laws and calculate across multiple turns.
Sakana AI says failures came mainly from gaps in knowledge of Japanese systems, not from calculation.
Construction pipeline
Sakana AI describes a human-in-the-loop agentic pipeline in three steps:
- A shared script transcribes each exam PDF page into layout-preserving Markdown with a VLM.
- A coding agent writes exam-specific code that splits questions, matches choices to answers, crops figures and removes page headers. It also generates side-by-side verification pages.
- A human checks all 321 questions against the originals and gives natural-language feedback, and the agent fixes the code.
Verification took about 2 hours per exam and about 12 hours in total. Sakana AI says the pipeline contains no finance-specific processing, so new exams or exam years need only agent-written exam-specific code and human verification.
Released resources
- Paper: arXiv 2611.18130
- Dataset: SakanaAI/JFinBench on Hugging Face
- Code: SakanaAI/JFinBench on GitHub
- Leaderboard: pub.sakana.ai/jfinbench
What this means
JFinBench measures knowledge tied to Japanese tax, accounting and financial rules, which Sakana AI says existing benchmarks do not capture. The reported gap between Kimi K3.1 Pro and GPT-6 Astra shows that parity on major benchmarks does not guarantee parity in this domain. Because the benchmark uses 2026 exams and a repeatable pipeline, Sakana AI says it can be refreshed with each new exam sitting while limiting the risk of training-data contamination.
Related Articles
Qwen4 Max scores 43 on Artificial Analysis Intelligence Index, 9 points above Qwen3.8 Max
Artificial Analysis reports that Alibaba's Qwen4 Max scores 43 on its Intelligence Index using xhigh reasoning effort. That is a gain of 9 points over Qwen3.8 Max, and 3 points behind Kimi K3.1 Pro.
Claude Fable 5.2 paints images through Python code, using about 700,000 simulated strokes in one piece
Anthropic's Claude Fable 5.2 produces images by writing Python programs that drive a simulated paint engine, with no external image model. Anthropic reports 64.9 percent on its internal RoboDojo Sim benchmark against 29.8 percent for OpenAI's GPT-6 Astra, though a real-world robot arm test showed limits in speed and fine motor control.
Gemini 4 Carbon tops MathArena at 85.4%, ahead of Claude Fable 5.2 and GPT-6.1 Sol
Google's Gemini 4 Carbon (high) leads the MathArena math benchmark with an expected performance of 85.4%, ahead of Anthropic's Claude Fable 5.2 (max) at 82.8% and OpenAI's GPT-6.1 Sol (max) at 80.6%. With the reasoning settings used in testing, Carbon costs roughly 15 times as much as GPT-6.1 Sol.
SPU wet-lab benchmark: top model Claude Opus 5.5 scores 0.486 and beats chance on only half of tasks
UniPat AI has released Science Prediction Universe (SPU), a benchmark of 116 tasks and 4,207 wet-lab experiments. Claude Opus 5.5 posted the highest average score, 0.486, but beat the random-ranking threshold on only half of its tasks. Raising GPT-6 Astra's reasoning effort multiplied runtime and cost without a visible score gain.
Comments
Loading...