Google has updated Android Bench, its benchmark for measuring how large language models handle Android app development, and the company’s own Gemini model is still not the one to beat.
The refreshed leaderboard adds eight models to the test: Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus, and Qwen 3.7 Max. Google said the benchmark remains built around 100 Android development tasks, with the leaderboard measuring accuracy as well as cost and efficiency.
That mix matters for developers who are using coding agents as more than autocomplete with a nicer demo reel. A model that patches one Android bug cleanly may be slow, expensive, or wrong on another. Android Bench is Google’s attempt to put some repeatable numbers behind those tradeoffs.
According to Google’s updated results, Gemini 3.1 Pro ranks fifth. The models Google identifies ahead of it include GPT 5.4, Claude Sonnet 5, and Claude Fable 5. Fable 5 leads the test with 84.5 percent accuracy, according to Google’s leaderboard.
The accuracy table is only half the story. Google’s numbers show that Fable 5 and GPT 5.5 are costly to run, each using more than $130 in tokens across the 100-problem, 10-run benchmark. Gemini 3.1 Pro scores lower, but Google lists its benchmark cost at $87.
Gemini 3.5 Flash, a model positioned as cheaper to operate than larger models, looks worse under this particular workload. Google’s leaderboard puts its run cost at $165, the highest shown, because it took far longer to complete the test. Google lists its runtime at 28 hours.
Google moves the benchmark to Harbor
Google also changed the machinery underneath Android Bench. The company said it has moved the benchmark to Harbor, a testing sandbox meant to make it easier for developers to run tasks, evaluate models, and share results.
That change means the old numbers and the new numbers are not a perfect before-and-after comparison. Google said it reran its earlier tests in Harbor to establish a new baseline, and some reported scores shifted even though the underlying tasks have not changed. The company said the previous results will remain available in an archive.
Google wants outside developers to feed more work into the benchmark. The company said developers can run their own Android development tasks against Android Bench and submit them for possible inclusion in the official suite. Google has updated the Android Bench GitHub repository with the new dataset and participation instructions.
The useful part of Android Bench is not that it crowns a universal coding champion. It is that it makes a narrower claim: here is how these agents performed on this defined set of Android tasks, under this test harness, at these costs. For teams choosing a model to touch production code, that is a better starting point than a product video and a vibes-based leaderboard.
This story draws on original reporting from Ars Technica.
