🔍 Read the full analysis: FrontierCode: What Makes An AI-Written Pull Request Reviewable? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 models on whether AI-generated code changes meet repository maintainers’ criteria. Claude Opus 5.5 led the results, but cleared all blocking checks in 57.7% of attempts; benchmark scores and costs vary widely. The results are from one benchmark setup and do not establish how models perform across all real-world pull requests.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that evaluates whether a proposed change is suitable for review and acceptance in a software repository. Claude Opus 5.5 led with a 52.2% score, but cleared every blocking check in 57.7% of attempts, showing that even the top result fell short of consistent success under the benchmark’s rules.
FrontierCode’s central question is whether a maintainer would accept a patch, rather than simply whether the code passes tests. Artificial Analysis used the same open-source agent harness, Stirrup, for all models, allowing up to 250 turns per attempt and three attempts per task. Its evaluation covered the benchmark’s 100-task Main set; the full benchmark contains 150 tasks.
The top results were closely grouped: GPT-6.1 Sol scored 49.1%, followed by Claude Fable 5.1 at 49.0%, GPT-6 Astra at 47.0% and Claude Sonnet 5.5 at 45.8%. Xiaomi’s MiMo-V2.6-Pro, described in the source as an open-weights model, scored 45.4%. Artificial Analysis reported that nine of the 100 tasks were not solved by any model on any attempt.
Scoring makes the benchmark stricter than a simple test pass. Checks cover hidden tests, command results, tests that must fail on the original code, adaptive tests, limits on file scope and diff size, and an LLM review against maintainer criteria. Failing any blocking check gives an attempt a score of zero; passing all blocking checks yields a score from 70% to 100%, with quality checks affecting the final result. Artificial Analysis used GPT-6 Luna (medium) for the review step.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Review Quality Changes the Ranking
The results matter because many coding evaluations focus on whether a model can produce working code, while maintainers must also judge whether a change is focused, tested and safe to merge. FrontierCode attempts to measure those additional demands. Its results suggest that passing tests alone is not a sufficient measure of review readiness, at least under the benchmark’s evaluation criteria.
The scores also put a limit on claims that software agents can handle pull requests without human oversight. The leading model passed every blocking check in 57.7% of attempts, according to Artificial Analysis. That is a benchmark result, not a direct estimate of how often the model would succeed in a company’s own repository. Still, it indicates that teams adopting such tools should distinguish assistance with coding from dependable, unattended changes.
Cost adds a practical trade-off. Artificial Analysis reported a per-task cost of $10.15 for Claude Opus 5.5 and $0.81 for GPT-6.1 Sol, whose scores differed by 3.1 percentage points. The source also identified GPT-6 Luna, MiMo-V2.6-Pro, GPT-6.1 Sol and Opus as points on its cost frontier, where no other tested model was both cheaper and higher-scoring. Those figures are specific to the evaluation setup and should not be treated as universal deployment costs.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was developed by Cognition with open-source maintainers. According to the source, it was announced in June 2026, with version 1.1 following in July and adding rules against looking up solutions online. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task. The tasks are private, intended to prevent models from having encountered their solutions in training.
The benchmark’s design targets behaviors that can make AI-generated pull requests difficult to review: overly broad diffs, edits to unrelated files and unhelpful tests. Cognition’s trajectory scanner can also assign a zero to attempts flagged for looking up a solution. The result is a score shaped by both correctness and the manner in which a model reaches its proposed change.
Results do not track perfectly with general model rankings. In the source’s comparison, Claude Sonnet 5.5 ranked second on the Artificial Analysis Intelligence Index but fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode. DeepSeek V4.1 Flash and GPT-6 Luna performed better on FrontierCode relative to their general-index positions. General capability rankings are not a substitute for task-specific evaluation, though this comparison covers only the models and measures listed in the source.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s FrontierCode announcement
Benchmark Limits and Missing Models
The results do not show how the same models would perform across different agent harnesses, repositories or review teams. The test used one common harness and a 100-task set, with GPT-6 Luna (medium) as the automated reviewer. The source does not provide enough information here to establish how closely that review process matches the judgment of human maintainers in everyday development.
Some models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their eventual scores could change the field’s ordering. It is also unclear from the reported results how performance would vary across repeated runs beyond the benchmark’s three attempts per task or how teams should translate these scores into expected success on their own codebases.
Further Results Could Shift Rankings
The next development to watch is whether results appear for the models missing from the October 13 comparison, particularly those the source says had performed well on other coding evaluations. Any new scores would add to the comparison but would need to be read against the same task set and scoring rules.
For engineering teams, the practical next step is to test candidate models against their own repositories and review standards rather than rely on a single leaderboard. FrontierCode provides a structured comparison of patch correctness, scope and review quality; it does not settle whether AI-generated pull requests can be accepted reliably without human review.
Key Questions
What does FrontierCode measure?
It scores AI-generated code changes against checks for correctness, regressions, test quality, scope and an LLM review based on maintainer criteria. A failure on any blocking check gives the attempt a zero.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) led the reported results with a 52.2% FrontierCode score. It cleared all blocking checks in 57.7% of attempts.
Does a 52.2% score mean the model passes 52.2% of tests?
No. The score combines benchmark rules and quality checks; it is not a percentage of tests passed. Artificial Analysis separately reported that Opus cleared every blocking check in 57.7% of attempts.
Can these results show how a model will perform on my team’s repository?
Not directly. The benchmark compares models on a defined task set and evaluation setup. Teams would need to test models against their own code, tools and review standards to estimate local performance.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
