Every AI Code Review Benchmark, Compared (2026)

Direct answer

Every published catch-rate number in AI code review was produced by a company selling a reviewer, with five exceptions. We catalogued 19 benchmarks as of 2026-10-18. Four come from parties with no reviewer to sell, one is vendor-published but fully auditable, and the other 14 are vendors scoring themselves - each of which wins its own test. No party without a commercial stake has published a false positive rate for any shipping commercial AI code reviewer. Four different vendors have separately declared themselves #1 on the same leaderboard.

Where can I compare AI code review tools?

Direct answer: Compare them here, then verify against the primary benchmark. We list all 19 published AI code review benchmarks with publisher, conflict of interest, sample size, metric and winner. Only 4 of 19 were published by a party that does not sell a reviewer. One more, GitHub ReviewBench, is auditable vendor research. Verified 2026-10-18.

The reason this page exists is that the comparison market is a hall of mirrors. A buyer searching for “best AI code review tool” lands on a vendor blog, reads a percentage, and has no way to tell whether that percentage came from a controlled experiment or a marketing brief. Usually it is the second one.

So we did the boring work. We traced every benchmark number we could find back to its publisher, checked whether that publisher sells a code reviewer, and recorded what the benchmark actually measured.

The pattern is not subtle. In all 14 vendor benchmarks we found, the publisher came first.

Who published each AI code review benchmark, and do they sell a reviewer?

Direct answer: 14 of 19 benchmarks were published by companies selling an AI code reviewer, and all 14 are won by their publisher. The 4 neutral ones are Martian Code Review Bench, CRJudgeBench, SWE-PRBench and CR-Bench. Only 5 of 19 publish raw data. Only 7 of 19 count false positives at all. Verified 2026-10-18.

This is the table. One row per benchmark.

Benchmark	Published by	Sells a reviewer?	Sample	Metric reported	Raw data public	False positives counted	Who won
c-CRAB (arXiv 2603.23448)	NUS + Zhejiang Univ + SonarSource	Partly - Sonar sells SonarQube	184 PRs / 67 repos / 234 comments	Executable test pass rate	Yes	No	No winner declared. Claude Code highest at 32.1%
Martian Code Review Bench	Martian	No - sells model routing	~300k PRs live / 50 PRs offline	F1, precision, recall	Yes, open source	Yes (precision proxy)	Disputed. CodeRabbit, Qodo and Baz each claim #1
CRJudgeBench (arXiv 2609.37216)	Academic	No	Review-comment judging set	Detection accuracy	Yes	n/a (judges comments)	Sentinel, the authors’ own model
SWE-PRBench (arXiv 2603.26130)	Deepak Kumar, independent	No	350 PRs, 100-PR paper sample	Overall score, detection rate, FPR	Yes (HuggingFace + GitHub)	Yes - FPR per model	Claude Haiku 4.5, 0.153 overall
CR-Bench (arXiv 2603.11078)	Nutanix	No	584 instances, 174 verified	Precision, recall, F1, SNR	Paper	Yes	GPT-5.2 Reflexion, 32.76% recall / 5.10% precision
CRBugBench (arXiv 2610.13668)	CMU + Microsoft Research	Yes - Microsoft owns Copilot	1,800 tasks / 98 OSS + 2 private repos	recall@5 only	Partial	No precision at all	claude-opus-5, 79.6% pre-merge recall@5
GitHub ReviewBench	GitHub	Yes - Copilot	219 PRs / 187 repos / 19 languages	Grounded + augmented precision, recall, Fβ	Yes, MIT + public runner	Yes	Copilot Code Review, 40.1% grounded F1 (snapshot 2026-10-12)
Entelligence Code Review Benchmark 2026	Entelligence	Yes	50 PRs / 5 repos / 67 golden comments	F1, precision, recall	Partial	Yes	Entelligence, 47.2% aggregate F1
Greptile Benchmarks	Greptile	Yes	50 PRs / 5 repos	Catch rate	No	No - explicitly excluded	Greptile, 82%
Augment 7-tool re-run	Augment Code (now Harness)	Yes	50 PRs / 5 repos, key rebuilt	Precision, recall, F-score	Yes	Yes	Augment, 59% F-score
DeepSource security benchmark	DeepSource (now Harness)	Yes	165 CVEs from OpenSSF set	F1	No	Yes	DeepSource, 84.51% F1
Qodo benchmark	Qodo	Yes	Own set, plus Martian claim	F1	No	Yes	Qodo, 60.1% own / 64.3% Martian claim
Propel benchmark	Propel Code	Yes	Augment’s dataset	F1	No	Yes	Propel, 64% F1
Macroscope benchmark	Macroscope	Yes	118 seeded bugs	Detection rate	No	No	Macroscope, 48.31%
Panto comparison	Panto	Yes	17 PRs, o3-mini classifier	False positives per PR	No	Yes	CodeRabbit 2 vs Greptile 11 (reported, not confirmed)
CodeAnt “300K-PR benchmark”	CodeAnt AI	Yes	Stated 300,000 PRs	Unclear	No	Unclear	CodeAnt
Baz leaderboard claim	Baz	Yes	790 PRs	Precision, F1	No	Yes	Baz, 70.9% precision / 52.5% F1
OpenSSF CVE Benchmark	OpenSSF (dataset only)	No (dataset) / Yes (only runner)	165 CVEs	Varies by runner	Yes (dataset)	n/a	DeepSource - the only published run
Sphinx (arXiv 2601.04252)	Academic + industry co-authors	Unverified	Unverified	BLEU, ROUGE and others	Partial	Unclear	Unverified

Read the “who won” column top to bottom. Every vendor row names its own publisher. That is 14 for 14.

Where to find independent AI code review tool reviews

Direct answer: Four benchmarks have no reviewer to sell: Martian Code Review Bench, CRJudgeBench, SWE-PRBench (arXiv 2603.26130) and CR-Bench (arXiv 2603.11078). Critical limit: SWE-PRBench and CR-Bench measure base models in a fixed harness, not shipping products. No independent precision figure exists for CodeRabbit, Greptile, Qodo or any other commercial reviewer. Verified 2026-10-18.

The nuance in that paragraph matters more than the headline, so let me be exact about it.

SWE-PRBench is the only benchmark in this market that publishes a false positive rate, and almost nobody has noticed. Its harness reports an overall score, an augmented detection rate and an FPR per model. Published figures as of 2026-10-18:

Model	Overall score	Detection rate	False positive rate
Claude Haiku 4.5	0.153	0.306	0.346
Claude Sonnet 4.6	0.152	0.297	0.227 (lowest)
DeepSeek V3	0.150	0.312	0.315
Mistral Large 3	0.147	0.305	0.353
GPT-4o	0.113	0.220	0.193
GPT-4o-mini	0.108	0.210	0.353
Mistral Small	0.106	0.257	0.251
Llama 3.3 70B	0.079	0.223	0.417 (highest)

But these are base models in a fixed prompt harness, not products. You cannot buy “Claude Haiku 4.5 reviewing a pull request at 0.346 FPR.” You buy CodeRabbit or Greptile or Cursor Bugbot, each of which wraps models in retrieval, repo indexing, severity filters and judge agents that change the output completely. SWE-PRBench tells you the floor for an unscaffolded model. It tells you nothing about any shipping reviewer.

CR-Bench has the same shape and the same limit. Nutanix built a single-shot agent and a Reflexion agent, ran them on GPT-5.2 and GPT-5-mini, and published precision alongside recall and a signal-to-noise ratio. The numbers are brutal: GPT-5.2 single-shot hit 27.01% recall at 3.56% precision, and the Reflexion variant improved recall to 32.76% while precision rose only to 5.10% and SNR fell from 5.11 to 1.95. The paper’s own framing is that false positives are costly, which is the correct framing and the reason it reports precision at all.

So the precise, defensible version of our thesis:

No party without a commercial stake has ever published a false positive rate, a precision figure, or a noise measurement for any shipping AI code review product. Not one, as of 2026-10-18. Everything buyers see in that column is either a vendor number or a measurement of a bare model.

That is the gap. We are filling it. Our protocol is published here.

Four vendors have each declared themselves #1 on the same leaderboard

Direct answer: Martian Code Review Bench is the one neutral leaderboard, because Martian sells model routing rather than a reviewer. CodeRabbit claimed #1 at 51.2% F1 on roughly 300,000 PRs on 3 March 2026. Qodo claimed #1 at 64.3% F1 on 15 March. Baz claimed #1 at 70.9% precision on 790 PRs. One leaderboard, four winners. Verified 2026-10-18.

Martian’s Code Review Bench is structurally the best thing in this market. Martian sells model routing, so it has no reviewer to flatter. It is open source. It runs two tracks: an offline track against 50 pull requests with human-verified golden comments, and an online track that continuously samples fresh real-world pull requests where review bots left comments, then measures bot suggestions matched to real fixes.

The online track is the strongest behavioural signal available, because fresh pull requests cannot be in anyone’s training data and because developer behaviour is hard to game.

Then it got used the way leaderboards always get used.

• 3 March 2026: CodeRabbit announced it topped the online track at 51.2% F1, 49.2% precision, 53.5% recall across nearly 300,000 pull requests in January and February.
• 15 March 2026: Qodo announced it was first at 64.3% F1.
• Then Baz announced it was first on precision at 70.9% and on F1 at 52.5% - on 790 pull requests.

Three self-declared first places inside twelve days, from one continuously-updating leaderboard, on sample sizes differing by nearly three orders of magnitude. CodeRabbit’s 300,000 and Baz’s 790 are not the same kind of measurement and should never appear in the same sentence without their denominators.

The structural lesson: a continuously-updating leaderboard with no snapshot discipline is a vendor press-release generator. If a vendor cites a leaderboard position without a date, the citation is worthless. If it cites one without a sample size, it is worse than worthless.

Greptile scored itself 82%. Augment’s re-run scored it 45%.

Direct answer: Greptile scored Greptile 82% on 50 PRs across 5 repos. Augment re-ran the same 5 repositories and scored Greptile 45%. Two things explain most of the 37-point gap: Augment rebuilt the answer key by hand, and Greptile’s catch rate explicitly excludes false positives while Augment reports an F-score that penalises them. Verified 2026-10-18.

This is the single most-cited pair of numbers in AI code review, and almost every citation of it is unfair to both parties. Here is the honest version.

Greptile’s methodology excludes false positives by design. Its benchmarks page states the rule plainly: scoring considered only detection of the original bug, and false positives, style suggestions and unrelated comments did not affect the catch rate. That is a recall measurement labelled as a catch rate. On a recall-only measure, 82% is not implausible.

Augment’s 45% is an F-score, not a catch rate. Greptile scored 45% precision, 45% recall and a 45% F-score in Augment’s table. The recall numbers are 82% versus 45%, which is still a large gap, but it is a 37-point gap between two different quantities measured against two different keys.

And Augment rebuilt the key. In its own words: “We expanded and corrected the golden comments by reviewing each PR manually, verifying issues, and validating them against tool outputs.” A larger, hand-corrected golden set mechanically lowers recall for every tool, because there is more to find.

The honest reading is that the answer key explains much of the gap. Not all of it - Augment won its own benchmark at 59% F-score, and that should be noted - but much of it. Anyone presenting 82-versus-45 as pure vendor dishonesty is skipping the methodology, and the methodology is the finding.

Then a third publisher weighed in. Macroscope’s 118-bug set put Greptile at 23.61%. Same tool, three publishers, three numbers across a 58-point range. The spread is the information. If you want the head-to-head detail, we keep it in CodeRabbit vs Greptile.

DeepSource wrote the best headline in the category, then omitted itself from it

Direct answer: DeepSource published a post titled “Every AI code review vendor benchmarks itself, and wins,” listing Greptile, Qodo, Augment, Propel, Macroscope and Entelligence with their figures. It omits DeepSource’s own 84.51% F1 self-score. DeepSource’s defence is that its page says Security Benchmarks, not AI Code Review Benchmarks. Verified 2026-10-18.

The post is deepsource.com/blog/ai-code-review-benchmarks, and its thesis sentence is better than anything we could write: “Several AI code review tools have published benchmarks. The problem: there’s no SWE-bench for code review.” It goes on to say that each vendor runs their own benchmark, on their own dataset, and wins.

Correct on all counts. The table it publishes lists Greptile at 82% recall, Greptile at 45% in Augment’s retest, Qodo at 60.1% F1, Augment at 59% F1, Propel at 64% F1, Macroscope at 48% detection rate and Entelligence at 47.2% F1.

DeepSource’s own 84.51% F1 - the highest self-reported figure in the market - is not in the table.

To be fair to DeepSource, it has an answer, and it is on the record: “The page says ‘Security Benchmarks,’ not ‘AI Code Review Benchmarks,’ because that distinction matters.” The 84.51% comes from a run on 165 CVEs from the OpenSSF CVE Benchmark, which is a security-detection task rather than a general review task. That is a real distinction.

It is also the exact distinction every vendor draws when its number is inconvenient. And it is worth noting that the OpenSSF CVE Benchmark dataset is genuinely independent while the only published run on it is DeepSource’s own self-winning one. An independent dataset with one vendor run on it is not an independent evaluation.

Since 9 September 2026, DeepSource is a Harness product. Since 8 October 2026, so are Augment Code’s Cosmos, Auggie CLI and Code Context Engine. Two of the most-cited benchmark publishers in this market now sit inside the same company.

The “40 to 70 percent false positive rate” attributed to c-CRAB does not exist

Direct answer: The 40-70% false positive rate universally attributed to c-CRAB (arXiv 2603.23448) is not in the paper. The phrase false positive appears twice, both conceptual, and the cited instance is about tuning the authors’ own comment classifier, not the reviewed tools. c-CRAB reports no false positive rate for any tool. Verified against arXiv 2603.23448v3, 2026-10-18.

This miscitation is now load-bearing in dozens of comparison pages, and it needs to stop.

What c-CRAB actually is. It converts human review comments into executable tests, so a review agent’s output can be checked by running code rather than by asking a model. Verbatim from the paper: “The resulting benchmark contains 184 pull request instances and 234 validated review comments, each associated with at least one executable test.” Dataset: 184 pull request instances, 67 repositories, 234 validated review comments.

What it actually reports. Verbatim: “Considering the union across all four tools, 97 out of the 234 tests were passed by at least one tool (41.5%), which shows there is still a substantial gap between AI-generated reviews and human reviews, even when multiple tools are combined.” Per tool: Claude Code 32.1%, Devin 24.8%, PR-Agent 23.1%, Codex 20.1%.

Where the miscitation comes from. 41.5% pass means 58.5% fail. Somebody inverted a pass rate into a “false positive rate,” rounded it into a range, and the range propagated. A failed test means the agent missed something. It says nothing about whether the agent also said things that were wrong. Those are opposite errors and conflating them is the single most common mistake in this category.

And one correction to our own earlier framing. c-CRAB is routinely labelled the independent academic benchmark, including by us. Checking the author affiliations on arXiv 2603.23448v3, the paper lists three institutions: the National University of Singapore, Zhejiang University, and SonarSource. Sonar sells SonarQube and acquired Gitar on 21 May 2026. SonarQube is not among the four tools evaluated, which blunts the conflict considerably, and the paper is careful, test-based and openly published. But “academic” and “disinterested” are different words, and any provenance table that lists c-CRAB as conflict-free is incomplete. Ours used to be. It is not now. One disclosure, because we hold other publishers to it: c-CRAB is not conflict-free. Five of its six authors are academic (National University of Singapore, Zhejiang University), but one lists SonarSource as their affiliation (arXiv 2603.23448v3, verified 2026-10-18), and the paper carries a disclaimer that its conclusions do not represent SonarSource policy. Sonar sells SonarQube and acquired Gitar, which is a paid sponsor of this site. SonarQube is not among the four tools c-CRAB evaluates, which blunts the conflict considerably - but we are not going to call out vendor-funded benchmarks elsewhere and stay quiet about this one.

Even the academic benchmarks are not disinterested

Direct answer: CRBugBench (arXiv 2610.13668) is a CMU and Microsoft Research collaboration that used copilot-cli as its agent harness, and Microsoft owns GitHub Copilot. It reports recall@5 and no precision at all. c-CRAB lists SonarSource as a co-author affiliation. Academic publication is a methodology signal, not an independence signal. Verified 2026-10-18.

CRBugBench is good work. 1,800 review tasks across 98 open-source repositories plus 2 proprietary industry repositories in 5 languages. Its best performer, claude-opus-5, catches 79.6% of bugs found before merge and 64.2% of bugs found after merge within its top five findings, at $2.97 per task. Human and agent together reach 82.1%, above the agent’s 79.6% alone, which is the paper’s actual thesis and a more interesting one than any leaderboard.

Two things to attach to every citation of it.

First, it reports no precision and no false positive figure. recall@5 means the right answer was somewhere in five findings. It says nothing about the other four, and nothing at all about the twenty findings a tool posts on a pull request where there is no bug. A recall-only benchmark cannot tell you whether a tool is usable.

Second, Microsoft Research co-authored it, Microsoft owns GitHub and Copilot, and the primary agent harness was copilot-cli. That does not invalidate anything. It does mean the standard “it’s peer-reviewed, therefore neutral” move does not work here.

GitHub ReviewBench is a third category: auditable vendor research

Direct answer: GitHub ReviewBench (5 October 2026) is vendor-published but openly auditable: MIT-licensed dataset, 219 PRs from 187 repos in 19 languages sampled from 103.9M pull requests, public rubric, self-serve runner at review-bench.ai, and 96.6% inter-rater agreement on golden labels. We propose it as a distinct tier between independent and marketing. Verified 2026-10-18.

The independent-versus-vendor binary is too crude, and ReviewBench is the proof. GitHub sells Copilot. GitHub also published the most methodologically transparent benchmark in the category. Both are true.

What makes it auditable.

• Dataset is public and MIT-licensed. 219 pull requests, 187 public open-source repositories, 19 languages. TypeScript in 68 PRs, Python in 41, C# in 25, Go in 19, JavaScript in 15.
• The sample is characterised against reality. GitHub analysed 103.9 million pull requests to model the real distribution of review workloads, then matched language and repository-size distributions. One deliberate deviation: pull request size is weighted toward the reviewable middle and tail, so tiny single-file changes are deliberately underrepresented. Size spread: 17 PRs at 50 lines or fewer, 40 at 51-200, 44 at 201-500, 40 at 501-1,000, and 78 above 1,000 lines.
• No repository dominates. The most-represented repository contributes 10 PRs, 4.6% of the corpus.
• Ground truth is multi-sourced then deduplicated. Candidate findings come from real human reviewers, issues inferred from author follow-up commits, deterministic analysis tools, and multiple frontier models across families. Overlapping findings are semantically deduplicated and checked against a shared rubric, rather than treating a finding’s origin as proof it is correct.
• Inter-rater agreement is published. Senior engineers who did not build the benchmark independently re-labelled every golden finding from scratch and agreed with ReviewBench’s true-and-false-positive judgments 96.6% of the time. This is the best published agreement figure in the category and nobody else reports one at all.
• Two metric families. Grounded precision and recall count only matches against known golden findings. Augmented metrics also credit real issues the golden set missed, so a tool is not punished for finding something new. β in the Fβ score is adjustable, so you can weight precision or recall to your own tolerance.
• Anyone can run it. scripts/try-agent.sh validates a container against a 25-PR test set, then registration at review-bench.ai/submit puts an agent through three rounds on the full 219.

And the caveats, which belong in the same breath.

• GitHub wins it. Copilot Code Review (Balanced, effort=medium) led at 40.1% grounded F1, 87.8% grounded precision, 26.0% grounded recall, 49.7% augmented F1 on a 2026-10-12 snapshot. Martian’s leaderboard ranks Copilot fourth.
• A fixed public set is hill-climbable. Publishing the 219 PRs and the rubric is what makes the benchmark auditable and is also what lets a vendor tune against it. Both effects are real.
• The judge is a model. Claude Sonnet 5 grades whether a finding is true, relevant and non-trivial. The 96.6% human agreement is strong evidence this works, and it is still an LLM in the scoring loop.
• Submissions cost money. Tuning runs burn your own inference credits for both your agent and the judge. ReviewBench covers judge inference only for a final submission. Small vendors will submit less often than large ones, which skews the leaderboard population.

Our proposal: three tiers, not two. Independent (no reviewer sold). Auditable vendor research (reviewer sold, but dataset, rubric and runner public). Vendor marketing (a number and no method). ReviewBench and Martian are the only entries in the first two tiers with real methodology behind them. Everything else is tier three.

Benchmarks decay, and most published scores are already stale

Direct answer: Any GitHub Copilot score measured before September 2026 describes a product that no longer exists - the 300-file and 20,000-line PR caps were removed on 27 August 2026. Ownership also moved: Harness acquired DeepSource on 9 September 2026 and Augment Code’s Cosmos, Auggie CLI and Code Context Engine on 8 October 2026. Verified 2026-10-18.

Three decay mechanisms, all of which invalidate specific published numbers.

1. Product changes invalidate the measurement. Copilot’s 300-file / 20,000-line pull request cap was removed on 27 August 2026, and bot-authored pull requests became supported at the same time. Every benchmark that scored Copilot on large pull requests before September 2026 - including Augment’s 20% precision / 34% recall figures and Entelligence’s 37.0% on Cal.com - measured a product with a hard structural limit that is gone. That is not a knock on those benchmarks. It is a knock on citing them in 2026 without a date.

2. Ownership changes the conflict map. Harness now owns DeepSource and the Augment assets. If you are building a provenance table, the Greptile re-run and the DeepSource CVE run are no longer two independent vendor perspectives. They are one company’s two datasets.

3. Non-determinism invalidates single runs. This is the quietest problem and the most serious. Without a deterministic static-analysis baseline, the same reviewer run twice on the same diff can produce different findings. Temperature, context assembly, retrieval order and model version all move the output. A single-run score is one sample from a distribution whose variance nobody publishes. Not one benchmark in our table of 19 reports multiple runs with variance. ReviewBench’s three rounds on the full set are the closest anyone comes, and the leaderboard still shows a point estimate.

This is why architecture-first pipelines matter more than percentages. DeepSource runs 5,000-plus deterministic rules across 30-plus languages before its AI agent sees the code. SonarQube’s core engine is rule-based. Rule output can be wrong, but it is wrong the same way every time, which means you can suppress a class of finding once instead of re-litigating it every Tuesday.

How to tell a credible AI code review benchmark from marketing

Direct answer: Apply six tests: independent publisher, raw data published, 100+ PR sample, precision and recall rather than a bare catch rate, multiple runs with reported variance, and a disclosed answer-key methodology. No benchmark in our set of 19 passes all six. GitHub ReviewBench passes five. Verified 2026-10-18.

Use this on any benchmark you are handed, including ours.

Test	What to look for	Why it matters	Benchmarks that pass
1. Independent publisher	Publisher sells no code reviewer, or discloses that it does	14 of 19 benchmarks are won by their own publisher	Martian, CRJudgeBench, SWE-PRBench, CR-Bench
2. Raw data published	Dataset, golden set and scripts downloadable	An unpublished answer key cannot be disputed, and the key drives the result	Martian, SWE-PRBench, GitHub ReviewBench, Augment, c-CRAB
3. 100+ PR sample	Stated denominator above 100 pull requests	Baz claimed #1 on 790 PRs against CodeRabbit’s ~300,000. Greptile, Augment and Entelligence all use 50	Martian, CRBugBench, SWE-PRBench, CR-Bench, GitHub ReviewBench, CodeAnt
4. Precision and recall, not a catch rate	Both error types counted, F1 or Fβ reported	A catch rate that ignores false positives is a recall measurement wearing a disguise. Greptile’s excludes them explicitly	Martian, Augment, Entelligence, SWE-PRBench, CR-Bench, GitHub ReviewBench
5. Multiple runs with variance	Repeated runs, standard deviation or range reported	Reviewers are non-deterministic. One run is one sample	None fully. ReviewBench runs 3 rounds but publishes a point estimate
6. Disclosed answer-key method	How golden comments were created, by whom, with agreement stats	This alone explains the Greptile 82-versus-45 gap	GitHub ReviewBench (96.6% agreement), SWE-PRBench (κ=0.75), Augment, c-CRAB

Scorecard. GitHub ReviewBench passes five of six, failing only on published variance. Martian passes four. SWE-PRBench passes five but measures bare models. Every pure vendor benchmark fails tests 1, 2 and 5 at minimum, and Greptile’s fails test 4 by its own stated rule.

If a benchmark fails test 4, stop reading it. A number that counts only what a tool found, and never what it got wrong, cannot help you choose. That is the whole reason we are building the ACR Noise Index.

How we verified this

All claims verified 2026-10-18 unless another date is stated inline.

Primary sources read directly.

• c-CRAB, arXiv 2603.23448 - abstract and v3 full text. Confirmed the 184-PR / 234-comment dataset sentence verbatim, the 41.5% union pass rate verbatim, the per-tool rates, and the three author affiliations including SonarSource. Searched the text for “false positive” and found two conceptual mentions, neither a measured rate for any reviewed tool.
• SWE-PRBench, arXiv 2603.26130 - abstract, the 350-PR dataset, κ=0.75 judge validation, the 15-31% detection range on the diff-only configuration, the ten-stage filter, the RVS ≥ 0.35 threshold, and the per-model FPR leaderboard from the evaluation harness. Dataset at huggingface.co/datasets/foundry-ai/swe-prbench, harness at github.com/FoundryHQ-AI/swe-prbench.
• CR-Bench, arXiv 2603.11078 - Nutanix. Confirmed 584 instances / 174 verified, the precision, recall, F1, usefulness-rate and SNR metric set, and the GPT-5.2 and GPT-5-mini figures.
• CRBugBench, arXiv 2610.13668 - task counts, repository counts, recall@5 figures, cost per task, the 82.1% human-plus-agent figure. Confirmed no precision metric is reported.
• CRJudgeBench, arXiv 2609.37216v1 - dataset composition and the Sentinel result.
• GitHub ReviewBench announcement and the repository - confirmed 219 PRs / 187 repos / 19 languages, the 103.9M-PR sampling base, the language and size distributions, the grounded-versus-augmented metric definitions, Claude Sonnet 5 as judge, the 96.6% inter-rater agreement, the MIT licence, and the scripts/try-agent.sh and review-bench.ai/submit workflow.
• Greptile Benchmarks - the scoring rule that false positives, style suggestions and unrelated comments did not affect the catch rate.
• Augment’s 7-tool benchmark - the precision, recall and F table, and the answer-key rebuild quote. Published 11 December 2025, updated 18 June 2026. Data at github.com/ai-code-review-evaluations.
• DeepSource, “Every AI code review vendor benchmarks itself, and wins” - confirmed the title verbatim, the thesis sentence, the seven-row vendor table, the absence of DeepSource’s own 84.51%, and DeepSource’s stated defence about Security Benchmarks versus AI Code Review Benchmarks.
• Martian Code Review Bench - the two-track structure and the precision definition.
• CodeRabbit’s newsroom post, 3 March 2026 - the 51.2% / 49.2% / 53.5% figures and the ~300,000-PR denominator.
• Entelligence Code Review Benchmark 2026 - aggregate and per-repository tables.

What we could not verify.

• Sphinx (arXiv 2601.04252). We confirmed the title, “Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review,” and the author list (Daoan Zhang, Shuo Zhang, Zijian Jin, Jiebo Luo, Shengyu Fu, Elsie Nallipogu). We could not extract institutional affiliations, dataset size or a precision figure from the PDF. It is in our table marked unverified, and we will not quote a number from it.
• c-CRAB’s 67-repository count. The 184-PR and 234-comment figures are quoted verbatim from the paper. The repository count of 67 comes from Table 4 / section 4.4.2 and we could not re-extract it from the v3 HTML in this pass. An earlier research round produced 56, which we believe is wrong. Treat 67 as our best figure with that caveat attached.
• Panto’s 11-versus-2 false-positives-per-PR pair. We confirmed Panto’s broader methodology (17 open-source PRs, comments bucketed with a false-positive category, OpenAI o3-mini as classifier with severity labels stripped) but could not open a primary Panto page stating that exact pair. Reported, not confirmed.
• CodeAnt’s “300K-PR benchmark.” We could not establish the metric, the answer-key method, or whether false positives were counted. We list the claim and label the method unclear.
• Baz’s leaderboard claim. Confirmed as a vendor announcement. We did not see the underlying 790-PR data.
• The current state of any leaderboard. Martian’s online track and ReviewBench both update continuously. Every leaderboard figure on this page is a dated snapshot and should be cited with its date or not at all.
• Three adjacent papers we found but did not score: AI-to-AI Code Reviews of GitHub Pull Requests (arXiv 2608.21311), BulkPR-Bench (arXiv 2608.02685, queue-level governance of interacting pull requests), and a five-model comparison on 150 samples (arXiv 2606.15689, 100 mutation-injected bugs plus 50 real bug-fix PRs across 8 repositories). Also noted but not verified: an Atlassian RovoDev online evaluation (ICSE-SEIP 2026) and HalluJudge (FSE 2026) on reference-free hallucination detection in review automation.
• Any independent false positive rate for a shipping commercial reviewer. This is the central finding of the page. If you have one, send it and we will publish it with attribution and correct this article.

Disclosure. Gitar is a paying sponsor of this site and Ito is an unpaid editorial baseline partner. Neither appears in any benchmark above for the same reason: neither has a published third-party benchmark. Sponsorship buys placement units, never a score. Our editorial assessment is at gitar-review.

Recommendations

Stated criteria: we rank benchmarks on auditability and metric completeness, not on how flattering their numbers are. A benchmark you can re-run beats a benchmark with a bigger percentage.

For methodology quality, read GitHub ReviewBench first. It is the only benchmark in the category whose dataset, rubric, judge and runner you can inspect yourself, and the only one publishing an inter-rater agreement figure (96.6%). Read it knowing GitHub wins it at 40.1% grounded F1 and that Martian ranks Copilot fourth. Both facts are load-bearing.

For behavioural signal, read Martian’s online track with a date attached. Fresh pull requests cannot be in training data and developer behaviour is hard to game, which makes “did the developer change the code” the best available precision proxy. Treat it as a snapshot: CodeRabbit at 51.2% F1 is a 3 March 2026 reading, not a standing fact.

For a precision-and-recall table across named products, use Augment’s re-run. It is a vendor benchmark and Augment wins it at 59% F-score. It also publishes precision, recall and F together for seven products and releases its corrected golden set, which no other vendor does. Those two facts coexist.

For the floor on bare models, use SWE-PRBench. It is the only benchmark publishing a false positive rate, and it measures models rather than products. Claude Sonnet 4.6’s 0.227 FPR is the best in that table. Do not translate it into a claim about any product.

Do not use CRBugBench, Greptile’s benchmark or Macroscope’s to compare noise. None of them count false positives in a way that lets you. CRBugBench reports recall@5 and no precision. Greptile’s rule explicitly excludes false positives. Macroscope reports detection rate only.

Do not buy on any published benchmark. Per-repository variance in Entelligence’s data exceeded per-tool variance, which means the repository you test on moves the score more than the tool you pick. Run 30 of your own merged pull requests, write the golden set before you run anything, classify every comment, and report the denominators.

And we are doing something about the gap rather than complaining about it. The ACR Noise Index will publish precision, recall and false positives per pull request on a fixed golden set we construct and release, with every per-finding classification public so anyone can dispute our calls. It will be the first noise measurement in this category from a party that does not sell a code reviewer. The method is published before the results, on purpose, so we cannot tune it to a conclusion: read the protocol. Target for results: Q4 2026.

Related reading: AI code reviewers with low false positive rates, the tools that explain their findings, our own model-level benchmark, the full ranked tool list, and the state of AI code review in 2026. New to the category? Start at what is AI code review.