AI Code Review False Positive Measurement: Our Method

Direct answer

This page publishes a method, not results. No AI code review false positive measurement exists from a party without a commercial stake, so we are building one: the ACR Noise Index. This article pre-registers the full protocol - a four-class comment taxonomy with explicit boundary rules, repository and pull request selection strata including deliberately clean pull requests, blind classification with inter-rater agreement, three runs per pull request with reported variance, and a public raw comment log. There are no tool scores anywhere on this page. First results target Q4 2026.

How do you measure AI code review false positives?

Direct answer: You classify every comment the tool emits, not just the ones that matched a known bug. Four exclusive classes: true positive, nitpick, false positive, off-topic. One label per comment, assigned by a classifier who does not know which tool wrote it. Precision is true positives over all comments. Most benchmarks skip this because golden-set scoring only measures recall. Protocol pre-registered 2026-10-18.

Here is the structural problem with almost every benchmark in this category.

A golden-set benchmark builds a list of known issues in a pull request, runs a tool, and counts how many it found. That number is recall. It is useful. It is also completely blind to everything the tool said that was not on the list, because an unmatched comment is simply not scored.

To measure noise you have to go the other direction. Start from the tool’s output, not from the answer key, and account for every single comment it produced. That is a different and more expensive exercise, which is why nobody independent has done it.

We catalogued 19 published AI code review benchmarks and traced each one to its publisher. Fourteen were published by companies selling a reviewer, and all fourteen are won by their publisher. The full provenance table is in every AI code review benchmark, compared.

Why this measurement is missing

Direct answer: False positives are the metric buyers care most about and no disinterested party has published one for any shipping AI code reviewer as of 2026-10-18. Two independent papers publish false positive data for bare models in a fixed harness - SWE-PRBench and CR-Bench - but neither evaluates a product you can buy. Every product-level noise figure is vendor-published.

Ask any engineering manager what killed their last code review tool rollout. They will not say it missed a bug. They will say the team stopped reading the comments.

Noise is the failure mode that actually loses deployments, and it is the one number nobody will publish.

The reasons are structural, not conspiratorial:

1. Recall is cheap and precision is expensive. Golden-set recall scales: build the key once, run tools against it. Precision requires a human to adjudicate every comment every tool produced, which is linear in output volume and gets worse for the noisiest tool.
2. Precision is the number that makes a vendor look bad. Greptile’s benchmark states the rule explicitly: false positives, style suggestions and unrelated comments did not affect the catch rate. That is a design choice, disclosed, and it is also exactly the choice that removes the risk.
3. Classification is judgement, and judgement is attackable. A vendor whose comment you labelled a nitpick will dispute it. Publishing a number you have to defend comment by comment is a commitment most publishers avoid.

Two papers do publish false positive data and deserve credit for it. SWE-PRBench (arXiv 2603.26130) reports an FPR per model across 350 pull requests. CR-Bench (arXiv 2603.11078, Nutanix) reports precision and a signal-to-noise ratio across 584 instances, and states outright that false positives are costly in production workflows.

Both measure base models in a fixed prompt harness, not products. You cannot buy the thing they measured. The products buyers actually evaluate - CodeRabbit, Greptile, Qodo, Cursor Bugbot, Copilot code review - wrap models in repository indexing, retrieval, severity filters and judge agents that change the output entirely.

That is the specific hole. A product-level noise measurement from a party with nothing to sell. That is what the ACR Noise Index is.

What the ACR Noise Index measures, precisely

Direct answer: For each tool, on each pull request, we measure: precision, recall, F1, false positives per pull request, nitpick rate, and run-to-run stability. Every comment is classified into exactly one of four classes. Scope is inline and summary comments on the pull request diff, at default configuration, on public repositories with pinned commit SHAs. Protocol pre-registered 2026-10-18.

The unit of analysis is a comment, not a pull request. Every published comment a tool emits gets exactly one label. Nothing is discarded as unscoreable. If we cannot classify a comment, it goes in a fifth residual bucket and the residual rate is published, because a hidden “unclear” pile is where a methodology goes to die.

What is in scope. Inline diff comments and the summary or overview comment, on the pull request as the tool posted it. Severity labels and confidence scores the tool attached are recorded and then stripped before classification.

What is out of scope. Chat follow-ups, IDE-surface findings, CLI-only output, and anything a human prompted the tool to produce. We are measuring the unprompted review, because that is what a team experiences on a Tuesday morning.

The comment classification taxonomy

Direct answer: Four exclusive classes with explicit decision rules: true positive (a real defect a maintainer would fix), nitpick (correct but trivial), false positive (asserts something untrue about the code), off-topic (outside the diff’s concern or duplicated). Boundary cases are defined in advance because boundaries are where benchmarks get gamed. Pre-registered 2026-10-18.

Class	Definition	Decision rule	Counts toward
True positive	A real defect a maintainer would fix	Would a maintainer of this repository accept a patch that changes the code as described? If yes, true positive, whether or not it is in our golden set	Precision numerator, recall numerator if golden-set matched
Nitpick	Factually correct but trivial - style, naming, formatting, preference	The claim is true, and acting on it changes no behaviour, no risk and no measurable maintainability outcome	Nitpick rate. Neither precision numerator nor false positive count
False positive	Asserts something untrue about the code	The comment makes a checkable factual claim about this code and the claim is wrong	False positive rate. Counts against precision
Off-topic	Outside the diff’s concern, or duplicated	Correct, possibly useful, but about code the diff did not change, or a restatement of another comment in the same review	Off-topic rate. Counts against precision
Residual	Cannot be classified	Published as a rate, with examples, so the pile stays visible	Reported separately

Every class is exclusive and every comment gets exactly one label. No comment is both a nitpick and a false positive. The decision rules above are applied in order: is the factual claim wrong (false positive), then is it about untouched code or duplicated (off-topic), then is it behaviourally trivial (nitpick), then it is a true positive.

The boundary cases, defined in advance

This is the part that decides the numbers, so it goes on the record before we run anything.

A correct comment about code outside the diff is off-topic, not a true positive. A reviewer who flags a genuine bug three files away from the change has found something real, and in a pull request review it is still noise, because the author cannot act on it in this change. We record off-topic separately rather than folding it into false positives, so a reader who disagrees can recompute with off-topic counted either way.

A comment that is correct but speculative is a false positive if the speculation is checkable and wrong. “This could be null” is a false positive when the type system, a prior guard, or a caller contract makes null impossible. It is a true positive when null is genuinely reachable. “Consider whether this could be null” with no claim either way is a nitpick.

A comment that is correct about a risk the repository has deliberately accepted is a nitpick, not a true positive. If a project pins to an older dependency on purpose, or disables a lint rule project-wide, a comment restating the tradeoff is not a defect report.

Duplicates inside one review are off-topic after the first instance. Three comments on the same missing error check is one finding and two pieces of noise. Counting all three as true positives rewards verbosity, which is precisely the behaviour we are trying to measure.

Severity disagreement is not a false positive. A tool that calls a cosmetic issue critical has made a calibration error, not a factual one. We record the tool’s severity label separately and report severity calibration as its own measure.

A hallucinated reference is always a false positive. An API that does not exist in the pinned version, a function signature that is wrong, a CVE that does not apply, a line number that points at unrelated code. These are the cleanest false positives available and we expect them to be the least disputed.

A comment that is right for the wrong reason is a true positive with a flag. If a tool identifies a real defect but its stated explanation is incorrect, the finding counts and the explanation quality is recorded separately. This connects to the other half of trust, which we cover in AI code review tools that explain their findings.

Repository and pull request selection criteria

Direct answer: Stratified selection across language, pull request size, and defect status, on public repositories with permissive licences and real review cultures. Roughly a third of pull requests will be clean refactors with no known defect. All commits pinned by SHA. Repository and pull request lists publish with the results so anyone can re-run the identical set. Pre-registered 2026-10-18.

A sample that is not stratified is a sample that measures whatever it happened to contain. Entelligence’s 2026 benchmark is the clearest warning here: in its data, per-repository swings were larger than per-tool swings, with Greptile moving from 52.6% F1 on Keycloak to 24.5% on Sentry. Repository choice moved the score more than tool choice did. Any benchmark on five repositories is partly measuring those five repositories.

Stratum	Target design	Why
Language mix	TypeScript, Python, Go, Java, Rust, C# represented, no single language above 35% of the sample	GitHub ReviewBench found TypeScript in 68 of 219 pull requests and Python in 41. A sample dominated by one language measures that language’s review conventions
Pull request size	Four bands: 50 lines or fewer, 51-200, 201-500, above 500. No band below 15% of the sample	Size is the strongest known driver of reviewer behaviour. Copilot had hard 300-file and 20,000-line caps until 27 August 2026, and CodeRabbit caps at 150 files on Essentials and 300 on Team
Defect status	Roughly two thirds with at least one known real defect, roughly one third clean	Clean pull requests are the only way to tell restraint from inactivity. See the next section
Defect provenance	Defects sourced from follow-up fix commits, linked issues, and human review comments on the same pull request. No synthetic or injected bugs	Mutation-injected bugs measure pattern matching. Real post-merge fixes measure the thing buyers care about
Repository concentration	No single repository above 10% of the sample	ReviewBench’s most-represented repository contributes 10 of 219 pull requests, 4.6%. That is the bar
Repository health	Active in the last 90 days, permissive licence, at least two substantive human review comments per selected pull request	A repository with no review culture has no ground truth to extract
AI contamination filter	Exclude pull requests where review comments are predominantly bot-authored, and exclude automated dependency-bump pull requests from the defect strata	SWE-PRBench excludes pull requests where more than 30% of review comments come from known AI bots. We adopt the same filter
Recency	Pull requests merged within the preceding 12 months, pinned by commit SHA	Older pull requests are more likely to sit in training data. Pinned SHAs make the run reproducible

Target size is the low hundreds of pull requests. We will state the exact number when the sample is frozen, and we will state it before we publish any score. We are not going to promise a figure here that the budget then forces us to cut.

Why including clean pull requests is non-negotiable

Direct answer: Silence is not precision. A tool that emits nothing on a clean refactor scores identically to a tool that emits nothing on a buggy pull request unless you test both. Graphite’s agent was reported to return zero comments on a review, and scored 0.0% F1 on Cal.com in Entelligence’s 2026 benchmark. A green check with no findings creates false reassurance. Verified 2026-10-18.

This is the methodological point I would most like other benchmarks to adopt, so let me make it carefully.

Golden-set benchmarks cannot distinguish a quiet tool from a broken one. Imagine two reviewers. The first reads the diff carefully, finds nothing wrong, and says nothing. The second is misconfigured and posts nothing on any pull request ever. On a golden-set-only benchmark run against clean pull requests, both score zero findings and zero false positives. They look identical and they are opposites.

The only way to tell them apart is to score them on both kinds of pull request and compare. A tool that is quiet on clean pull requests and vocal on buggy ones is calibrated. A tool that is quiet on both is off.

The reason this is not hypothetical: Graphite’s agent was reported to return zero comments on a review, and in Entelligence’s 2026 benchmark Graphite scored 0.0% on Cal.com and had the lowest aggregate F1 in that table at 13.4%. Graphite was acquired by Cursor on 19 December 2025 and the Diamond name was retired in favour of Graphite Agent, so any figure from that period describes a product under a different owner. The point is not about Graphite specifically. The point is that a passing check with no findings looks exactly like safety and is not.

So: clean pull requests are scored on silence. A clean refactor, a rename, a dependency bump with no behavioural change, a test-only addition. For these, the only true positive available is restraint, and every comment that is not a defect report is noise. We will report a separate clean-PR noise rate: comments per clean pull request, broken down by class.

Blind classification and inter-rater agreement

Direct answer: Every comment from every tool goes into a single randomised queue with the tool name, severity label, formatting signature and ordering stripped, then gets classified in that order. Two independent classifiers label an overlapping subset and we publish raw agreement plus Cohen’s kappa. The disagreement set publishes in full. Pre-registered 2026-10-18.

Unblinded classification is the largest bias available in this measurement and it costs nothing to remove.

If a classifier knows a comment came from a tool they expect to be noisy, borderline comments drift toward false positive. If they know it came from a tool they rate highly, the same comments drift toward true positive. Nobody has to be acting in bad faith for this to move a number by several points, and several points is the entire spread between tools in most published tables.

The blinding procedure:

1. Collect every comment from every tool across every run into one pool.
2. Strip the tool name, any vendor branding in the comment body, the severity or confidence label, and distinctive formatting (emoji prefixes, collapsible sections, signature footers). Record all of it in a separate keyed file.
3. Randomise comment order across tools and across pull requests, so comment sequence does not leak tool identity.
4. Classify in that order, with the diff and the repository available, and the four decision rules applied in sequence.
5. Re-join labels to tool identity only after the entire queue is labelled.

Severity labels get stripped specifically. They leak vendor identity, and more importantly a severity label is a vendor’s own confidence claim, not evidence. Letting a tool’s self-assessed “critical” influence whether a comment is scored a true positive would make the tool a participant in its own grading. Panto’s methodology stripped severity labels for the same reason, and we think that was the right call.

Inter-rater agreement. A second classifier independently labels an overlapping subset of the queue. We publish raw percentage agreement, Cohen’s kappa, and the full disagreement set with both labels and the diff, so readers can see exactly which judgement calls were close.

The bar to beat is published. GitHub ReviewBench had senior engineers who did not build the benchmark re-label every golden finding from scratch, reaching 96.6% agreement on true-and-false-positive judgments - the best agreement figure in the category and the only one most benchmarks do not report at all. SWE-PRBench validated its LLM-as-judge framework at κ=0.75. Those are the numbers our agreement will be compared against, and we are saying so before we have ours.

Multiple runs per pull request, with variance reported

Direct answer: Three runs per tool per pull request, same pinned commit, same seven-day window, default settings. We report the mean and the full range for every metric, plus the share of findings that appeared in all three runs. Not one of the 19 benchmarks we catalogued reports multiple runs with variance. Pre-registered 2026-10-18.

Without a deterministic static-analysis baseline, the same review run twice can produce different findings. Temperature, context assembly order, retrieval results, judge-agent sampling and silent model version rollouts all move the output. A single-run score is one draw from a distribution whose variance nobody has ever published.

This matters more than it sounds. If run-to-run variance on precision is five points, then a published three-point gap between two tools is noise, and every comparison table built on single runs is reporting coin flips as rankings.

So we run three times and publish the spread. Mean, minimum, maximum, and a stability rate: the share of findings present in all three runs. A tool with 90% stability and a tool with 50% stability can have the same mean precision and are not the same product to live with.

We will also record and publish the model version and tool version observed at run time, because a benchmark that cannot say which build it measured is not reproducible. This is the failure that makes most older figures unusable: Copilot’s 300-file and 20,000-line pull request caps were removed on 27 August 2026, so any Copilot score from before September 2026 describes a product that no longer exists.

All tools run at default settings. This understates what a tuned deployment achieves, and we will say so every time we publish a number. Defaults are the right choice for a comparison because a tuned tool against an untuned one is not a comparison, and because defaults are what a team experiences in week one. For what tuning actually buys you, see reducing CodeRabbit noise.

The metrics we will report, and why

Direct answer: Precision, recall, F1, false positives per pull request, nitpick rate, off-topic rate, clean-PR noise rate, stability rate, and severity calibration. Not a catch rate. A catch rate counts only what a tool found and never what it got wrong, which makes it recall in disguise. Pre-registered 2026-10-18.

Why not a catch rate. Greptile’s benchmark states its rule plainly: scoring considered only detection of the original bug, and false positives, style suggestions and unrelated comments did not affect the catch rate. Disclosed, and still the wrong metric for a buying decision, because a tool can post forty wrong comments and one right one and score well on it.

Why precision and recall separately, never collapsed. The two errors have opposite costs. False positives spend reviewer attention. False negatives ship bugs. A single blended number hides which one a tool trades away, and teams differ: a team drowning in review load should buy precision, a team shipping payment code should buy recall. We will report F1 and also publish Fβ at β values of 0.5 and 2 so a reader can weight it themselves. GitHub ReviewBench’s adjustable β is the right idea and we are copying it.

Why false positives per pull request. It is the metric engineers actually feel and almost nobody publishes. “0.42 precision” is abstract. “Six wrong comments on every pull request” is a staffing decision. Panto’s reported 11-versus-2 false-positives-per-PR comparison is the only product-level figure of this shape in circulation, and we could not confirm it at a primary source.

Why nitpick rate is separate from false positive rate. They are different problems with different fixes. A nitpick problem is solved with a severity threshold or a path filter. A false positive problem is a reasoning failure and configuration will not fix it. Qodo exposes a numeric inline_comments_severity_threshold, CodeRabbit has a chill profile and path_filters, Bugbot reads rules from .cursor/BUGBOT.md. Those levers work on nitpicks. They do not work on a model asserting that parameterised SQL is injectable.

The results table publishes empty. Here is the shape it will take. There are no numbers in it because we have not run the test.

Tool	Precision	Recall	F1	FP per PR	Nitpick rate	Off-topic rate	Clean-PR noise rate	Stability (3 runs)
Results publish Q4 2026	-	-	-	-	-	-	-	-

If you see a Noise Index figure attributed to a specific tool before Q4 2026, it did not come from us.

We will publish the raw comment log

Direct answer: When results publish we release the repository and pull request list with pinned commit SHAs, every comment from every tool across all three runs with its assigned class, the inter-rater disagreement set with both labels, and the observed tool and model versions. Publishing the raw log is what makes us a primary source rather than another aggregator. Pre-registered 2026-10-18.

This is the part that matters most and it is the part almost nobody does.

Of the 19 benchmarks we catalogued, five publish raw data: Martian, SWE-PRBench, GitHub ReviewBench, Augment’s re-run, and c-CRAB. The other 14 publish a percentage. A percentage you cannot audit is a claim, not a measurement. One disclosure, because we hold other publishers to it: c-CRAB is not conflict-free. Five of its six authors are academic (National University of Singapore, Zhejiang University), but one lists SonarSource as their affiliation (arXiv 2603.23448v3, verified 2026-10-18), and the paper carries a disclaimer that its conclusions do not represent SonarSource policy. Sonar sells SonarQube and acquired Gitar, which is a paid sponsor of this site. SonarQube is not among the four tools c-CRAB evaluates, which blunts the conflict considerably - but we are not going to call out vendor-funded benchmarks elsewhere and stay quiet about this one.

What we will release:

• The repository and pull request list, with pinned commit SHAs, so the identical set can be re-run.
• Every comment emitted by every tool on every run, verbatim, with its assigned class and the rule that decided it.
• The stripped metadata file - original severity labels, tool identity, formatting - so you can check the blinding.
• The inter-rater disagreement set in full, with both classifiers’ labels.
• Per-run findings, so you can recompute variance yourself.
• Observed tool version and model version per run.

Expect to disagree with some of our calls. That is the design. If you think a comment we labelled a nitpick was a real defect, you will be able to point at the exact comment, and we will publish the correction with attribution. An aggregator restates other people’s numbers. A primary source publishes the evidence that lets you prove it wrong.

Pre-registration: the method before the results

Direct answer: This article is a pre-registration. Every element of the protocol - taxonomy, boundary rules, selection strata, blinding, run count, metrics - is published on 2026-10-18, before a single tool has been run. This page stays up unchanged as the record. If the published results deviate from this protocol, we will state the deviation and the reason. Pre-registered 2026-10-18.

Pre-registration is standard practice in clinical trials and absent from software benchmarking, and the reason it exists applies exactly here.

If you choose the method after seeing the data, you will choose the method that produces the conclusion you wanted. Not deliberately. The taxonomy boundary that happens to make your preferred tool look better is also the boundary that feels most defensible once you have seen which comments it moves. The only protection is to commit before you look.

That is the entire reason this article contains no numbers. We could have run a quick pass on ten pull requests and published something illustrative. A plausible-looking illustrative figure on the page whose purpose is that the numbers are real would have destroyed the asset. So the page has a method, an empty table, and a date.

Specific commitments, on the record:

• The four-class taxonomy and the boundary rules above are fixed as of 2026-10-18.
• The selection strata are fixed. The exact sample size will be stated when the sample is frozen, before any scoring.
• Three runs per tool per pull request. If budget forces fewer, we will publish that fact rather than quietly publish a single-run number.
• All tools at default settings. Tuned runs, if we do them, publish as a clearly separate measurement.
• No tool gets a preview of its results. No vendor reviews our findings before publication.
• Deviations from this protocol get stated explicitly in the results article, with the reason.

Known limitations, honestly

Direct answer: Four we can name now: our sample will be in the low hundreds of pull requests rather than CodeRabbit’s roughly 300,000 on Martian’s online track; classification involves human judgement and we will get some calls wrong; our sample is public repositories, which carries a language and review-culture bias; and default settings understate what a tuned deployment achieves. Pre-registered 2026-10-18.

1. Sample size. CodeRabbit’s 51.2% F1 on Martian’s online track came from nearly 300,000 pull requests over January and February 2026. Ours will be three orders of magnitude smaller, because every comment in our sample is hand-classified and that does not scale. Our confidence intervals will be wider and we will publish them. A small, fully-audited sample and a large, automatically-scored one answer different questions; ours answers “what is in the output,” theirs answers “what do developers act on.”

2. Classification is judgement. The boundary between a nitpick and a true positive is a call, and reasonable engineers will disagree on perhaps one comment in ten. Publishing the disagreement set and the kappa does not eliminate this. It makes it visible and arguable, which is the best available outcome.

3. Open-source bias. Public repositories with strong review cultures skew toward certain languages, certain project sizes, and reviewers who write detailed comments. Private monorepos with 450,000 files behave differently. We cannot fix this without private code we cannot publish, so we will state it rather than pretend the sample generalises.

4. Default settings understate tuned performance. Every result will carry this caveat. A team that sets a severity threshold, adds path filters for generated code and lockfiles, and lets a learning system accumulate corrections will see a lower noise rate than we report. Most teams reporting a noise problem have never pulled those levers.

5. We may simply get it wrong. The protocol is published so that if we do, you can prove it. Re-run the set, publish your numbers, and tell us where we diverged.

Reproduce it, or send us a repository

Direct answer: The protocol is public so anyone can re-run it. Vendors can submit repositories and merged pull requests with their defect ground truth, and submissions are recorded publicly as vendor-submitted so readers can recompute without them. A submission buys nothing: no tuned configurations, no curated subsets, no pre-publication review. Pre-registered 2026-10-18.

For readers and researchers. Everything you need to run this on your own code is specified above, and the protocol doubles as an internal procurement test - arguably its highest-value use, since per-repository variance in every multi-repository benchmark published exceeds per-tool variance. If you run it, two asks: publish your repository and pull request list, and publish your raw comment log. The numbers matter less than the evidence.

For vendors. Send repositories and merged pull requests you consider representative, with whatever defect ground truth you hold. We will evaluate them against the published selection criteria. Conditions:

• Vendor-submitted pull requests are labelled as vendor-submitted in the published dataset, so any reader can recompute the metrics with them excluded.
• No tuned configurations. Defaults, same as everyone.
• No curated subsets and no pre-publication review of findings.
• Disputes after publication get published, with attribution, alongside our response.

Disclosure. Gitar is a paying sponsor of this site and Ito is an unpaid editorial baseline partner. Neither gets a different protocol, a preview, or a submission channel anyone else does not have. Gitar has no published third-party benchmark, which we note wherever review quality comes up, exactly as we would for any other tool. Our editorial assessment is at gitar-review.

How we verified this

All claims verified 2026-10-18 unless another date is stated inline. This article publishes no measurement of any tool, so what follows is verification of the claims we make about other people’s methods.

Primary sources read directly.

• GitHub ReviewBench announcement and repository - confirmed 219 pull requests from 187 repositories across 19 languages, the 103.9M-PR sampling base, the language distribution (TypeScript 68, Python 41, C# 25, Go 19, JavaScript 15), repository concentration at 10 PRs maximum (4.6%), the grounded-versus-augmented metric definitions, the adjustable β, Claude Sonnet 5 as judge, the 96.6% inter-rater agreement on golden true-positive re-labelling, the MIT licence, and the three-rounds-on-219 submission process.
• SWE-PRBench, arXiv 2603.26130 - confirmed the 350-pull-request dataset, the κ=0.75 judge validation, the ten-stage hard filter, the exclusion of pull requests where more than 30% of review comments are bot-authored, the RVS ≥ 0.35 inclusion threshold, and that it reports an FPR per model.
• CR-Bench, arXiv 2603.11078 - Nutanix. Confirmed the 584-instance dataset with 174 verified, and the precision, recall, F1, usefulness-rate and signal-to-noise-ratio metric set. Confirmed the paper’s framing that false positives are costly.
• Greptile Benchmarks - confirmed the scoring rule that false positives, style suggestions and unrelated comments did not affect the catch rate.
• Entelligence Code Review Benchmark 2026 - confirmed the per-repository tables, the Greptile 52.6% Keycloak versus 24.5% Sentry spread, Graphite’s 0.0% on Cal.com and 13.4% aggregate F1.
• Martian Code Review Bench - the two-track structure and the precision definition. CodeRabbit’s 3 March 2026 newsroom post for the 51.2% F1 figure and the ~300,000-PR denominator.
• Vendor configuration documentation for CodeRabbit (.coderabbit.yaml, chill profile, path_filters), Qodo (inline_comments_severity_threshold) and Cursor Bugbot (.cursor/BUGBOT.md).
• Product change confirmations: Copilot’s 300-file / 20,000-line cap removal on 27 August 2026; CodeRabbit’s 150-file cap on Essentials and 300 on Team; Graphite’s acquisition by Cursor on 19 December 2025 and the retirement of the Diamond name.

What we could not verify.

• The zero-comment Graphite result. Reported by a third-party 2026 test with a stated clean-configuration setup. We did not reproduce it. Entelligence’s 0.0% Cal.com result is consistent with it. Reported, not confirmed, and the product has changed owner since.
• Panto’s 11-versus-2 false-positives-per-pull-request pair. We confirmed the broader methodology (17 open-source pull requests, comments bucketed with a false-positive category, OpenAI o3-mini as classifier with severity labels stripped) but could not open a primary Panto page stating that exact pair.
• Any product-level false positive rate from a disinterested party. This is the gap the protocol exists to fill. We searched and found none as of 2026-10-18. If one exists, send it and we will publish it with attribution.
• Our own sample size. Not yet frozen. We will state it before we publish any score, and we are deliberately not promising a number here that a budget could force us to cut.
• Our own inter-rater agreement. Unknown until the classification pass runs. We have published the bar we will be measured against (ReviewBench’s 96.6%, SWE-PRBench’s κ=0.75) before knowing our own figure, which is the point.

Recommendations

Stated criteria: what to do between now and Q4 2026, ranked by how much decision-making weight each option can carry.

First, do not wait for us. Run the protocol on 30 of your own merged pull requests. Per-repository variance exceeds per-tool variance in every multi-repository benchmark published, so your own code is the only sample that answers your question. The four-class taxonomy and the boundary rules above are the whole method; a day of engineering time produces a number you can defend in a budget meeting.

Second, when you read any published noise figure, check four things. Who published it and do they sell a reviewer. What the denominator is. Whether false positives were counted at all. How the answer key was built. Three of the 19 benchmarks we catalogued fail the fourth test by not disclosing their key at all. The full provenance table is in every AI code review benchmark, compared.

Third, judge noise by architecture until real data exists. A deterministic-first pipeline is the strongest available predictor of predictable output: DeepSource runs 5,000-plus static rules across 30-plus languages before its AI agent sees the code, and SonarQube’s core engine is rule-based. Rule findings can be wrong, but they are wrong the same way every time, so you suppress a class once instead of re-litigating it. Our current read on which tools this favours is in AI code reviewers with low false positive rates.

Fourth, pull the severity lever before you conclude a tool is noisy. Qodo’s inline_comments_severity_threshold, CodeRabbit’s chill profile with path_filters, Bugbot’s .cursor/BUGBOT.md. Most teams reporting a noise problem are running defaults, and our benchmark will run defaults too, which is exactly why our numbers will be a floor rather than a ceiling.

Fifth, treat any undated benchmark figure as unusable. Claude Code review went GA to Claude Team and Enterprise on 15 October 2026 with at least $10,000 in review credits (Anthropic’s own figure; press reports a $10,000-$50,000 range scaled by active Claude Code seats) for eligible Enterprise organisations, Copilot’s large-PR caps came off on 27 August 2026, and Harness acquired both DeepSource (9 September 2026) and Augment Code’s Cosmos, Auggie CLI and Code Context Engine (8 October 2026). The market changed faster than most published benchmarks did.

Hold us to this page. The protocol is dated, the table is empty, and the method cannot be revised after the fact without us saying so. That is the only reason our eventual number should be worth more to you than a vendor’s.

More context: the full benchmark provenance table, tools that explain their findings, the ranked tool list, the state of AI code review in 2026, and AI code review versus manual review.