How to Evaluate an AI Code Reviewer: A 14-Day Pilot Protocol

Direct answer

No published AI code review benchmark is trustworthy enough to buy from, and results swing hard by repository and language, so you have to measure on your own code. This is a runnable 14-day protocol: how to pick pilot repos, how many pull requests you need, how to run two tools in parallel without drowning the team, a comment rubric, and a decision rule you write before you start. Verified 2026-10-18.

How to evaluate an AI code reviewer

Direct answer: Pick two repositories, run two tools on the same pull requests for 14 days, classify every posted comment against a written rubric as true positive, nitpick, false positive or off-topic, and compare severity-weighted precision, review latency, reviewer-hours and developer sentiment against a decision rule you wrote before day one.

The reason you cannot skip this is specific, not philosophical.

In Entelligence’s own published benchmark, one tool scored perfect precision but only 33.3% recall on Grafana (a Go codebase) while scoring completely differently on Cal.com. Same tool, same benchmark, same methodology, two repositories, opposite conclusions. That is not a measurement error. That is the actual behaviour of LLM review: performance is a function of the repository’s language, size, convention density and PR shape, and none of those are properties of the tool.

So the question “which AI code reviewer is best” has no portable answer, and every page that gives you one is guessing. The question that does have an answer is “which of these two is better on our code”, and fourteen days is enough to answer it if you set it up properly.

A second reason, from our Q4 2026 market review: no party without a commercial stake has published a false positive rate for any shipping commercial AI code reviewer. As of 18 October 2026, every precision figure in circulation came out of a harness built by somebody selling a reviewer. c-CRAB (arXiv 2603.23448, National University of Singapore) measured recall against executable tests and found four agents collectively caught 41.5% of issues across 184 pull requests, but reported no false-positive rate for any tool. CRBugBench (arXiv 2610.13668, CMU and Microsoft Research) reports recall@5 and cost per task and no precision at all. The number you most need is the one nobody disinterested has measured. One disclosure, because we hold other publishers to it: c-CRAB is not conflict-free. Five of its six authors are academic (National University of Singapore, Zhejiang University), but one lists SonarSource as their affiliation (arXiv 2603.23448v3, verified 2026-10-18), and the paper carries a disclaimer that its conclusions do not represent SonarSource policy. Sonar sells SonarQube and acquired Gitar, which is a paid sponsor of this site. SonarQube is not among the four tools c-CRAB evaluates, which blunts the conflict considerably - but we are not going to call out vendor-funded benchmarks elsewhere and stay quiet about this one.

You are going to have to measure it. Here is how.

The 14-day protocol

Three phases: five days of prep before the clock starts, fourteen days of live reviewing, two days of analysis.

Day -5 to -1: prep (do this before you start any trial clock)

1. Write the decision rule first. Before you see a single comment. Four thresholds and a null option, timestamped somewhere you cannot quietly edit - a dated doc, a commit, a Slack message in a channel with history. A worked example:

We adopt a tool only if all four hold: severity-weighted precision >= 60%; median time from PR open to first substantive comment <10 minutes; reviewer-hours per PR down >= 15% against the two-week baseline; and >= 60% of pilot developers vote keep. If no tool clears all four, we adopt none and re-run in two quarters.

The null option is the part teams leave out, and leaving it out is exactly how a disappointing pilot turns into a purchase anyway. Write it down.

2. Pick two repositories, deliberately. Not your quietest repo and not your scariest one.

Criterion	Target	Why
Merge volume	>= 10 merged PRs per repo per fortnight	Below this you cannot reach a usable comment count
Language	One repo in your primary language, one in a secondary	Catches language-specific collapse, which is the failure mode the Grafana result demonstrates
Size	One under ~50k LOC, one over	Context-handling differences only show up on the larger one
Convention density	At least one repo with real house conventions	Tests whether the tool can learn them or just lectures you about generic best practice
PR shape	Normal distribution of your real PRs	Do not exclude the 1,000-line PRs. GitHub’s ReviewBench deliberately weighted toward larger changes because that is where review actually hurts
Sensitivity	Neither should contain secrets, PHI or regulated data	You are granting a third party read access. Start where the blast radius is small

3. Do the security review now, because there is no sandbox.

No major AI code review vendor offers a demo or sandbox repository (verified 2026-10-18). CodeRabbit, Greptile, Qodo, Sourcery, CodeAnt AI and Cursor Bugbot all require you to connect a real repository through a platform app install. There is no synthetic playground that shows you output quality before you grant access. That means repository access is the first thing your security reviewer asks about, and if you discover that on day 3 of a 14-day trial you have burned a fifth of your window.

Prepare before you install:

• Scope the app install to the two pilot repositories, not the organisation. Every major platform supports per-repo selection on install. Teams click “all repositories” by reflex.
• Pull the vendor’s trust centre documents: SOC 2 report (and check Type I versus Type II - cubic.dev is Type I only as of 2026-10-18, Panto’s audit was still underway, and Kodus’s SOC 2 badge had no documents behind it), DPA, subprocessor list, and a BAA if you need one.
• Establish whether your code is retained and whether it trains anything. Ask in writing. Ask about the vendor’s model subprocessor separately, because the vendor’s retention policy and the LLM provider’s are two different documents.
• If you are in a regulated environment, note that ISO/IEC 42001:2023 is the AI-specific standard covering model integrity and AI pipeline risk, and only 7 of the 26 vendors we checked hold it: Anthropic, GitLab, GitHub Copilot, Cursor, Augment Code, Aikido and Veracode. Aikido’s certificate scope explicitly names “Deep PR Review”, which is the only certificate scope in this market that names a review feature.
• “HIPAA certified” is not a real certification scheme. What you need is a signed BAA. Four vendors offer one: Anthropic, Cursor (Enterprise), Augment Code and CodeAnt AI.

4. Capture a baseline. Two weeks of data from before the trial, from the same repositories:

• Median and p95 time from PR open to first human review comment
• Median and p95 time from PR open to merge
• Reviewer-hours per PR (easiest honest proxy: number of distinct human reviewers who commented, times a self-reported average minutes-per-review collected once from each reviewer)
• Count of PRs merged with zero substantive review comments

Without a baseline, “it felt faster” is all you will have at the end, and “it felt faster” is what vendors sell.

5. Recruit a classification panel of two or three people, not the whole team. Classification has to be consistent to be useful, and consistency across twelve engineers is a fantasy. Two or three people, same rubric, calibrated on the same first ten comments before they split up.

Day 1 to 14: run it

Both tools on every pull request. Only one posts inline.

This is the mechanic that makes a two-tool trial survivable. If both tools comment inline on every PR, your developers see double the comment volume, review fatigue sets in inside a week, and your sentiment data is measuring the trial design rather than either tool. That is the failure mode that kills most parallel pilots.

So: keep both running on all PRs so the comparison stays paired - identical inputs, which is statistically worth a lot - but collect the second tool’s output through its API or dashboard and have the panel grade it off-PR. Alternate which tool is the inline one by PR number parity, so neither gets a systematically easier sample.

If a tool cannot run silently, fall back to alternating whole PRs between tools. That protects the team but costs you the pairing and roughly doubles the number of PRs you need.

Classify every posted comment within 48 hours. Not at the end. Memory of whether a finding was real decays fast, and a comment classified three days late gets classified as “probably fine”.

Day 15 to 16: analyse and decide

Compute the four metrics, compare against the rule you wrote on day -5, and write the decision down even if the answer is “neither”. Then turn the pilot config into your production config, because the tuning you did during the trial is the most valuable artifact it produced.

The comment classification rubric

Direct answer: Classify every comment into exactly four buckets: true positive (a real defect or a real convention violation you would fix), nitpick (correct but not worth a comment), false positive (factually wrong about the code), and off-topic (correct and irrelevant). Nitpicks are not false positives, and conflating them is the most common pilot error.

Four buckets, one judgement per comment, no “partially”.

Class	Definition	Test the classifier applies	Counts toward precision?
True positive (TP)	A real defect, security issue, or violation of a convention your team actually holds	”Would I fix this if a senior engineer wrote the same comment?”	Yes
Nitpick (NIT)	Factually correct but not worth a reviewer’s comment. Style preferences, micro-optimisations, suggestions on untouched code	”Correct, but would a human have bothered?”	No. Tracked separately
False positive (FP)	Wrong about the code. Misreads control flow, invents an API, flags a guarded null as unguarded, claims a test is missing when it exists	”Is the claim about the code factually false?”	No. This is the number that decides adoption
Off-topic (OT)	Correct and irrelevant. Comments on the diff’s context lines, repeats a lint rule already in CI, restates the PR description	”Is this duplicating something we already have?”	No

Severity-weight the true positives. A tool that finds twenty formatting issues and misses an auth bypass is worse than one that finds the auth bypass and nothing else, but unweighted precision scores them the same. Tag each TP P1 (would have caused an incident or a security issue), P2 (would have caused a bug or a rollback), P3 (correctness or maintainability, no production impact). Report precision overall and precision on P1 plus P2 only, which is the number that should drive the decision.

Calibrate the panel before you split up. Have all classifiers independently grade the same first ten comments, then compare. If agreement is below about 80%, your rubric is ambiguous and you need to tighten the definitions before continuing. GitHub’s own ReviewBench reported 96.6% agreement between its labels and independent human checkers, and they had a whole annotation protocol behind that. Two people in a Slack huddle will not hit 96.6%, but you should hit 80% before you trust any aggregate.

Keep a missed-defect log too. Precision is only half the picture. Every time a human reviewer or a production incident catches something the tool did not, log it. You will not get a clean recall number in 14 days, but a list of twelve real defects neither tool flagged tells you something that no precision figure will.

What to measure

Direct answer: Four metrics. Severity-weighted precision from your rubric. Review latency change, as p50 and p95 time to first substantive comment and to merge. Reviewer-hours per pull request against your pre-trial baseline. And developer sentiment as a single repeated question. Cost per review is a tiebreaker, not a criterion.

Metric	How to compute	Target to beat	Common mistake
Severity-weighted precision	TPs tagged P1 or P2, divided by all posted comments	Set it yourself. Vendor-published figures cluster 45-65% F1, so 60% precision on P1 plus P2 is ambitious but not absurd	Counting nitpicks as false positives, which understates precision and kills a usable tool
Review latency	p50 and p95 minutes, PR open to first substantive comment; and PR open to merge	Beat your baseline p50 by at least 25%	Reporting the median only. A 4-minute p50 with a 25-minute p95 feels slow to whoever hits the tail
Reviewer-hours per PR	Distinct human reviewers who commented, times self-reported average minutes per review	15% reduction	Measuring reviewer count instead of time, which can fall while time per review rises
Developer sentiment	One question, same wording, days 4, 9 and 14: “On a 1-5 scale, is this tool helping you ship?”	Mean >= 3.5 and rising	A single end-of-trial survey. The trend matters more than the level, because a falling score is the “cry wolf” effect starting
Cost per review (tiebreaker)	Total trial spend divided by PRs reviewed	Compare to the vendor’s published model	Treating cost as a primary criterion. The precision spread between tools is usually wider than the price spread

The sentiment trend is the most predictive number in the whole pilot. A tool that scores 4.1 on day 4 and 2.9 on day 14 is going to be uninstalled in month three regardless of its precision, because the team has started skimming. A tool that scores 3.2 on day 4 and 3.9 on day 14 is being successfully tuned. Measure three times, not once.

How many pull requests you actually need

Honest arithmetic, because this is where pilots quietly become theatre.

For a single-tool precision estimate, the standard error of a proportion is sqrt(p(1-p)/n). At an observed precision near 50% with 150 classified comments, SE is about 4 points, so your 95% interval is roughly plus or minus 8 points. That is good enough to tell 60% from 40%. It is not good enough to tell 60% from 55%. Target 150 or more classified comments, which is typically 30 to 50 pull requests at the 3 to 6 comments per PR most tools produce.

To compare two tools as independent samples, detecting a 15-point precision gap at 80% power and 5% significance needs about 166 classified comments per tool. Detecting a 10-point gap needs about 390 per tool. Almost no team reaches 390 comments per arm in a fortnight, which means an honest 14-day trial can tell you which tool is substantially better and cannot tell you which is marginally better.

This is exactly why you run both tools on the same pull requests. A paired comparison - same PR, same diff, same context, two sets of comments - lets you compare tools on identical inputs and removes the repository and PR-shape variance that dominates the unpaired estimate. It needs far fewer PRs for the same confidence, and it is free. Take the pairing.

If your two pilot repositories merge fewer than about 20 PRs in a fortnight combined, extend the window to 21 or 28 days rather than accepting an underpowered result and calling it data. Most vendors will extend a trial if you ask during it.

How to compare AI code review tools before buying

Direct answer: Shortlist from published benchmarks, decide from your own pilot. Use vendor benchmarks only to pick which two tools to trial, because every benchmark in this market except c-CRAB and Martian’s was published by somebody selling a reviewer. Then compare on your repositories using a paired 14-day trial.

The shortlisting step is where published numbers are genuinely useful, and the deciding step is where they are actively harmful. Keep those separate.

What published benchmarks are good for: ruling tools out. If a tool scores near zero on a credible dataset, that is real information. Graphite scored 0.0% F1 on Cal.com in Entelligence’s benchmark, which is worth knowing even though Entelligence sells a competing reviewer and won its own benchmark at 47.2% aggregate.

What they are not good for: ranking the top of the list. Here is the variance, in one place:

• Greptile scored Greptile at 82% on an internal 50-PR, 5-repo evaluation.
• Augment’s re-run scored Greptile at 45% precision, recall and F on the same repositories. Note carefully: that 45% is an F-score, not a catch rate, and Augment did not reuse Greptile’s answer key. Their words: “We expanded and corrected the golden comments by reviewing each PR manually.” So the 82-versus-45 gap is partly a different answer key and partly self-dealing. Both things are true.
• DeepSource scored itself at 84.51% F1 and a competitor at 36.19%. DeepSource also published an essay arguing that every vendor benchmarks itself and wins, listing six vendors and omitting its own 84.51%.
• Four separate vendors have declared themselves #1 on Martian’s single leaderboard: CodeRabbit at 51.2% F1 (3 Mar 2026), then Qodo claiming #1 at 64.3% (15 Mar 2026), then Baz claiming #1 on 790 PRs.
• And as of 8 October 2026, two of the market’s most-cited benchmark publishers, DeepSource and Augment Code, are both owned by Harness. The two datasets that used to check each other now share a parent.

The full provenance audit is in our benchmark comparison. The short version: when a vendor quotes you a precision figure, the correct next question is “whose repositories, whose answer key, and who paid for the run?”

One more thing to check before you shortlist, because it will invalidate older numbers: GitHub removed Copilot’s 300-file / 20,000-line PR cap on 27 August 2026. Any Copilot evaluation run before September 2026 measured a product that could not read large PRs at all.

AI code review tool vendor shortlist

Direct answer: Shortlist on four hard filters before you look at review quality: does it support your git platform and deployment model, does your compliance posture permit it, what is the repository or seat cap on the tier you can afford, and is there a 14-day trial without a credit card. Most tools fail one of these before quality matters.

Run the filters in that order. It cuts a 51-tool market to three or four candidates in an afternoon.

Filter 1: platform and deployment. GitLab Duo Code Review reached GA with self-hosted models in GitLab 18.4. Anthropic Code Review supports GitHub Enterprise Server; Copilot code review does not. Codacy’s AI Reviewer is cloud-and-GitHub only, while GHES requires Codacy Self-hosted, which means GHES customers structurally cannot get Codacy’s AI Reviewer. If you are on GHES or self-managed GitLab, this filter alone eliminates most of the market.

Filter 2: compliance. Anthropic Code Review is unavailable to organisations with Zero Data Retention, the HIPAA configuration, or customer-managed encryption keys, and is not covered by Anthropic’s BAA. The configuration that makes Claude Code PHI-eligible disables Code Review. If you need a signed BAA, your list is four vendors long.

Filter 3: the cap on the tier you can actually buy. This is the filter everyone skips and everyone regrets.

• CodeRabbit Essentials ($24/dev/mo annual) covers one repository. Team covers 5, Advanced 10, Enterprise 20. Hourly review limits are 5 / 8 / 10 / 12 by tier, degrading to 1 per hour under fair usage. Max files per review: 150 on Free and Essentials, 300 on Team and above. Full pricing breakdown.
• CodeRabbit’s free tier no longer covers private repos. Free is public repositories only, plus 1 summary-only private PR review per hour.
• Greptile Pro is $30/seat/mo with 50 credits, and credits are per author and do not pool. A “seat” is any developer who received a review. Apex depth costs 10 credits per review. Greptile pricing.
• Cursor Bugbot is no longer a standalone product. The $40 seat was removed effective at each customer’s first renewal after 8 June 2026; it is now usage-billed from the Premium plan ($30/mo) upward, at roughly $1.00-1.50 per review, with no free tier.
• DeepSource bills AI review separately from the seat: $8 per 10K processed LOC on Standard, $15 per 10K on Advanced, against a $100 annual AI credit per user. The ~$24 seat is not the real cost.
• Macroscope has no seat fee at all: $0.05 per KB reviewed with a $0.50 floor, published average review $0.95, $100 credit on new workspaces. Re-reviews bill each time.
• Copilot code review is not on every Copilot plan. On individual tiers it starts at Pro+ ($39/mo). GitHub also retired annual billing entirely on 1 June 2026, and private-repo reviews now consume Actions minutes on top of AI Credits.

Filter 4: trial mechanics. Covered in the next section.

For the ranked list with criteria attached once the filters are applied, see best AI code review tools.

Trial mechanics, verified 2026-10-18

Tool	Trial length	Card required	Notes
CodeRabbit	14 days	No	Free forever on public repos. Private repos no longer in the free tier
CodeAnt AI	14 days	No	Capped at 100 PR reviews. Free for open source. SOC 2 plus ISO 27001
Qodo	14 days	Not required to start	Unlimited reviews during trial. No permanent free tier. Credits pool across the team at $0.012 each, no rollover
Sourcery	14 days	Card only on conversion	Trial is org-wide, not per-seat. Pro $12/seat/mo, Team $24, billed annually
Greptile	14 days (Pro)	-	Starter is free for 1 active dev with 50 credits/mo. 50% off pre-Series A under $2M revenue
Cursor Bugbot	14 days	-	No free tier at all. Usage-billed after trial
cubic	7 days	No	Starter free at 20 PR reviews/mo per GitHub org. SOC 2 Type I only
Bito	14 days (Professional)	No	5,000 lines per seat, then $5 per 1,000

Note the alignment. Six vendors run exactly 14 days. That is why this protocol is 14 days: you can run two trials on overlapping clocks at zero cost, which is the entire economic argument for the paired design.

And note what is missing from that table. There is no “sandbox” column, because no major vendor offers one. Every entry above requires connecting a real repository.

What should I prepare before getting an AI code review demo?

Direct answer: Bring three artifacts: a list of 5 to 10 real pull requests from your repos with known outcomes, your four pre-written decision thresholds, and your security questionnaire. Then ask the vendor to review your PRs live rather than watching their prepared demo, and ask who funded every benchmark number they quote.

A vendor demo on a vendor-chosen repository tells you nothing. Every tool in this market looks excellent on a demo PR the vendor picked. Take control of the inputs.

Bring these:

1. Five to ten real pull requests with known outcomes. Ideal mix: two that caused a production incident, two that a human reviewer caught something important in, two that were clean, and two that are unusually large. You already know the right answers. That is the point.
2. Your four decision thresholds, written. Share them with the vendor. A good vendor will tell you whether they can hit them; a bad one will tell you the thresholds are the wrong thresholds.
3. Your security questionnaire, sent before the call so you are not burning demo time on it. Retention, training use, subprocessors, SOC 2 type, DPA, BAA, data residency.
4. Your repo and seat arithmetic. How many repos, how many PRs a month, how many developers who will receive reviews (not just those who author code - Greptile bills on received reviews).

Ask these, in these words:

• “Can you run these five PRs live, now, on this call?” If the answer is no, ask when.
• “Who funded the benchmark that number comes from, and which repositories was it run on?”
• “What is the repository cap and the file-per-review cap on the tier you are quoting me?”
• “What is the p95 time to first comment, not the median?”
• “What happens to my configuration if you are acquired?” This is not a hostile question in 2026. Five vendors in this category were acquired between December 2025 and October 2026, including DeepSource and Augment Code into Harness within 29 days of each other.
• “Can I scope the app install to two repositories rather than the organisation?”
• “If I turn off every category except security and correctness, how many comments per PR do you expect?” Volume control is the whole game in month three.

Do not ask for a reference customer in your industry. You will get one, it will be positive, and it will tell you nothing about your repositories.

Evaluate the config surface, not just the output

Direct answer: Whether a tool survives month three is decided by its configuration surface, not its day-one output quality. You need three things: severity thresholds you can raise, categories you can suppress, and a way to encode team conventions in-repo. Test all three during the trial, not after you buy.

This is the part buyers consistently skip, and it is the single best predictor of whether a tool is still installed in six months.

Here is why. Every tool’s default configuration is tuned for an average repository, which is not yours. Day-one output will contain a tail of comments that are technically correct and useless on your code: style preferences you deliberately rejected, suggestions on generated files, lint rules you already run in CI, performance notes on code that runs once at boot. If you cannot turn those off, the comment volume stays high, developers start skimming, and the “cry wolf” effect takes the tool down regardless of how good its P1 findings were.

So during the trial, deliberately try to configure each tool wrong, then configure it right, and see how much work each takes.

Tool	Config mechanism	What you can control	Notes
CodeRabbit	.coderabbit.yaml plus Learnings	Path filters, per-path instructions, tool toggles across 40-plus linters, review profile (chill versus assertive), auto-review triggers	Most powerful surface in the market and the steepest curve. Learnings is the differentiator: reply to a comment telling it your preference and it persists that across future reviews
Cursor Bugbot	BUGBOT.md	Freeform Markdown rules	Walks up the directory tree from each changed file to find the nearest BUGBOT.md, so per-package rules in a monorepo are trivial. No schema to learn
Anthropic Code Review	REVIEW.md plus per-repo depth	What to flag, at what severity, what to skip. Quick / Standard / Deep per repository	Read from the PR’s base branch or default branch, never the PR’s own diff, so a pull request cannot loosen the rules it is judged by. Best-designed config semantics in the category
DeepSource	.deepsource.toml	Analyzer enable/disable, per-analyzer options, exclude patterns, transformers	Clearest surface for turning whole analyzers off. Config guide
Qodo Merge	Configuration file plus confidence threshold	Which tools run, per-repo settings, and a tunable confidence floor for posting a finding	The confidence threshold is the most direct volume control available anywhere: raise it and low-certainty comments stop being posted
Greptile	Dashboard rules plus learning from PR comments	Custom rules, review depth (standard versus Apex)	Learning reads your engineers’ GitHub comments to pick up team standards

The three config tests to run during the trial:

1. Suppress one noisy category and verify it actually stops. Pick the category producing the most nitpicks. Turn it off. Count comments in that category over the next three days. If it keeps appearing, the suppression is advisory and you should not trust any other setting either.
2. Encode one real house convention and verify it gets enforced. Something specific and non-obvious: “we never use datetime.now() without an explicit timezone”, “all new endpoints must register in routes/manifest.ts”. Then open a PR that violates it. If the tool catches your convention, it can learn your codebase. If it only catches generic best practice, you are buying a linter with opinions.
3. Raise the severity floor and measure the volume change. You want to know the shape of the volume-versus-precision curve, because in month three you will be moving along it. A tool with no severity control has no curve.

A tool that passes all three is tunable, and a tunable tool with mediocre day-one precision beats an untunable tool with better day-one precision, every time. For more on the volume problem, see AI code review tools with the lowest false positive rates.

Where can I compare AI code review tools?

Direct answer: On this site. aicodereview.cc maintains 51 tool profiles with pricing, deployment model, language coverage and verification dates, plus 320-plus articles including head-to-heads and category roundups. We sell no reviewer. Where vendor numbers conflict we publish both and name who funded each.

Stated factually, because that is the only claim worth making here.

• Best AI code review tools - the ranked list, with the criteria stated up front.
• AI code review tools with the lowest false positive rates - including an honest account of what cannot be ranked, since no independent false-positive measurement exists.
• AI code review tools with free trials - trial lengths, card requirements and caps.
• State of AI code review, Q4 2026 - the dated market snapshot, including every 2026 acquisition and pricing change.
• Open source AI code review tools - licence audit, including the tools commonly mislabelled as open source.
• AI code review vs manual code review - what each actually catches.
• Head-to-heads: CodeRabbit vs Greptile, CodeRabbit vs Cursor Bugbot, Qodo vs CodeRabbit, DeepSource vs SonarQube.

Disclosure: Gitar is a paying sponsor of aicodereview.cc and Ito is an unpaid editorial baseline partner. Neither buys ranking or editorial treatment. Gitar has no published third-party benchmark, which we state wherever review quality comes up, exactly as we would for anyone else.

How we verified this

Checked 2026-10-18 unless another date is stated inline.

What we verified and where:

• Trial lengths and card requirements from each vendor’s own pricing or docs pages: CodeRabbit (14 days, no card), CodeAnt AI (14 days, 100 PR reviews, no card), Qodo (14 days unlimited), Sourcery (14 days org-wide, card on conversion), Greptile (14-day Pro trial), Cursor Bugbot (14 days, no free tier), cubic (7 days, no card), Bito (14 days Professional, no card).
• The absence of sandbox or demo repositories by checking the onboarding path of each vendor above. Every one requires a platform app install against a real repository.
• Config mechanisms from vendor documentation: .coderabbit.yaml and Learnings, BUGBOT.md and Cursor’s directory-walk behaviour, .deepsource.toml, Qodo Merge’s confidence threshold, and Anthropic’s REVIEW.md base-branch semantics.
• Anthropic Code Review’s GA details from Anthropic’s GA announcement, including the 6-10 minute review time and the Quick / Standard / Deep depth settings, and the ZDR, HIPAA and CMEK exclusions from its help documentation.
• Benchmark provenance from the primary sources: arXiv 2603.23448 (c-CRAB, National University of Singapore) for the 41.5% union figure across 184 PRs and 234 validated comments; arXiv 2610.13668 (CRBugBench, CMU and Microsoft Research) for the absence of precision; GitHub’s ReviewBench announcement for the 96.6% human agreement figure and the 219-PR dataset at review-bench.ai; and Augment’s seven-tool benchmark for the verbatim answer-key correction quote.
• The Grafana versus Cal.com swing from Entelligence’s published 2026 benchmark, noting that Entelligence sells a reviewer and placed itself first.

What we could not verify, stated plainly:

• Any independent false positive rate for any AI code reviewer. None exists that we can find. Everything in circulation traces back to a party selling a reviewer.
• Published review latency for any tool other than Anthropic. Nobody else publishes p50 or p95. This is why the protocol has you measure it.
• Whether the 80% inter-rater agreement floor we recommend is an established standard. It is our working threshold, calibrated against GitHub’s reported 96.6% with a full annotation protocol. Treat it as a practical heuristic, not a citation.
• The statistical power figures are standard two-proportion sample-size calculations at 80% power and 5% significance. They are arithmetic, not a measurement, and they assume independent samples, which is exactly why we recommend a paired design instead.
• Gemini Code Assist’s free tier. Status genuinely unclear: one source says it stopped serving requests on 18 June 2026, Google’s docs still describe 100-plus PR reviews per day for a GitHub app install. Excluded from the trial table rather than guessed at.
• Tabnine pricing. The pricing page redirects to a Tricentis contact form following the acquisition announced 30 July 2026, so no per-seat figure is quotable.

Recommendations

Criteria: a team of 5 to 50 developers choosing a reviewer in the next quarter, optimising for a tool still in use in six months rather than the best day-one demo.

Run the protocol, not a longer version of it. Fourteen days of live reviewing, five days of prep, two days of analysis. The temptation is to extend the trial until the data is conclusive. Resist it, because the limiting factor is not time, it is comment volume, and a 28-day trial on a low-volume repo produces the same underpowered result more slowly. If you need more comments, add a repository, not a week.

Write the decision rule first or do not bother. Every pilot I have seen that skipped this ended with a purchase, because by day 14 somebody has spent political capital on the trial. The four thresholds and the null option take twenty minutes and they are the whole reason the exercise is honest.

Trial exactly two tools, paired. One tells you nothing comparative. Three triples your classification load and halves your rigour. Two, on the same pull requests, with only one posting inline.

Weight the config test as heavily as the precision number. If I had to choose between a tool at 65% precision with no suppression controls and a tool at 55% precision with per-path instructions, a tunable severity floor and persistent learning, I would take the second one, and I would expect it to pass the first within a month. Day-one precision is a starting position. The config surface is the slope.

Pick on sentiment trend as the tiebreaker, not the final-day score. A score that rises across days 4, 9 and 14 means the tuning is working. A score that falls means the team has started ignoring it, and nothing you do in month two reverses that.

And do not buy from anyone’s benchmark, including ours. Three of the numbers in the comparison above contradict each other on the same tools, and the two datasets that used to check each other now share a parent company. Published numbers build your shortlist. Your repositories make the decision. Start with the trial roundup, apply the four filters, and run the fourteen days.