From $8,000
Your AI reviewer has a number to beat. We build the test that proves it.
This is for the firm that already lets AI touch real work. An agent reviews pull requests, drafts tickets or summarizes documents, and someone on your team signs off. You trust it a little more every month, with no measurement that says you should. When an auditor or an insurer asks how good it is, the honest answer today is "pretty good." That is not a number. We build the number from your own history, so it measures your work, not a public benchmark's.
What you get
The deliverables, stated plainly.
A work-path inventory
Every place AI touches your firm: what goes in, what comes out, who signs off.
A golden set of 100–300 items
Drawn from your real history — pull requests, tickets, documents — with labels triangulated from your senior people, the follow-up fixes that actually shipped, static checks, and multiple models.
A severity rubric
Written definitions for blocker, major, minor and nit, so two people grading the same item land on the same word.
A calibrated grader
An agreement score against the golden set, stated as a percentage you can put in front of anyone.
A one-page scorecard
Your team can rerun it monthly without us.
Optional eval retainer
We rerun the set on every model or tool change and send the delta.
How it runs
Week one is inventory. We get system access within 5 days of signing, list the AI-touched work paths, and agree which ones go into the set. Week two is sampling and labeling: we pull items from your history, your senior people label a slice, and we triangulate the rest against follow-up fixes, static checks and model votes. Anything that disagrees goes back to a human. Weeks three and four are calibration: rubric, grader, agreement score. You see every draft within 3 days of asking, and we hold a weekly check-in throughout. The scorecard ships at acceptance with instructions to rerun it.
What it costs
Payment is 65% at signing and 35% on acceptance. The scope and the final number are set after the first conversation; the founder confirms the number on the call.
The number we quote
On October 6, 2026, GitHub published ReviewBench: 219 real pull requests from 187 repositories across 19 languages, sampled to match the distribution of 103.9 million GitHub pull requests. The golden labels were triangulated from human reviewers, the authors' own follow-up commits, static analysis and multiple frontier models. Independent senior engineers agreed with those labels 96.6% of the time, and offline scores tracked production outcomes. That is the bar your reviewer has to beat. The method is public and it works. What the public benchmark cannot do is run on your history, with your definitions of blocker and nit, against the tools you actually use. We can. The eval set is yours, it lives in your tenancy, and it lets you say your real number instead of "pretty good."
Questions people ask
We do not write software. Does this apply to us?
Yes. Code review is the clearest example because the public benchmark exists, but the method is the same for any work path where AI produces something a human approves: correspondence, intake summaries, reconciliations, claim notes.
How many items do we need?
One hundred covers one narrow path well. Three hundred covers several. We size it in week one and tell you why.
Who owns the golden set?
You do. It is built from your records, stored in your systems, and the scorecard runs without us. The retainer is for firms that would rather we run it.
What if the grader scores badly?
Then you learned the most useful thing this engagement can teach, before an auditor or a client learned it for you. The report says where it fails and what to change.
Book 15 minutes
The founder confirms the number on the call. Fifteen minutes, no deck, no pitch — we talk about your situation and tell you if there is nothing to build.