Free tool · No email required
The rubric that makes a golden set agree.
A golden set is only as good as the agreement between the people who labeled it. Two people grading the same item have to land on the same word. This is the severity rubric and labeling procedure we use on day one — print it and keep it next to your labelers.
Severity rubric
Blocker
The item must not ship or be acted on as submitted. Acting on it would move money wrongly, expose data, break a legal or contractual obligation, or take down something people depend on.
Examples: Payment captured twice. A customer's records visible to another customer. A migration drops a column still in use. A letter commits the firm to a term it cannot honour.
Action: Stop. Return to the author or discard the output. A named person confirms the fix before anything proceeds.
Major
A real defect that produces a wrong result or a user-visible failure in normal use, with a bounded blast radius and no irreversible side effect.
Examples: Wrong tax rate on one category. A null case that crashes a report. An intake summary that omits the client's deadline. A reconciliation that double-counts a line.
Action: Fix before release or before the output is used. May ship behind a flag with a documented workaround if the owner signs off.
Minor
A defect that degrades quality or maintainability without producing a wrong result in normal use. Would be fixed if noticed; nobody is harmed if it is not.
Examples: Duplicated logic. A misleading variable name. An unclear sentence in a client note. An edge case that is unreachable today.
Action: Fix when convenient. Record it; do not block on it.
Nit
Style, formatting or preference. No effect on behaviour or meaning.
Examples: Whitespace. Import order. Oxford comma. Wording the reviewer would have chosen differently.
Action: Optional. The author decides. Never counts against the agent or the reviewer.
None
The reviewer looked and found nothing worth raising. A deliberate verdict, not a skipped item.
Examples: A clean refactor with tests. A routine correspondence draft with correct facts and dates.
Action: Approve. Record that the item was reviewed and the verdict was None.
Two tests settle most arguments. First, can the consequence be undone? If not, it is at least Major and probably a Blocker. Second, would a client, a regulator or an auditor notice? If yes, it is at least Major. If neither applies, it is Minor or Nit, and the difference is whether anything about the output is actually wrong.
Labeling procedure
- 1
Sample the items. Pull them from real history — pull requests, tickets, documents, intake notes, reconciliations. Do not write synthetic items; the set should look like the work. One hundred items covers one narrow work path. Three hundred covers several.
- 2
Two labelers label independently. Each reads the item, writes the expected finding in their own words, and picks a severity from the rubric. They do not confer and they do not see each other's column until both are done.
- 3
Resolve disagreements with a third person. Where the two severities differ, a third labeler reads both rationales and decides. The decision and reason go in the Severity and Rationale columns. The original two labels stay in place so the disagreement rate is visible.
- 4
Record the rationale for every item, agreed or not. One line is enough. A label without a rationale cannot be audited and cannot be used to train the next labeler.
- 5
Run the grader against the set and log every run on the Grader Runs sheet: date, tool, version, items graded, agreements, true positives, false positives, false negatives. The scorecard reports agreement, precision, recall and the gap to the 96.6% bar.
- 6
Re-label 10% quarterly. Pick items at random, have a fresh pair label them blind, and compare with the stored labels. If agreement on the re-labeled slice falls below the inter-labeler agreement you started with, the rubric has drifted or the work has changed. Fix the rubric first.
What the numbers mean
Inter-labeler agreement is the ceiling. A grader cannot be expected to agree with the golden set more often than the humans who built it agree with each other. If that figure is below 90%, tighten the rubric before judging any tool.
Grader agreement is the number you quote. It answers "how good is it" with a percentage, a date and a method. Precision says how many of the grader's findings were real. Recall says how many of the real findings it caught. A grader that raises everything has high recall and poor precision; a grader that stays quiet has the reverse. You want both high.
When the kit is not enough
The starter kit gets a firm to a first honest number in a few weeks of part-time labeling. What it does not do is pull the items from your systems, triangulate the labels against follow-up fixes and static checks, calibrate a grader to your rubric, or hand you a scorecard that reruns on every model change. That is the Private Eval-Set Build, and the set it produces is yours, in your tenancy.
Source: GitHub ReviewBench, published October 6, 2026.