What Does "95% Accurate" Mean When Two Experts Agree Only 75% of the Time?

What does “95% accurate” mean when two experts agree only 75% of the time?

Every legal AI product is sold with a number. Ninety-five percent accurate. Ninety-eight percent recall. The number is meant to end the conversation, and usually it does, because the obvious follow-up question — accurate against what? — sounds pedantic until you find out the answer.

A recent measurement study makes the problem unusually concrete. It is not a legal paper; it is an operations paper about corporate AI-risk disclosure. But it does something most vendor benchmarks do not: it publishes what happened when it asked two qualified human beings to do the same classification task, and it publishes how much they disagreed.

The number under the number

The study built a two-stage language-model pipeline to read 11,271 corporate posts and score them against a risk taxonomy of 7 top-level and 20 sub-level categories drawn from 18 prior studies. One agent decides whether a category is triggered at all. A second agent, only if the first said yes, assigns an intensity score from 0 to 10 against a written rubric.

To validate it, two independent raters with backgrounds in AI governance and technology-risk management reviewed a stratified sample — 144 posts, roughly 246 post-category pairs. Here is what came back:

• Raw agreement between the two humans on whether a category was triggered: 75%.
• Reliability of their intensity scores against each other: ICC(2,1) = 0.76.
• The pipeline, measured against the cases where both humans agreed a trigger was present: mean absolute error 0.65 on the 0–10 scale, correlation 0.86.

Read those in order. Two trained specialists, applying a written rubric to the same text, reached different answers on a quarter of the decisions. The machine was then scored against the subset where they happened to agree.

The authors say so themselves, and the sentence is worth quoting because no vendor deck contains it: because the benchmark conditions on rater agreement, the statistics evaluate conditional scoring consistency rather than end-to-end classification validity.

That is a careful researcher telling you what their own number is not.

Why this matters to a firm buying software

Nothing about the 75% figure is a scandal. Trained people disagree; that is why appellate courts exist. The point is what it does to a vendor’s accuracy claim.

If human experts agree 75% of the time on a task, then “95% accurate” cannot mean what a reader naturally assumes. It means the system matched some particular reference set, built some particular way, on some particular sample. Change the reference set and the number moves — without a line of code changing.

In legal work the equivalent tasks are everywhere. Is this document responsive. Is this passage privileged. Does this testimony support that proposition. Is this clause unusual. Each is a judgement call on which two competent lawyers will sometimes split, and each is routinely marketed with a percentage that implies they would not.

Five questions that make an accuracy claim mean something

1. What was the ground truth, and who made it? One reviewer, a panel, or a previous version of the same model? A benchmark labelled by the vendor’s own system is a measure of self-consistency.
2. What did the humans agree on with each other? If that figure has not been measured, the accuracy claim has no ceiling to be compared against. If it has been measured and is not disclosed, ask why.
3. Was the machine scored only on the cases where the humans agreed? This is the specific trap the study names in itself. Scoring on the easy subset inflates the number and hides exactly the cases you were worried about.
4. What sample, and how was it drawn? Stratified, random, or the cases that worked? Sample size and construction decide how much a percentage is worth.
5. What version produced it? The study’s production recommendation is to version the model, the prompt, the taxonomy, the rubrics, the temperature and the retry logic — so that a change in the output can be attributed to a change in the documents rather than a change in the measuring instrument. A benchmark from a version you are no longer running is a historical record, not a guarantee.

A vendor who can answer all five has done real work. A vendor who treats the questions as hostile has told you what you needed to know.

The naming discipline

One more thing from the paper transfers cleanly, and it is the part most likely to save an awkward conversation.

Having built the score, the authors conclude that it should be described as risk-disclosure salience, not risk intensity — because what the pipeline actually measures is how prominently a firm communicates about a risk, not how exposed the firm is. Same number, honest label, much narrower claim.

Legal tooling needs the same discipline. A system that surfaces passages matching a concept is doing retrieval, and calling that output “the privileged documents” rather than “documents to review for privilege” moves a decision from the lawyer to the software without anyone deciding that it should. The label is not marketing detail. It is the boundary of what someone is entitled to rely on.

What we would rather be measured on

We publish no accuracy percentage for Lawnova PDF, and this is why: for the work it does — finding the passage, returning the exchange around it, showing where it came from — the honest metric is not whether the machine was right. It is how long it takes you to confirm whether the machine was right.

Every answer carries a document, page and line reference you can open. Confirming takes seconds, which means confirming actually happens, which means the number that matters is one you can check yourself rather than one we assert. We have argued the same thing about verification time in document review, and it is the same principle behind the certification standing orders now in force in the District of Colorado: a citation you can open in seconds is a certification you can sign honestly.

A percentage asks for your trust. A citation gives you a way to withhold it.

Source: Tianhui Li (Renmin University of China) and Yueyang Zhong (London Business School), “AI-Risk Disclosure Is Not One-Size-Fits-All: An Agentic Measurement System and Network Evidence,” SSRN working paper 7400879. The validation figures are from the paper’s operationalisation section and Appendix B.3. The paper studies corporate disclosure, not legal document review; what transfers is its measurement discipline, not its findings. This page is commentary, not legal advice.