Editor's note: this post describes inter-rater reliability concepts. Pondral's monitoring grades one response per engine for each question on each run and reports the mean across all graded answers. Changes under 15 points between runs are directional, and the dashboard does not show a margin of error. Consulting audits ask each question several times per engine and report a margin of error. See /methodology for the shipped specification.
Update, October 2026. Since October 14, 2026 our Context grader is not told which brand it is reading. We re-measured its agreement on the same 150 randomly drawn real AI engine responses as the August measurement. Against a different-provider grader it agrees on 86.0% of cases (Cohen's kappa 0.6168); the grader it replaced measured 87.3% (kappa 0.6238) on those same responses. Against a larger grader from the same provider it agrees on 83.3% (kappa 0.6461), where the grader it replaced measured 72.0% (kappa 0.4017). The different-provider comparison is the bar we hold the grader to, which answers the question the August update below left open. These figures compare AI graders with each other, not with human-labelled answers, and are single, directional measurements (PROXY). The August update is kept as it was written.
Update, August 2026. Two things below need separating. Measuring our Context grader's agreement is done. The product does not surface kappa flagging today. We changed the grader in August 2026, so there are two sets of numbers and we publish both rather than the flattering one. Measured on 150 randomly drawn real AI engine responses, the grader in production from July 31 to October 14, 2026 agreed with a different-provider grader on 87.3% of cases (Cohen's kappa 0.624), which clears the common 0.6 rule of thumb. Against a larger grader from the same provider it agrees on 72.0% (kappa 0.402), which does not clear it and is a real drop from the previous grader. Which of those two comparisons is the bar we should be held to is genuinely unsettled, and we would rather say so than quote whichever number flatters us. The previous grader measured 82.7% and kappa 0.645 same-provider, 0.599 cross-provider; those are the figures that stood on this page until August 2026 and they describe a grader that is no longer running. We are stating all of this plainly rather than leaving the convention on this page and our own number on another. These are single measurements on our own benchmark responses and are directional (PROXY), not settled figures. They supersede an earlier published figure of 0.947, which was measured on short sentences written by our own team rather than on real engine responses. See the correction note for what was withdrawn and why.
A question we get often: "How do you handle the fact that AI models give different answers each run?"
The short answer: we take one reading per engine for each question on each run, link every score to the stored answer it was graded from, and treat changes under 15 points as directional. Cohen's kappa comes in when we check how consistently our grader agrees with another grader.
Today, Pondral runs each (query, engine) pair once and reports the mean across all results in the audit. The product does not flag rater disagreements or run prompt variations. For a plain-English explanation of each factor we measure, see the five-factor rubric, explained slowly.
The shipped behavior is: a single sample per (query, engine), aggregated into a mean score. We publish the methodology change log so every score is traceable to a specific rubric version and run configuration. See methodology.