Why do AI engines give different answers each time for the same query?

AI language models have temperature settings that introduce controlled randomness into each response. They also update their training data and search indices over time. These factors mean no two runs of the same prompt will produce identical outputs, even on the same day.

What is inter-rater reliability in AI measurement?

Inter-rater reliability measures how consistently two independent raters (human or AI) grade the same output. In AI visibility measurement, it quantifies how often two rater models agree on whether a brand is prominently cited, neutrally mentioned, or absent. Cohen's kappa is the standard statistic. Above 0.6 is generally considered acceptable agreement. For transparency about our own numbers: Pondral changed its Context grader in August 2026, so there are two sets of figures and we publish both. The grader now in production, measured on 150 randomly drawn real AI engine responses, agrees with a different-provider grader on 87.3% of cases (kappa 0.624), clearing that 0.6 convention. Measured instead against a larger grader from the same provider, agreement is 72.0% (kappa 0.402), which is a real regression from the previous grader and does not clear it. Which of those two comparisons should be the bar we hold ourselves to is a question we have not settled, so we do not present either alone. The previous grader measured 82.7% and kappa 0.645 same-provider, 0.599 cross-provider; those figures appeared on this page until August 2026 and describe a grader that is no longer running. Disagreements in both cases concentrate on a single neutral-versus-positive boundary. All of these are single measurements on our own benchmark responses and are directional (PROXY), not settled figures.

How does Pondral handle AI answer variability?

By default Pondral grades one response per (query, engine) pair and reports the mean across the audit. For brands that want to measure consistency directly, repeated sampling is available on request: it evaluates several responses per pair and reports Mention Rate, Quality When Mentioned, and a combined Visibility Index. The raw responses we ship with every audit expose per-cell variance. Multi-run averaging by default, prompt variations, and inter-rater kappa flagging are described in this post as planned methodology evolution.

What is a confidence interval in the context of AI visibility scores?

A confidence interval shows the range of plausible true values for a score given the variability across runs. A score of 68 ± 14 means the true value is likely between 54 and 82. A wide interval means you need more data before acting on that score. Displaying confidence intervals directly on the dashboard, rather than hiding score uncertainty, is on Pondral's methodology v3 roadmap.

← All postsJournal

Inter-rater reliability for AI answers.

Published Feb 12, 2026By Pondral TeamRead time 14 min read

Editor's note (2026-05-18): this post describes inter-rater reliability concepts and the methodology Pondral plans to adopt at scale. Current implementation grades one response per (query, engine) pair and reports the mean across the audit. Repeated sampling (Mention Rate, Quality When Mentioned, and a combined Visibility Index) is available on request, not enabled on standard audits. Multi-run averaging by default, Cohen's kappa flagging, and 95% confidence intervals are roadmap items for further statistical rigor. See /methodology for the shipped specification.

Update, August 2026. Two things below need separating. Measuring our Context grader's agreement is done. Surfacing kappa flagging inside the product is still a roadmap item. We changed the grader in August 2026, so there are two sets of numbers and we publish both rather than the flattering one. Measured on 150 randomly drawn real AI engine responses, the grader now in production agrees with a different-provider grader on 87.3% of cases (Cohen's kappa 0.624), which clears the 0.6 convention described below. Against a larger grader from the same provider it agrees on 72.0% (kappa 0.402), which does not clear it and is a real drop from the previous grader. Which of those two comparisons is the bar we should be held to is genuinely unsettled, and we would rather say so than quote whichever number flatters us. The previous grader measured 82.7% and kappa 0.645 same-provider, 0.599 cross-provider; those are the figures that stood on this page until August 2026 and they describe a grader that is no longer running. We are stating all of this plainly rather than leaving the convention on this page and our own number on another. These are single measurements on our own benchmark responses and are directional (PROXY), not settled figures. They supersede an earlier published figure of 0.947, which was measured on short sentences written by our own team rather than on real engine responses. See the correction note for what was withdrawn and why.

A question we get often: "How do you handle the fact that AI models give different answers each run?"

The short answer is: with statistics. The long answer involves Cohen's kappa, bootstrapped confidence intervals, and a methodology paper we'll publish later this quarter.

The framework we describe below is the published end-state. Today, Pondral runs each (query, engine) pair once and reports the mean across all results in the audit. Cohen's kappa flagging and multi-run prompt variations are slated for the methodology v3 release. For a plain-English explanation of each factor we measure, see the five-factor rubric, explained slowly.

The shipped behavior is: a single sample per (query, engine), aggregated into a mean score. We publish the methodology change log so every score is traceable to a specific rubric version and run configuration. See methodology.

← The death of brand-keyword search.

Last updated June 2026Run a free audit →