How We Built Pondral's Scoring Rubric: Design Decisions and Trade-offs

The problem with existing approaches

When we started building Pondral, the AI visibility tools that existed used one of two approaches: binary (mentioned/not mentioned) or opaque (a score from an undocumented algorithm). Neither was good enough.

Binary measurement tells you whether you're in the conversation but nothing about the quality of that presence. Being mentioned in a negative context, or mentioned last in a list of ten competitors, registers the same as being the first-recommended brand with a citation link. That is not a useful signal for making optimization decisions.

Opaque scores have the opposite problem. They provide a number, but without knowing what the number measures, you cannot act on it. If your score drops from 72 to 65, you need to know whether that is because you lost citation links, the way you're described changed, or a competitor gained share. A black-box score hides the actionable detail.

We designed Pondral's 5-factor scoring rubric to solve both problems: multi-dimensional (not binary) and fully transparent (not a black box).

Why five factors, and why these five

We considered more candidate factors than we kept before settling on five. The selection criteria were: each factor must be independently measurable, actionable (you can act on it), and meaningful (it describes something a reader of the answer would notice, such as where and how the brand appears).

The five factors we selected are Presence (20%), Prominence (25%), Context (20%), Citation Link (20%), and Competitive Presence (15%). Each measures a distinct dimension of AI visibility, and together they capture the full picture of how well a brand is represented in an AI-generated response. For a plain-language walkthrough of each, see the five-factor rubric, explained slowly.

Factors we considered and rejected include response length (too noisy, varies by engine), source count (measures the AI's behavior, not your brand's presence), and query-level relevance (already implicit in query selection, not a per-response metric).

Why Prominence gets the highest weight

Prominence, where in the response your brand appears, is weighted at 25%, more than any other factor. This was a deliberate judgment call, not the output of a fitted model.

Our reasoning is about how people read. A brand mentioned in the first sentence is the one a reader is most likely to see. A brand mentioned in the last sentence may go unread. We have not measured click-through from AI answers, so this is a design assumption, not a finding.

It is the same assumption behind ranking positions in traditional search results. In the rubric, being mentioned first counts for more than being mentioned at all.

Ordinal buckets, not continuous floats

Each factor is scored into one of five buckets: 0, 25, 50, 75, 100. We chose ordinal buckets over continuous floats because of how we measure: an LLM rater evaluates each response against the rubric.

LLM judges are reliable at ordinal grading (“this is better than that”) but unreliable at fine-grained regression (“this deserves a 73, not a 76”). Using continuous floats would create false precision. The rater would generate numbers that look precise but aren't reproducible. Ordinal buckets match the resolution the underlying signal actually supports.

This decision also makes scores more interpretable. A Prominence score of 75 means the brand appeared early in the response. A score of 25 means it appeared late. There are no arbitrary decimal points to interpret or explain.

One reading per engine, per run

Pondral's monitoring takes one reading per engine for each question on each run, and the score is the mean across every graded answer.

AI engines give different answers between runs. A change of under 15 points between two runs is directional: read it as a reason to look at the answers, not as a settled shift. The dashboard does not show a margin of error.

Consulting audits work differently: they ask each question several times per engine and report a margin of error alongside the score.

Separate rater for Context

The Context factor (how well the brand is described: portrayed favourably, described accurately, and placed in the right category) is graded by a separate LLM rater, not the same engine that generated the original response. This avoids self-assessment bias, where an engine might evaluate its own output more favorably than an independent judge would.

Today only the Context factor uses a separate LLM rater (Claude Haiku). We do not run a second rater on every answer, and the dashboard does not flag rater disagreements. The agreement checks we have run on the Context rater are written up in inter-rater reliability.

Transparency as a design principle

Each score on the Pondral dashboard links to the stored answer it was graded from, so you can read the answer yourself and check the grading.

This was a non-negotiable design decision. A score from an undocumented algorithm gives you nothing to check it against. AI answers vary between runs, so if you ask the same question yourself you may get a different answer. The stored answer is the one we graded.

For the full technical specification, read our methodology page, which documents every formula, weight, and threshold in the scoring system.

What we got wrong and changed

Building the scoring rubric was iterative. Some decisions we changed along the way:

Original Competitive Presence weight: 25%. We initially weighted Competitive Presence higher, but found it was too volatile. A competitor's absence from a single run inflated scores without any action from the brand being measured. We reduced it to 15% and shifted weight to Prominence.

Three-bucket scoring. Our first version used three buckets (0, 50, 100). This was too coarse. There was no way to distinguish between a brand mentioned in the second sentence and one mentioned in the second-to-last paragraph. Five buckets provided the resolution raters could support reliably.

Single rater for all factors. Using a single rater introduced self-assessment bias on the Context factor. We split Context out to a separate rater to remove that path.

Try it yourself

The best way to understand the scoring rubric is to see it applied to your brand. Run a free visibility check on Pondral's homepage and see a factor-by-factor breakdown across 2 AI engines. Paid plans cover all 5.