We Audited Pondral with Pondral.
Here's What We Found.

We ran our own AI visibility tool against our own brand, then against 20 other brands with known visibility profiles. Three things came out of it.

1. Our published methodology and our production code didn't match. They do now.
2. Our actual AI visibility on category queries is essentially zero. That's not a failure, it's a measurement. Now we know where we start.
3. In one run across 20 brands spanning household names to startups no one has heard of, the reconciled formula's ranking matched our visibility tiers with a rank correlation of 0.961. That run held Context at neutral and left out Gemini, so it is directional (see the caveats below).

If you're using an AI visibility tool, including ours, and the vendor can't show you data like this, ask why.

Why we did this

A number like “your AI visibility score is 47” means little without knowing how it was computed, what it correlates with, or whether it would give the same answer if rerun. We promised something different on /methodology: a public formula, evidence behind every score, and a versioned changelog. We grade one response per query and engine and report the mean.

In May we ran our own audit against ourselves and discovered we hadn't fully kept that promise. Two formulas existed in our system. The one we published was a 5-factor weighted-sum. The one our code actually used was a 4-factor SOV-based formula. They were both reasonable, but they weren't the same. Customers who read our methodology page would have computed a different number than the one we showed them.

This post is the audit trail. The disclosure of what we found, the math we changed, and the data showing the change is real.

What we found

1. Two formulas, one product

The /methodology page described five factors with specific weights:

That formula lived in the codebase, correctly implemented. But it wasn't what produced your customer score. The number on your dashboard came from a separate aggregator computing: 37% Brand SOV + 33% Generic SOV + 25% Owned Citations + 5% Grounded Mentions.

Both formulas are defensible. Neither was wrong. But they weren't the same, and the gap between them was invisible to customers, and to most of us internally.

2. Our own visibility was barely measurable

In a single run on 2026-05-05 under the old 4-factor formula (PROXY), here's what we got for pondral.com:

Not 30%. Not “limited.” Zero. Across 18 attempts spanning ChatGPT, Gemini, Perplexity, and Claude, asking category questions without naming us, we appeared in zero responses.

This is exactly what a brand-new tool with no Wikipedia entry, no G2 listing, and no third-party press should expect. But until we measured it, we were guessing.

3. Three production bugs surfaced during the run

The audit didn't run cleanly the first time. Three real bugs, bugs that had been silently broken in production code, came out:

A missing database column. Our audit pipeline tried to write ‘sentiment’ to a column that didn't exist in any migration. Every audit attempt failed at insert time.

An environment variable name mismatch. Our Gemini provider read GOOGLE_API_KEY while the actual variable was GOOGLE_AI_API_KEY. Every Gemini call silently failed.

A stale model reference. Our Anthropic provider referenced a model name that doesn't exist on their API. Every Claude call returned 404.

We're disclosing these because methodology integrity isn't optional and isn't private. If we hide the boring failures, you have no reason to trust the numbers we publish.

What we changed

Reconciled formula (v2.0.3)

We wired the published 5-factor module into the actual production code path. Now when you see a score, it was computed from the formula on /methodology, not from a different formula behind it. The old 4-component weight constants are deprecated in the codebase.

Measuring the Context grader

The 5-factor formula has a “Context” component that grades whether the AI's mention of you is accurate, in the right category, and not damaging. That requires an LLM judge. We needed to measure how reliable that judge is.

PROXY. Single measurement, methodology v2.8.0 (the brand-blind grader), October 2026. 150 responses drawn at random from our own benchmark audit runs, not from customer data. This compares AI graders with each other, not with human-labelled answers. One measurement on one body of responses is directional, not a settled figure, and we have not published a margin of error for it because we have not yet computed one we can show our working for.

The other numbers. The 0.6168 above is agreement with a grader built by a different AI company, which is the bar we hold the grader to. Measured instead against a larger grader from the same company as ours, agreement is 83.3% and kappa is 0.6461. On the same 150 responses, the previous grader, which was told the brand's name (methodology 2.3.0 through 2.7.0), measured 87.3% and kappa 0.6238 against the different-provider grader, and 72.0% and kappa 0.4017 against the same-provider grader. We publish all four rather than the flattering one.

Correction, July 2026. From June 2026 until July 2026 this section published a Cohen's kappa of 0.947 and a calibration figure of 0.957, both in the visible text and in the structured data this page serves to search and AI engines. We have withdrawn both. The 0.947 was measured on 30 short sentences our own team wrote, not on real AI engine responses, so it never measured how the grader performs on the messy real inputs it actually sees. The 0.957 came from a script that used a different rubric from the one running in production, and that counted every failed grader call as a perfect match. Neither number should have been published. We are leaving this note in place rather than quietly deleting the old figures.

Construct validation: does the formula actually rank brands correctly?

We tested 20 brands against the reconciled formula. We grouped them into three tiers based on their real-world AI visibility: T1 (household names that AI regularly recommends), T2 (established but not dominant), and T3 (niche or new, rarely mentioned by AI unprompted).

Two caveats on this run. Context was held at the neutral bucket for all brands because the grader was still in calibration that day. And Gemini was excluded due to quota limits, with Claude partial (rate-limited mid-run). The neutral Context bucket alone is worth 10 points, which is the T3 mean. We have not run a confirmation with all 5 engines and live Context grading.

All scores in this study reflect publicly observable AI engine responses and do not use any proprietary data from the named companies. Tier assignments are based on frequency and quality of unprompted AI mentions, measured by the same methodology we apply to every Pondral audit.

What this means for your number

We have no paying customers as of this writing, so a score-shift analysis across customers would be meaningless. Instead of pretending: every methodology change, and what it does to scores, is recorded on the methodology changelog at /methodology/changelog. If your score changed more than you expected after a methodology change, contact us.

What we still don't know

1. We haven't proven the score predicts business outcomes. Construct validity (the score reflects known visibility) is not the same as predictive validity (the score predicts pipeline). We don't have the panel data to test that, and we have not run a customer outcome study.

2. AI engines change underneath us. The exact same query can return a different answer next week because the model updated, the index refreshed, or the search tool changed behavior. The engine and model version is recorded on every stored answer. Monitoring takes one reading per run, so a rerun that moves a score by under 15 points is treated as directional.

3. The competitor list shapes one factor. The Competitive Presence factor depends on which competitors are listed for your brand, so scores are partly shaped by that list.

What you can do with this

If you're a Pondral customer: Read the methodology changelog for the v2.0.3 entry. The 20-brand construct validation summary and the methodology are public. Check it yourself. If your specific score moved more than you expected, contact us.

If you're evaluating AI visibility tools: Ask each vendor for the formula. The exact one. Not “we use a proprietary methodology.” Ask for the inter-rater reliability number. If they can't compute one, the score isn't reproducible. Ask for construct validation data.

If you're another B2B SaaS founder: the cost of writing this post is publishing your gaps before someone else does. That cost is much smaller than the cost of methodology integrity disclosure done in crisis mode after a customer or competitor finds the gap first.

What's next

We have not run a customer outcome study, so the formula's predictive validity is untested. The methodology changelog at /methodology/changelog is the canonical source for any future change.

If you want to follow methodology updates, follow Pondral on LinkedIn.

-- Philipp Groubii, Founder, Pondral