GEO measurement and procurement

Interpreting before-and-after AI visibility evidence

Use an eight-field comparison record to judge whether before-and-after AI visibility tests are comparable, what changed, and which conclusions remain unsupported.

Published by SALIENS · Published 2026-10-10

Treat before and after as a comparison, not automatic proof

A later AI visibility test can show a different answer, mention, link or factual representation from an earlier test. That is evidence of an observed change under two sets of conditions. It does not, by itself, prove that a GEO supplier caused the change, because the system, interface, model, query interpretation, source set, session context and wider web may also have changed.

HM Treasury's Magenta Book distinguishes monitoring observed performance from evaluating what caused an outcome. It recommends planning the baseline and evaluation before an intervention and considering what might have happened without it. For a GEO engagement, the practical minimum is to freeze the comparison design, preserve raw evidence, log every material intervention and identify concurrent changes before interpreting the result.

Complete an eight-field comparison record

Use one controlled record for each question or question group. Agree the fields before implementation so the supplier cannot redesign the baseline after seeing the follow-up result.

Before-and-after AI visibility comparison record
Field	What to record	Acceptance check
1. Question and decision	Exact prompt, buyer intent, topic group, whether the brand is named, and the procurement decision the test informs.	The same question version is used in both periods, or the change is documented and results are not merged.
2. System and interface	Provider, product or feature, access route, account or session state, model label if displayed, and relevant settings.	The report does not combine materially different products or interfaces into one trend without a separate breakdown.
3. Market and language	Country or location condition, language, device class, logged-in state and any personalisation controls that are known.	Before and after use the same declared conditions; unknown conditions remain labelled unknown.
4. Sampling plan	Test dates and times, number of repeats, question order, valid-response rule, exclusions and retained evidence format.	Both periods have enough valid observations for the stated conclusion, and invalid runs are not silently removed.
5. Baseline evidence	Full answer or capture, brand mention, own-domain link or citation, cited third parties, factual accuracy and refusal or no-answer state.	Raw evidence is timestamped and can be reclassified by another reviewer.
6. Intervention log	Pages, facts, content, technical changes, publication dates, approvals and known crawl or index state.	Each claimed intervention has a live or inspectable delivery record; a recommendation is not recorded as implemented.
7. Follow-up evidence	The same evidence fields as the baseline, plus the elapsed time from each relevant change.	The result is reported by question and condition before any roll-up percentage is presented.
8. Context and conclusion	Model or interface changes, search or logging incidents, other campaigns, seasonality, competitor or source changes, decision and confidence.	The conclusion separates observed change, plausible contribution and causal proof, and lists unresolved alternatives.

Compare like with like before calculating a percentage

Keep branded and brand-blind questions separate. A prompted brand answer tests factual representation when the brand is already supplied; an unprompted buyer question tests discovery among alternatives. Keep mentions, own-domain links or citations, third-party citations, factual accuracy and answer completion as separate fields because movement in one does not imply movement in the others.

For each metric, show the numerator, denominator and valid sample count for both periods. A change from one mention in two valid tests to two mentions in two tests is not the same evidence as a change from 100 in 200 to 200 in 200. Report the question-level results and exceptions alongside any rate so that a buyer can see whether a headline movement came from broad improvement or one repeated prompt.

Use the same fixed question set, wording version and classification rules.
Match market, language, interface, session treatment and sampling schedule.
Retain answers, links, visible citations, timestamps and exclusions.
Show counts and denominators; do not report only a percentage change.
Separate factual correction from discovery, traffic and enquiries.

Check the reporting system and external changes

Google states that AI Overviews and AI Mode can use different models and techniques, so their responses and links may vary. It also says their site traffic is included in overall Web search traffic in Search Console. Unless an authorised report supports a more specific split, a change in overall Search Console clicks or impressions should not be relabelled as a change caused by one AI feature.

Search Console's Performance report also depends on the selected search type, date range, dimension and aggregation. Its newest data can be preliminary, and weekly or monthly views can help smooth daily fluctuations. Before comparing periods, inspect the same filters, include complete dates and check Google's data-anomalies log. A platform logging issue or aggregation change is not a website performance change.

Use an interpretation ladder that matches the evidence

A procurement report should state the strongest conclusion its design supports, then stop. The supplier can explain a plausible mechanism, but should not promote it to attribution without evidence that addresses credible alternatives.

Evidence strength and supportable interpretation
Evidence	Supportable statement	Do not claim
Comparable retained tests show a different answer	AI visibility changed for these questions under these recorded conditions.	The supplier caused the change or the result applies to all users and questions.
Comparable tests plus an implemented change log and timing	The change is consistent with the intervention and may have contributed.	Other platform, source, market or concurrent changes have been ruled out.
Stable comparison group or another suitable counterfactual design	The evaluation provides stronger evidence about additional effect, subject to its assumptions and uncertainty.	Every observed difference is caused solely by GEO work.
Search Console movement over the same dates	Overall Google Web search performance changed for the selected filters and aggregation.	A named AI feature, prompt or supplier action produced that movement.
Website visit or verified enquiry evidence	A visit or enquiry occurred and can be attributed only under the documented matching rule.	A visibility observation automatically became a visit, lead or sale.

Write the supplier acceptance test

Ask the supplier to return the frozen question set, condition register, sampling plan, raw baseline and follow-up evidence, classification dictionary, intervention log, platform and anomaly check, question-level results, calculation workbook, exclusions, confidence statement and decision recommendation. Select several improved, unchanged and worse results and trace each from the summary back to the raw evidence.

Reject a report that shows only favourable screenshots, changes prompts or markets between periods without disclosure, hides the denominator, treats a recommendation as implemented, merges mentions with citations, or calls correlation a guaranteed GEO effect. Also reject a claim that publication, crawling, indexation, AI inclusion, traffic or enquiries were achieved when the corresponding evidence was not collected.

Can another reviewer repeat the test from the recorded instructions?
Are all included, excluded and invalid observations accounted for?
Can every implemented change be traced to an approval and live check?
Are platform changes, incidents and concurrent activity documented?
Does the final wording distinguish observation, contribution and attribution?

How SALIENS can support a defensible comparison

SALIENS provides GEO audits, AI visibility monitoring, content and technical improvements, and ongoing GEO delivery for UK and international organisations. We can design a brand-blind and branded baseline, preserve repeatable test conditions, map approved interventions, review English and Simplified Chinese results and report delivery, visibility, traffic and enquiries as separate evidence layers.

Email info@oxfordintelligence.co.uk with your website, target markets and languages, priority buyer questions, current measurement access and the changes you expect to test. SALIENS is operated by Oxford Intelligence Limited. This work does not guarantee crawling, indexation, mentions, citations, rankings, visits or enquiries.

Sources

Prepared with AI assistance using the linked sources and SALIENS service information.