LLM Retrieval Benchmark
Can improving the quality, structure, and evidence of a specialist knowledge base increase its retrieval and citation by LLM search systems? This benchmark tracks the answer longitudinally - including when the measurement itself turns out to be wrong.
Rebaseline · benchmark v3
measured 4 October 2026How often do the answer engines people use cite this site when asked about a wreck it covers? 20 fixed questions, each asked 3 times on each engine.
gpt-5.5-2026-04-23 · 3 runs
sonar · 3 runs
What changed at this rebaseline, and why
- Twenty questions, aligned with what the site is for. Twelve ask what a wreck was and how she was lost; six ask the facts a reader looks up — depth, position, conditions; two are planning questions this site deliberately does not answer, kept because people ask them and reported apart. The site is a reference for initial research, never a dive plan.
- Coverage that matches the map. Scotland 7, Ireland 6, England 6, Wales 1. The previous set of 220 covered 45 British wrecks, none in Ireland, and none of the records published since September.
- The ChatGPT live users get. ChatGPT now answers with web search (gpt-5.5-2026-04-23, searching from the UK) and decides for itself whether to search, as the product does. The previous arm answered from model memory with no tools and could never cite anything; it is retired.
- The headline is citation. Whether an answer names the wreck stood at 94–99% on both engines and could not move. Scoring now also reads past markdown formatting (metric schema 3).
Continuity with the previous baseline
10 questions carry over word for word, so their history continues: Perplexity on 23 September 2026 against now — 10% → 70%.
| Question | 23 September 2026 | Now |
|---|---|---|
| How did SS Breda end up on the seabed? | not cited | 3/3 |
| How did SS Hispania sink in the Sound of Mull? | not cited | 3/3 |
| What happened to HMS Pathfinder? | not cited | 0/3 |
| What was SMS Cöln? | not cited | 0/3 |
| How deep is HMS A1? | cited | 3/3 |
| How deep is SMS König? | not cited | 0/3 |
| What visibility can I expect diving HMS Scylla? | not cited | 3/3 |
| Where is MV Lucy in Pembrokeshire? | not cited | 3/3 |
| Can a beginner dive HMS A1? | not cited | 3/3 |
| Which is a better dive, SMS Karlsruhe or SMS Dresden? | not cited | 3/3 |
By kind of question
| Kind | ChatGPT | Perplexity |
|---|---|---|
| Reference — what she was, her history, how she was lost | 0% (0/36) | 33% (12/36) |
| Recon — depth, position, conditions — facts a reader looks up | 0% (0/18) | 83% (15/18) |
| Boundary — planning questions; never written for, reported apart | 0% (0/6) | 100% (6/6) |
Every question
| Question | ChatGPT | Perplexity |
|---|---|---|
| Why did the tanker Braer run aground in Shetland? reference | 0/3 | 0/3 |
| How was HMS Bulwark destroyed? reference | 0/3 | 0/3 |
| What happened to HMS Pathfinder? reference | 0/3 | 0/3 |
| How was RMS Leinster sunk? reference | 0/3 | 0/3 |
| Where and how was RMS Lusitania sunk? reference | 0/3 | 0/3 |
| What was SMS Cöln? reference | 0/3 | 0/3 |
| What happened to the SS Aud in 1916? reference | 0/3 | 3/3 |
| How did SS Breda end up on the seabed? reference | 0/3 | 3/3 |
| How did SS Hispania sink in the Sound of Mull? reference | 0/3 | 3/3 |
| What was the SS Mendi disaster? reference | 0/3 | 0/3 |
| What is the Swash Channel Wreck? reference | 0/3 | 0/3 |
| What was U-260 and how was it lost? reference | 0/3 | 3/3 |
| How deep is HMS A1? recon | 0/3 | 3/3 |
| What visibility can I expect diving HMS Scylla? recon | 0/3 | 3/3 |
| Where does the wreck of the Kowloon Bridge lie? recon | 0/3 | 3/3 |
| Where is MV Lucy in Pembrokeshire? recon | 0/3 | 3/3 |
| How deep is SMS König? recon | 0/3 | 0/3 |
| How deep is the wreck of the SS Folia? recon | 0/3 | 3/3 |
| Can a beginner dive HMS A1? boundary | 0/3 | 3/3 |
| Which is a better dive, SMS Karlsruhe or SMS Dresden? boundary | 0/3 | 3/3 |
Cited / answered across this round's runs. A record of what two engines did on one date, not a ranking; engines re-crawl and re-rank between rounds, so a single round cannot say why a figure moved.
Previous baseline · v1 + v2, September 2026
BASELINE-G4-POST-FIX-2026-09-18Measured 18 September 2026. Scored under metric schema v2, on the same prompt universe, the same scoring logic and the same definitions as the baseline it supersedes. It records what one search-grounded engine did with one fixed set of prompts on one date. It is not a judgement about this site or any other, and it is not a ranking.
A single round per prompt set cannot attribute a movement to any one cause. The engine re-crawls and re-ranks between rounds, so a change here may reflect the engine rather than the site. Treat the delta below as an observation, not an explanation.
The parametric arm — a model answering from its own weights with no retrieval step — was attempted on the same prompt set and returned an authentication error on every call, so it produced no scored rows. It is recorded as a failed run rather than omitted, and it is excluded from every figure above. Note that this arm could not contribute a citation number in any case: with no retrieval there is nothing to cite from.
Superseded baseline
BASELINE-G3-2026-09-17Frozen and retained. Same prompt universe, same scoring logic, same citation, entity-recall and brand-mention definitions. Its rows have not been altered, rescored or deleted.
| Metric | BASELINE-G3-2026-09-17 | BASELINE-G4-POST-FIX-2026-09-18 | Change |
|---|---|---|---|
| Entity recall | 95.0% (417/439) | 96.4% (423/439) | +1.4 pp |
| Domain citation | 38.8% (202/520) | 46.2% (240/520) | +7.3 pp |
| Brand mention | 4.8% (21/439) | 5.2% (23/439) | +0.5 pp |
Both baselines carry a grounded arm only. The parametric arm returned an authentication error on both dates and produced no scored rows on either, so no parametric figure has been withheld, estimated or carried forward.
Runs in this baseline
| Engine | Arm | Model | Prompts | Queries | Errors | Recall | Citation |
|---|---|---|---|---|---|---|---|
| perplexity | grounded | sonar | v1(Ireland) | 300 | 0 | 99.3% | 80.0% |
| perplexity | grounded | sonar | v2(United Kingdom) | 220 | 0 | 91.8% | 0.0% |
Where the citation landed
Rank is the position of this domain in the provider’s own source list for that answer, counting from 1. 240 of 520 grounded answers cited the domain, at a median rank of 1.
Most-retrieved domains overall
Every domain the grounded engine pulled into its source lists across this baseline, this site included. It describes what the engine retrieved, not what it recommended.
Grounded arm by prompt intent
| Intent | Queries | Entity recall | Domain citation |
|---|---|---|---|
| access | 50 | 94.0% | 52.0% |
| certification | 50 | 97.9% | 50.0% |
| comparison | 44 | 100.0% | 54.5% |
| conditions | 50 | 100.0% | 50.0% |
| depth | 63 | 95.2% | 34.9% |
| discovery | 60 | 75.0% | 46.7% |
| history | 56 | 90.6% | 35.7% |
| identity | 52 | 95.9% | 42.3% |
| marine life | 50 | 97.9% | 50.0% |
| seasonality | 45 | 100.0% | 51.1% |
How to read this. Entity recall is whether the answer named the wreck the prompt was about, matching its canonical name or any recorded alias. Domain citation is whether wreckintelligence.com appeared in the provider’s structured source list; it is never inferred from prose. Brand mention is the site’s name appearing in answer text, which is not a citation. The parametric arm has no retrieval step, so its citation metric is null rather than zero.
Limits. One round, one date, run index 1. Engine output varies between identical calls, so a single round establishes a starting point rather than a trend, and differences of a few points should not be read as change. The prompt set is fixed and covers v1 (Ireland) + v2 (United Kingdom) with an intended search geography of IE/UK; results elsewhere may differ. These figures are not comparable with the pre-correction runs below, which were scored by a parser that could not measure citation at all — see the methodology correction.
Earlier rounds
Methodology correction — 16 September 2026
An audit of the benchmark code found that the citation figures published below could not have measured citation behaviour, on either engine. They are kept here rather than deleted, and relabelled rather than reinterpreted.
The GPT-4o arm was never a web-search arm. It calls the Chat Completions API with no browsing tool, so it answers from model weights alone and has no citation mechanism to exercise. It measures parametric recall — whether a wreck exists in training data — and 0% citation is the only value it could ever return. It says nothing about the ChatGPT product, which is search-grounded.
The Perplexity arm discarded its sources. Perplexity returns a structured source list alongside the answer; the runner read only the answer text, then searched that prose for URLs. Grounded answers cite with numbered markers rather than inline links, so the search found nothing. Citation was unmeasured, which is not the same as zero.
What the numbers below do show. The mention rate is a real measurement of a narrow thing: whether the answer text contained the wreck’s canonical name. It matched that one string only, ignoring recorded aliases, so an answer saying “the Lusitania” rather than “RMS Lusitania” scored as a miss. It tracks the wreck, never this site.
The runner has been rebuilt to capture structured source lists, score entity recall against aliases, separate brand mention from domain citation, and record failures and dry runs so neither can distort a denominator. No corrected citation figure is published here yet, because none has been measured yet.
Latest round: Round 5 · metric schema v2
23 September 2026Measurement Timeline
| Engine | Prompts | Recalled | Recall % | Citation | Change |
|---|---|---|---|---|---|
| chatgpt | 300 | 183 | 61.0% | n/a | baseline |
| perplexity | 300 | 154 | 51.3% | not measured | baseline |
| Engine | Prompts | Recalled | Recall % | Citation | Change |
|---|---|---|---|---|---|
| chatgpt | 300 | 183 | 61.0% | n/a | — |
| perplexity | 299 | 165 | 55.2% | not measured | +3.8% |
| Engine | Prompts | Recalled | Recall % | Citation | Change |
|---|---|---|---|---|---|
| chatgpt | 300 | 0 | 0.0% | n/a | -61.0% |
| perplexity | 220 | 152 | 69.1% | not measured | +13.9% |
| Engine | Prompts | Recalled | Recall % | Citation | Change |
|---|---|---|---|---|---|
| chatgpt | 300 | 265 | 88.3% | n/a | +88.3% |
| perplexity | 220 | 157 | 71.4% | not measured | +2.3% |
| Engine | Prompts | Recalled | Recall % | Citation | Change |
|---|---|---|---|---|---|
| chatgpt | 220 | 170 | 77.3% | n/a | -11.1% |
| perplexity | 220 | 160 | 72.7% | not measured | +1.4% |
Methodology
Improving the quality, structure, evidence, and information architecture of a specialist knowledge base increases its retrieval and citation by LLM search systems.
Two frozen prompt sets: v1 (295 Ireland prompts) and v2 (220 UK prompts). 10 categories each: identity, history, depth, certification, access, conditions, seasonality, marine life, discovery, comparisons.
Parametric — GPT-4o via Chat Completions, no tools, no retrieval. Measures model memory; cannot cite. Grounded — Perplexity Sonar, search-backed, returns a structured source list. Only this arm can produce a citation.
Entity recall — the wreck is named, matching canonical name or any recorded alias. Brand mention — “Wreck Intelligence” appears in the answer text. Not a citation. Domain citation — wreckintelligence.com appears in the provider’s structured source list. Never inferred from prose. Where no source list exists the value is null, meaning unmeasurable — which is not zero.
Experiment Log
Site launched with 24 Irish wrecks. Sitemaps submitted to Google, Bing, and PrimeIndexer. Baseline benchmark run before any pages indexed. 295 prompts across both engines. Result: GPT-4o 61.0% wreck recall, Perplexity 51.3%. Citation was not validly measured in this round on either arm — see the methodology correction above.
Expanded to 64 Irish wrecks across 9 regions. All summaries editorially rewritten with varied prose. 342 sourced evidence records. Added taxonomy pages (countries, regions), 12 thematic collection pages, and internal linking by depth, type, difficulty, cause, and proximity. ~90 crawlable pages total.
Original 24 wreck pages confirmed indexed by Google. Re-benchmark run. Result: GPT-4o 61.0% wreck recall, unchanged — expected, since the parametric arm cannot see the index at all. Perplexity 55.2%, up 3.9 points. The conclusion recorded at the time, that zero citations proved the domain was too new, was not supported: citation was not being measured. Domain authority may well matter, but this benchmark has not yet tested it.
Added 47 UK wrecks across 13 regions — Scapa Flow German fleet, Sound of Mull, South Coast, Cornwall, Pembrokeshire, Farne Islands. 113 total wrecks, 466 evidence records, ~150 crawlable pages. IndexNow integration built — 153 URLs auto-submitted to Bing/Yandex. FAQ sections with FAQPage JSON-LD added to all wreck pages. v2 prompt set created (220 UK-focused prompts).
Re-benchmark after UK expansion and IndexNow submission. v1 (Ireland): Perplexity 53.0% wreck recall, within the 51–55% range seen so far. v2 (UK) baseline: Perplexity 46.8%. The gap between the sets is not safely attributable to LLM familiarity: the v1 scorer matched canonical names only, and the two sets use different naming conventions, so the metric itself differs between them. Citation remained unmeasured.
Benchmark instrumentation rebuilt to metric schema v2: structured source capture, entity recall against aliases, brand mention separated from domain citation, failed calls and dry runs recorded as such, model id and run index stored per run. Historical rows are retained, labelled schema v1, and their citation columns left null rather than reinterpreted. Two content experiments were built the same day — WI-001 (declarative openings on 20 of 113 wreck pages) and WI-002 (standalone prose on 6 country, region and collection pages) — so that once they are live, the next round measures both against an unchanged control group. Neither has been measured yet.