Does the source support the answer?

The question comes first.

Does checking the link between a claim and a specific source passage reduce unsupported answers — and at what cost?

Why it matters

A citation can look credible even when its source addresses a different situation, supports only part of a sentence or is outdated. In document-based work, this distinction affects human decisions.

What we propose to compare.

The first pilot will use public or synthetic documents. Tasks will include answerable questions, insufficient evidence, conflicting documents and information that changes over time. Private case files and Navigator conversations are outside this proposed dataset.

The sample size, final tasks and analysis plan will be fixed before the pilot. This draft does not prescribe a target improvement.

A proposed path from task to evidence.

1. 1

   Finalise the protocol

   Set tasks, scoring criteria, model and source versions, budgets and comparison rules before assessing outcomes.

2. 2

   Prepare the materials

   Document provenance, licences, expected evidence and cases in which the appropriate outcome is to leave a question unresolved.

3. 3

   Compare the variants

   Run the same tasks across variants. Record differences in information access, number of calls, latency and resource use.

4. 4

   Human assessment

   Assess significant claims against a shared rubric, using separate ratings and a documented disagreement process. Conceal the variant where feasible.

5. 5

   Analyse and share

   Report the size and uncertainty of observed differences, costs and error types. Share reproducibility materials within licensing constraints.

What we will measure.

The following dimensions form the proposed assessment. Scoring rules and denominators must be finalised in the protocol so the results can be interpreted.

Unsupported claims

The proportion of assessed significant claims without sufficient support in the available material.

Evidence alignment

Whether the cited passage supports the content, scope and context of the specific claim.

Missed contradictions

Relevant contradictions or outdated information missed despite being identified in the task specification.

Appropriate abstention

Separately: justified withholding of an answer and unnecessary refusal when the material supported an answer.

Time and resources

Latency and resource use per task and per correctly supported outcome.

Rating agreement

Agreement between human assessors and cases requiring adjudication.

What this study will not establish.

• Support from a document does not establish that the document is true or sufficient in the real world.
• Outcomes depend on tasks, language, model version and source access. They do not automatically generalise to other domains.
• Human assessment can also be ambiguous. The report must describe disagreements and limitations of the rubric.
• A pilot does not establish the safety of an entire platform and is not AI system certification.

Data, roles and accountability.

Public or synthetic materials with checked usage rights. Provenance, version and retrieval date will be part of the dataset documentation. User data is not the default source of research material.

Research lead and assessors

To be appointed before the study starts. No research team is announced in this draft.

Review

Reviewer and scope to be defined. This draft has not undergone independent scientific review.

Funding and relationships

The study budget, funding sources and relevant relationships have not yet been specified in this protocol. They will be disclosed before the pilot begins.

Materials and corrections

Planned materials will include versioned methods, usage rights and a record of changes. Protected data will not be included in public reproducibility files.

Planned outputs. Results still ahead.

1. Protocol and scoring rubric
2. Dataset description and source versions
3. Comparative results and error examples
4. Limitations, reproducibility materials and correction history