A questions-first experiment / R02
Questions first.
Answers on trial.
Can qualified relations help a retrieval system find useful context—and help a reader give a better answer? Start with real information needs. Freeze the questions. Let both claims face the evidence.
01 / Results
Two claims.
The same scrutiny.
The main contrasts compare the development-selected typed method with the development-selected baseline. The selected comparison remains fixed even when another method scores better on the final questions.
Positive differences favor typed retrieval on the stated endpoint. Both co-primary tests require the declared joint Holm adjustment. A confidence interval or a small p-value does not establish human usefulness or general performance.
Metrics across methods
Figure pending.
Paired effects and uncertainty
Figure pending.
Method scores, controls, and the selection lock
| Method | Pooled nDCG@5 | Grade ≥ 2 precision@5 | Agreed obligation coverage@5 |
|---|---|---|---|
| Evaluation pending. | |||
The pool is incomplete. Its ideal ranking is not the ideal over the full corpus; an unjudged item is not a confirmed negative.
Parameter lock pending.
Full recorded results and agreement
02 / Question explorer
The need comes
before the match.
Browse all 48 frozen tasks, one at a time. Original source wording, adapted prompts, conditions, and answer obligations stay distinct.
Task collection pending
Link to this task ↗The frozen task collection will appear here when it is included in this build.
Useful answer obligations
Conditions to preserve
Source wording, attribution, and adaptation
Compare the recorded top five. On narrow screens, scroll horizontally to keep the ranks side by side.
Selected baseline
PendingSelected typed method
PendingRankings are recorded experiment outputs. Scores are meaningful within their own method. Relevance grades are AI assessments of the supplied definitions, not certificates of truth. Filtering stays in your browser; no question text is uploaded.
03 / Qualified relation viewer
Every arrow
has conditions.
Inspect a directed path or a single typed relation with its literal evidence and full qualifications. A two-edge route does not establish a new relation between its endpoints.
Qualified path data are pending.
04 / Randomized reader test
A better ranking
is only the start.
Read the recorded answers and their assessments against the task's prewritten obligations. Insufficient context and unsupported claims remain part of the record.
What this question asks the answer to preserve
Answer scoring, assignments, and full records
Two separate graders score obligation coverage, role and condition fidelity, and evidence discipline. The normalized mean grader score is the answer endpoint. Unsupported claims, false citations, and qualified abstention are recorded separately.
Only supplied definitions may support the answers. A polished answer can still be unhelpful or insufficiently grounded. The raw responses are preserved before grading.
Each reader process handled 36 items sequentially. It never saw the same question twice, but information can carry over between different questions within that process. When both methods supply identical definitions for the current question, an answer difference cannot be attributed to changed current-question definitions; it may reflect reader or scoring variation, or carryover from earlier assigned contexts. Separate contexts also share a model family; this is not an independent human reader study.
The retrieval sign-flip test assumes exchangeable paired signs. Unlike the answer-allocation test, it is not justified by randomized treatment assignment. Neither procedure removes the limits of purposive tasks, shared model errors, or cross-question carryover.
The scorer initially rejected the obligation indices from both graders because they used 1–4 instead of 0–3. After both graders confirmed the order, only the indices were mechanically corrected. Original files were retained; scores and rationales did not change, and no regrading occurred.
Answer component diagnostics pending.
05 / Method and limits
A narrow test.
An inspectable record.
The procedure is locally prospective, not externally preregistered. The constraints on the claim belong beside every result.
Questions and overlay come separately
Two isolated author contexts collect information needs before seeing Carrier concepts or ranks. A separate definition-based process builds the typed overlay without questions. Shared model family and possible pretraining exposure remain limitations.
Develop, lock, then evaluate
Within each domain and origin, one task develops the method and three enter final evaluation. Baseline and typed settings are selected on development pooled nDCG@5. Controls cannot replace the primary method after final outcomes.
An incomplete, judged pool
Every compared top five is judged by two isolated AI assessors. Their ordinal grades are averaged under the declared rule. Pooling does not establish exhaustive recall, and unjudged records are not confirmed negatives.
Both outcomes must be reported
Retrieval and answer quality are co-primary contrasts with joint multiplicity control. Query-level uncertainty is conditional on the tasks, labels, corpus, and implementation. It does not turn AI assessments into human validation.
Full protocol and explicit limitations
What the study cannot establish
- General usefulness for a representative population of real users.
- Human agreement, expert-validated relation precision, or universal answer truth.
- Independence of model family or freedom from public-source pretraining contamination.
- Completeness of the typed overlay or logical entailment along its paths.
- Unbiased relevance to future systems outside this method-pooled evaluation.
The orchestrator cannot be fully blinded. Source questions are real public questions; adaptations, prospective scenarios, relevance labels, readers, and graders are AI-mediated. Retrieval context contains definitions rather than full source documents. No live Carrier records are edited by this experiment.
From independent preparation to two final comparisons
Study-design figure pending.
Methodological references
06 / Downloads
Take the evidence
with you.
The report, source attribution, frozen inputs, and raw assessments should travel with the conclusion. Download the complete record or take masked contexts to two fresh readers.
Technical report
Methods, both endpoints, controls, uncertainty, and limitations.
PendingZIP / ReproductionExperiment kit
Frozen inputs, scientific code, recorded outputs, and reproduction instructions.
PendingZIP / Fresh readerReader challenge
Prepared materials for a new reader. Consult the included instructions and masking limits.
PendingJSON / Information needsEvery task and its source
Original wording, attribution, adaptations, obligations, and conditions.
PendingJSON / Recorded outcomesFull results
All retrieval methods, uncertainty, controls, and per-question results.
PendingJSON / Frozen planProtocol
Sampling, comparison rules, randomization, statistics, and claim boundaries.
Pending