A questions-first experiment / R02

Questions first.
Answers on trial.

Can qualified relations help a retrieval system find useful context—and help a reader give a better answer? Start with real information needs. Freeze the questions. Let both claims face the evidence.

48information needs
6domains in the protocol
12 / 36development / final split
2co-primary comparisons

01 / Results

Two claims.
The same scrutiny.

The main contrasts compare the development-selected typed method with the development-selected baseline. The selected comparison remains fixed even when another method scores better on the final questions.

Evaluation outputs are pending. This page does not substitute example numbers for measured results.
Retrieval · typed − baselinePendingFinal-query pooled nDCG@5
Answers · typed − baselinePendingNormalized answer utility and grounding
Relevance judge agreementPendingExact agreement; model-generated judgments
Typed relation coveragePendingShare of corpus concepts touched by the overlay

Positive differences favor typed retrieval on the stated endpoint. Both co-primary tests require the declared joint Holm adjustment. A confidence interval or a small p-value does not establish human usefulness or general performance.

Measured comparisons

Metrics across methods

Method comparison chart; numerical results are also available in the table below.

Figure pending.

Within-question differences

Paired effects and uncertainty

Paired retrieval and answer effects with uncertainty intervals; endpoint estimates are shown above.

Figure pending.

Method scores, controls, and the selection lock
Final-query metrics from the recorded evaluation
MethodPooled nDCG@5Grade ≥ 2 precision@5Agreed obligation coverage@5
Evaluation pending.

The pool is incomplete. Its ideal ranking is not the ideal over the full corpus; an unjudged item is not a confirmed negative.

Parameter lock pending.

Full recorded results and agreement

02 / Question explorer

The need comes
before the match.

Browse all 48 frozen tasks, one at a time. Original source wording, adapted prompts, conditions, and answer obligations stay distinct.

Two origins, clearly labeled. Public questions are human source material; their concise benchmark framing is an AI adaptation. Scenario tasks are prospectively AI-authored. The collection is purposive, not a population sample of users.
0 loaded

Task collection pending

The frozen task collection will appear here when it is included in this build.

Useful answer obligations

    Conditions to preserve

      Source wording, attribution, and adaptation

      Compare the recorded top five. On narrow screens, scroll horizontally to keep the ranks side by side.

      Selected baseline

      Pending

        Selected typed method

        Pending

          Rankings are recorded experiment outputs. Scores are meaningful within their own method. Relevance grades are AI assessments of the supplied definitions, not certificates of truth. Filtering stays in your browser; no question text is uploaded.

          03 / Qualified relation viewer

          Every arrow
          has conditions.

          Inspect a directed path or a single typed relation with its literal evidence and full qualifications. A two-edge route does not establish a new relation between its endpoints.

          Navigation, not entailment. This research overlay is not a promoted Carrier ontology. Direction comes from the source definition; stored contextual links do not supply semantic direction. Missing edges mean abstention, not negation.

          Qualified path data are pending.

          Overlay coverage and interpretation

          Where the overlay reaches

          Typed relation coverage and counts by predicate; full statistics appear below.

          Figure pending.

          04 / Randomized reader test

          A better ranking
          is only the start.

          Read the recorded answers and their assessments against the task's prewritten obligations. Insufficient context and unsupported claims remain part of the record.

          Two reader processes, not 36 independent people. Each reader receives each final question once. Context assignments are complementary and balanced within domains; method labels are hidden during collection. Labels shown here are a retrospective disclosure.

          What this question asks the answer to preserve
          Reader answers pending.
          Answer scoring, assignments, and full records

          Two separate graders score obligation coverage, role and condition fidelity, and evidence discipline. The normalized mean grader score is the answer endpoint. Unsupported claims, false citations, and qualified abstention are recorded separately.

          Only supplied definitions may support the answers. A polished answer can still be unhelpful or insufficiently grounded. The raw responses are preserved before grading.

          Each reader process handled 36 items sequentially. It never saw the same question twice, but information can carry over between different questions within that process. When both methods supply identical definitions for the current question, an answer difference cannot be attributed to changed current-question definitions; it may reflect reader or scoring variation, or carryover from earlier assigned contexts. Separate contexts also share a model family; this is not an independent human reader study.

          The retrieval sign-flip test assumes exchangeable paired signs. Unlike the answer-allocation test, it is not justified by randomized treatment assignment. Neither procedure removes the limits of purposive tasks, shared model errors, or cross-question carryover.

          The scorer initially rejected the obligation indices from both graders because they used 1–4 instead of 0–3. After both graders confirmed the order, only the indices were mechanically corrected. Original files were retained; scores and rationales did not change, and no regrading occurred.

          Answer component diagnostics pending.

          05 / Method and limits

          A narrow test.
          An inspectable record.

          The procedure is locally prospective, not externally preregistered. The constraints on the claim belong beside every result.

          01 / Independent preparation

          Questions and overlay come separately

          Two isolated author contexts collect information needs before seeing Carrier concepts or ranks. A separate definition-based process builds the typed overlay without questions. Shared model family and possible pretraining exposure remain limitations.

          02 / Selection

          Develop, lock, then evaluate

          Within each domain and origin, one task develops the method and three enter final evaluation. Baseline and typed settings are selected on development pooled nDCG@5. Controls cannot replace the primary method after final outcomes.

          03 / Evidence

          An incomplete, judged pool

          Every compared top five is judged by two isolated AI assessors. Their ordinal grades are averaged under the declared rule. Pooling does not establish exhaustive recall, and unjudged records are not confirmed negatives.

          04 / Interpretation

          Both outcomes must be reported

          Retrieval and answer quality are co-primary contrasts with joint multiplicity control. Query-level uncertainty is conditional on the tasks, labels, corpus, and implementation. It does not turn AI assessments into human validation.

          Full protocol and explicit limitations

          What the study cannot establish

          • General usefulness for a representative population of real users.
          • Human agreement, expert-validated relation precision, or universal answer truth.
          • Independence of model family or freedom from public-source pretraining contamination.
          • Completeness of the typed overlay or logical entailment along its paths.
          • Unbiased relevance to future systems outside this method-pooled evaluation.

          The orchestrator cannot be fully blinded. Source questions are real public questions; adaptations, prospective scenarios, relevance labels, readers, and graders are AI-mediated. Retrieval context contains definitions rather than full source documents. No live Carrier records are edited by this experiment.

          Procedure at a glance

          From independent preparation to two final comparisons

          Study workflow: independent task and overlay preparation, frozen pools, development selection, parameter lock, final retrieval and reader evaluation.

          Study-design figure pending.

          Methodological references