News / Retrieval Lab R02
News · Carrier Retrieval Lab R02
The questions came first.
The retrieval gain did not.
Carrier’s second retrieval study found no demonstrated benefit from its selected typed-relation method on the primary retrieval endpoint. The experiment also tests a separate question: whether the retrieved definitions help an AI reader produce a more useful, grounded answer.
R02 starts with information needs rather than intended target concepts. Two fresh, instructed author contexts collected 24 public human questions and adapted them into benchmark tasks, then wrote 24 additional scenarios. The authors prepared answer obligations and conditions before seeing Carrier. The six domains are mathematics, physics, statistics, philosophy, biology, and economics.
Twelve tasks selected the baseline and parameters; 36 were reserved for final evaluation. A separate definition-based process prepared 146 qualified typed assertions touching 236 of the frozen corpus’s 6,257 concepts. The overlay is a research artifact, not a change to Carrier’s checked records.
The primary retrieval finding
The paired change was -0.003402, with a 95% paired bootstrap interval of [-0.010205, 0.000000]. The two-sided sign-flip p-value was 1.0000; the jointly adjusted retrieval p-value was 1.0000. There were no improvements, one decline, and 35 ties on this metric. The finding neither establishes a benefit nor proves that every typed graph is ineffective.
The two changed lists show why inspection matters. On a website-experiment task, a generic experiment definition moved above the more relevant multiple-comparisons concept. On a lake-restoration task, a food-web definition replaced environmental monitoring. The second change preserved the relevance score but reduced coverage of the task’s answer obligations.
The separate answer test
The separate answer endpoint also does not establish an advantage for the typed-context assignment.
The baseline-context mean was 0.908565 and the typed-context mean was 0.922454 on the normalized 0–1 rubric. The paired difference was +0.013889, with a 95% paired interval of [-0.003472, 0.031279], a blocked-randomization p-value of 0.1411, and a Holm-adjusted p-value of 0.2822.
Both endpoints were specified in advance, with joint Holm adjustment. Two reader processes are not 36 independent human participants. On 34 questions the retrieved contexts were identical. Answer differences there cannot be attributed to changed definitions for the current question; they may reflect reader or scoring variation, or carryover from earlier assigned contexts.
What is open for inspection
The lab exposes all tasks, original source wording and attribution, both rankings, qualified paths, recorded answers, scores, controls, and limitations. The complete kit includes local model assets and reproduction instructions. No remote inference API is required.
Across 666 pooled task–candidate pairs, the two AI relevance assessors agreed exactly 81.38% of the time, with quadratic weighted κ of 0.9259. This is strong measured agreement within the exercise, not independent proof of correctness. Shared model errors, purposive sampling, and an incomplete relevance pool limit the claim.