When a useful relation does not improve retrieval: Carrier R02 findings

Carrier · Research findings · R02

A useful connection
must serve the question.

A relation supported by the source text can still move a search result in the wrong direction. Carrier R02 tests that possibility by starting with questions, preserving their conditions, and evaluating both retrieved context and the answers built from it.

The primary retrieval result is modest and clear. The selected dense baseline scored 0.674080 on final pooled nDCG@5; the selected typed method scored 0.670678. The paired difference was -0.003402, with a 95% interval of [-0.010205, 0.000000] and a raw two-sided sign-flip p-value of 1.0000. The tested typed contribution did not establish a retrieval advantage. That result stays in view while we examine the design, the separate answer test, and the cases that explain the average.

Begin with something a person needs to know

In the first retrieval study, authors wrote questions with intended concepts already visible. That tested whether a system could recover those targets. R02 reverses the order. Two fresh author contexts were instructed to avoid Carrier concepts, previous study outputs, target lists, and accepted answers while collecting needs and writing answer obligations. Only after the tasks were frozen did retrieval supply possible context.

The collection contains 48 tasks across mathematics, physics, statistics, philosophy, biology, and economics. Half are adapted from actual public human questions on Stack Exchange. Their benchmark wording and answer obligations are AI-authored adaptations, not verbatim statements by the original people. The other half are original AI-authored decision scenarios. Source wording, authors, URLs, retrieval dates, and reported licenses remain separate from the adaptations.

This is a purposive collection of conceptual questions, not a random sample of users or search traffic. The separation between preparation stages is procedural: instructed contexts sharing a model family, not external human independence or a security boundary. One early search exposed unsolicited answer excerpts for candidates excluded from the final set; the collection log retains that disclosure. Public source text may also have appeared in model pretraining.

Within each domain and origin, one of four tasks went to development and three to final evaluation. That produced 12 development tasks and 36 final tasks. Roles, conditions, and three to five useful-answer obligations were fixed before candidate exposure. A definition earns relevance by helping with those obligations, rather than by matching a preassigned answer label.

Keep the verb and its qualifications together

The underlying corpus is unchanged: 6,257 checked definitions. Its stored contextual links are not automatically semantic assertions. A source-only process, working without the questions, extracted a separate overlay from explicit definition language. It contains 146 assertions touching 236 concepts, or 3.77% of the corpus.

Each assertion retains its predicate, semantic direction, literal evidence, full source and target definitions, and scope. A statement that one property depends on temperature is not rewritten as a claim that an entity cannot exist without temperature. A contrast is not logical contradiction. Support is weaker than enabling, and a classifying phrase is not a certified proof of formal subsumption.

The overlay deliberately leaves some relation types empty when the definitions do not justify them. Missing edges mean abstention, not evidence that no relation exists. Every displayed path keeps its component assertions and their qualifications; following two edges does not create a new entailed relation between the endpoints. These limits also distinguish structural verification from expert validation of meaning.

Choose the comparison before seeing its final score

Three baselines competed on the development tasks: BM25 lexical matching, a pinned local MiniLM embedding model, and reciprocal-rank fusion of the two. Dense retrieval won narrowly, with development nDCG@5 of 0.748424 against 0.746056 for fusion. It therefore became the locked baseline.

The typed method starts from the top 20 positive baseline results and follows qualified directed paths. Development selected one hop and a 0.1 blend. When no typed path is reachable, it returns the baseline exactly. The overlay’s coverage and this conservative setting matter: a reachable path occurred on 15 of the 36 final tasks, but the ordered top five changed on only two.

Final scores do not reopen the selection. Fusion scored 0.740736 on the final tasks, higher than the selected dense baseline. That is an informative descriptive result, not permission to replace the baseline after inspection. Selecting whichever method looks best on the final set would spend the evidence intended for evaluation on another round of development.

Judge alternatives without inventing a complete gold standard

The pool combined the top five from every fixed candidate setting, plus specified lexical alternatives and a seeded random candidate for each task. Across 57 settings, this produced 666 distinct task–candidate pairs. Two isolated AI assessors graded every pair, for 1,332 relevance judgments. They saw shuffled definitions and opaque local codes, with method, rank, split, corpus ID, and graph degree withheld.

The scale distinguishes irrelevant material, related background, help with an explicit obligation, and direct support for a central distinction. The declared score is the arithmetic mean of the two ordinal grades. Assessors also recorded obligation coverage and a reason. Exact agreement was 81.38%, with quadratic weighted κ of 0.9259. Agreement measures consistency between these assessors; shared assumptions can still make them consistently wrong.

The primary metric, nDCG@5, rewards placing higher-graded material near the top. Its ideal ranking comes from the incomplete judged pool. Every compared top-five item is judged, but unseen corpus items are not confirmed negatives. The result is therefore pooled relevance under this procedure, not exhaustive recall, universal answer truth, or an unbiased benchmark for every future retrieval system.

Two changed lists explain more than one rounded number

Task A21 asks about a randomized website experiment with several outcomes and daily checking. A useful response must distinguish repeated looks from multiple outcomes, separate statistical evidence from practical effect size, and preserve an interpretable error guarantee. The baseline places “multiple comparisons problem” first and “experiment” second. The typed method swaps them.

Both assessors rated the specific multiple-comparisons definition 3 and the broad experiment definition 1. Promoting the broader entry lowers this task’s nDCG@5 from approximately 0.4710 to 0.3486. The relation leading toward a general experiment concept does not need to be false for that promotion to be unhelpful. The missing step is a reason to prioritize it for this particular question.

Task B18 concerns a cloudy lake: reduce nutrient inflow, remove a fish, or combine interventions while separating treatment effects from seasonal variation. The typed method replaces “environmental monitoring” with “food web” in fifth place. Both entries receive grade 3, so nDCG does not change. Yet agreed obligation coverage falls from three quarters to one half.

A scalar relevance grade can treat two entries alike while their contributions differ. One helps describe an ecological mechanism; another helps address how to observe change. A context window is small. Adding relevant material can still remove something the answer needs. These two cases were observed after the final evaluation; they explain its behavior, rather than providing a new rule tuned on the holdout.

The answer has to face its own test

Retrieval quality and useful answering are separate endpoints. Two reader processes each received every final question once, with either the baseline or typed top-five context. Their assignments were complementary and balanced three per method within each domain. Method labels and relevance judgments were hidden, and definitions retained their retrieval order.

Readers could use only the supplied definitions. They had to cite opaque evidence labels, preserve conditions, and state when the context was insufficient. Two separate graders then assessed each answer against its prewritten obligations, role and condition fidelity, and evidence discipline. The 72 first responses were preserved before the 144 answer assessments. Unsupported claims and false citations remain visible separately from the normalized score.

The frozen scorer initially rejected both grading files because their obligation numbers ran from 1 to 4 rather than 0 to 3. After both graders confirmed the order, a mechanical subtraction of one corrected only those indices. The original files are retained; scores and rationales were unchanged, and no regrading occurred.

The separate answer endpoint also does not establish an advantage for the typed-context assignment. The baseline-context mean was 0.908565 and the typed-context mean was 0.922454 on the normalized 0–1 rubric. The paired difference was +0.013889, with a 95% paired interval of [-0.003472, 0.031279], a blocked-randomization p-value of 0.1411, and a Holm-adjusted p-value of 0.2822.

The answer randomization test preserves the actual within-domain allocation balance. Its role is distinct from the descriptive paired bootstrap interval. Both were specified before the response scores were compared, and the answer result is reported alongside the retrieval result rather than chosen as a substitute.

The rubric also has a ceiling limit. Mean obligation scores were 0.8628 for baseline and 0.8854 for typed answers. Role and condition fidelity scored 1.0000 and 0.9931; evidence discipline scored 1.0000 in both groups. These last two nearly saturated components account for one-third of this six-component composite. Neither grader flagged an unsupported claim or false citation. Those are model judgments, not independent evidence that such errors were absent.

In the two changed-context cases, the answer differences were +0.083333 for A21 and +0.166667 for B18. Both typed contexts were assigned to reader A. Two cases cannot separate a graph contribution from the particular reader realization. The 34 same-context questions had a mean difference of +0.007353. These are descriptive diagnostics, not new confirmatory tests.

Only two final questions received changed context, while 34 received identical ordered definitions under both methods. Answer differences on these 34 tasks cannot be attributed to changed definitions for the current question. They may reflect reader or scoring variation, or carryover from earlier assigned contexts. Complementary randomization reduces confounding, but two reader processes do not become 36 independent human participants. Each process handled its 36 items sequentially, allowing carryover between different questions despite never seeing the same question twice. A careful partial answer may be well grounded while still failing to resolve the user’s decision.

Report the awkward secondary results too

The primary retrieval contrast has no wins, one loss, and 35 ties. With only one nonzero paired difference, flipping its sign preserves the absolute mean difference; the two-sided sign-flip p-value is therefore 1. This does not prove exact equivalence. It describes how little changed under this method and task set.

The selected untyped comparator improved the mean by +0.011112. Its percentile bootstrap interval, [0.000292, 0.024054], was positive, while its raw sign-flip p-value was 0.0743, above 0.05. Both belong in the report. They come from different inferential procedures and should not be reduced to whichever offers the more attractive headline. This comparison is exploratory and does not isolate typing: graph topology and the traversal algorithm also change.

Reversing asymmetric edges, permuting relation labels, and removing the typed graph supply further controls. Type weights cancel when a node has only one outgoing edge, so label permutation tests relative branching effects, not every possible value of meaningful predicates. Counterfactual control edges are numerical interventions, never promoted to truthful assertions.

The two co-primary comparisons use the declared Holm adjustment. The retrieval sign-flip test assumes exchangeable paired signs; unlike the answer-allocation test, it is not justified by randomized treatment assignment. Query-level paired intervals and sensitivity analyses express uncertainty within this experiment. They do not absorb every uncertainty about task sampling, shared model errors, relation extraction, or future users. Neither a secondary improvement nor a well-written answer can substitute for the primary retrieval finding.

Improve the decision rule before enlarging the graph

The next useful step is to audit displaced evidence and disputed judgments with people who understand the domains. Then ask whether a proposed retrieval change can distinguish a general neighbor from a definition needed for a particular obligation. Coverage-aware selection, stronger baselines, and better treatment of uncertainty are candidates for a new study, not improvements demonstrated here.

More edges would change coverage, but also create more opportunities to promote a plausible distraction. Any expansion should preserve literal evidence, direction, modality, and sense distinctions. If methods change after this analysis, they need fresh evaluation tasks. The now-public final set can support inspection and diagnosis; it cannot remain an unseen future holdout.

The lab, report, and full kit make that next investigation possible. They expose the source adaptations, frozen plan, selected settings, every judged candidate, both contexts, qualified paths, and recorded answers. Reproducing the calculations from archived judgments is different from obtaining the same judgments from a new model. The kit includes the pinned model and assets for local inference; no remote inference API is required.

Carrier’s checked records and their existing author credits remain intact. The new research layer and this publication add a testable claim with a visible boundary: the selected typed method did not establish better retrieval here. A connection should keep its meaning, and a retrieval system should still have to explain why that connection deserves the reader’s limited attention.

Technical reportFrozen protocolTasks and attributionRetrieval resultsAnswer resultsFull experiment kitReader challenge

Scientific figures: MetricsPaired effectsRelation coverageStudy design