Do contextual links improve retrieval? Findings from Carrier R01

Research report · Carrier Retrieval Lab R01

Do contextual links
improve retrieval?

In this first controlled comparison, the tested graph blend produced a very small change in intended-target retrieval and no reliable gain over the selected lexical baseline.

BM25’s mean reciprocal rank at ten was 0.914778. Combining it with a walk over Carrier’s contextual graph produced 0.915184. The absolute paired difference was +0.000405, with a 95% percentile bootstrap interval of [0, 0.001157] and a two-sided sign-flip p-value of 0.49435. Both methods retrieved the intended concept at rank one for 214 of 240 held-out questions and within the first ten for 232.

The study therefore leaves the proposed retrieval benefit unconfirmed. Its finding concerns this snapshot, query set, representation and parameter-selection procedure. It supplies no basis for concluding that graphs never help, or that Carrier’s links have no value for reading, navigation or composition.

Scientific figures: Retrieval metrics Paired effects and intervals Graph degree distribution Study design

The question and the object being tested

Carrier contains compact definitions and contextual connections. A plausible hypothesis is that those connections can help a search system recover a concept when the question and definition use different words. R01 tests a particular implementation of that idea against lexical retrieval.

The frozen approved Wave III snapshot contains 6,257 checked definitions. Its stored relations yield 20,422 distinct undirected pairs. The resulting graph has 50 isolated nodes and 52 connected components; its largest component contains 6,204 nodes. No definitions or relations were added or edited during the experiment.

These edges provide context. They do not assert equivalence, entailment, causation, endorsement or permission. The retrieval calculation uses an undirected projection, while the definitions and their qualifications remain available for inspection. A graph path is not a proof of an answer.

A deliberately bounded benchmark

The target sample contains 120 concepts: twenty from each of six equal-width sequential-ID bands, selected with a fixed random seed. Eligible definitions have at least twelve whitespace-separated words. These bands distribute historical position; they are not subject-domain strata, and the length rule means the sample is not an unrestricted sample of all definitions.

Two AI authors each received half the target families. They wrote three questions per target: a paraphrase, an application and a boundary question. The full set has 360 questions. Forty families were reserved for development and eighty for the held-out comparison, yielding 240 held-out questions.

The authors knew the target names and definitions. They were not supplied graph edges, retrieval rankings, other vocabulary or explicit split labels. They also received numeric target IDs, historical bands and sequential family keys. Family order made the development assignment potentially inferable. The original inputs were preserved, and a methodological addendum corrects the initial stronger blinding description. There is no evidence recorded here that an author actually inferred the split.

This is a synthetic, definition-aware query set. An application-shaped question does not by itself establish independent transfer or real-world demand. The intended target is the concept from which the question was authored. Other returned concepts are unjudged, not automatically irrelevant. This distinction matters whenever relevance judgments are incomplete; Buckley and Voorhees examined that evaluation problem.

Development choices were kept separate from the test

The primary protocol was frozen locally before scoring. It was not externally registered or externally timestamped. Query and snapshot hashes, validation history, development scores and the parameter lock are retained in the experiment kit. The original primary comparison remained fixed while the dense and reader supplements were specified separately before held-out results.

Each retrieval document contains the name twice followed by the definition. Lexical processing uses lowercase ASCII alphanumeric tokens, a fixed stop list and no stemming. It does not index IDs, provenance, relation labels or whole source articles.

The lexical baselines were BM25 with k1 = 1.2 and b = 0.75, cosine TF-IDF, and reciprocal rank fusion with constant 60. RRF combined only positive-scoring lexical matches; arbitrary zero-match tail ranks could not become graph seeds. BM25 won the development comparison. Background for the method family is given by Robertson and Zaragoza’s account of BM25; the rank-fusion baseline follows the formula introduced by Cormack, Clarke and Büttcher, with the positive-candidate restriction stated explicitly here.

The graph method started from the twenty highest positive lexical scores, normalized as a seed distribution. It ran twenty walk iterations, returning dangling mass to that distribution. Development compared two teleport values and three nonzero blend weights. The selected setting used teleport 0.5 and a graph blend of 0.1, combining separately max-normalized lexical and graph scores. Positive scores were ranked, with checked ID breaking ties. With no lexical evidence, no seed was created.

The walk uses the personalized-PageRank idea of a source distribution and repeated returns to it. This implementation is a fixed small-graph calculation, not a replication of the bidirectional search algorithms studied by Lofgren, Banerjee and Goel. The six development candidates all contained a nonzero graph contribution: the procedure could not hide a negative graph result by selecting the unchanged baseline.

What the primary metric measures

For each question, reciprocal rank at ten is 1/r when the intended concept appears at rank r ≤ 10, and zero otherwise. The three question scores are averaged within each target family, then averaged over the eighty held-out families. Since each family has three questions, the overall mean also equals the query mean, but the resampling unit is the family.

The primary contrast is graph minus the development-selected lexical baseline. Its uncertainty uses 10,000 paired family bootstrap resamples and 10,000 two-sided family sign flips with a plus-one Monte Carlo correction. Sign flipping relies on a null exchangeability or symmetry assumption; this is not a randomized treatment experiment with human participants. The choice and interpretation of significance tests in retrieval deserve care, as discussed in Urbano, Lima and Hanjalic’s empirical study.

The interval is conditional on the observed synthetic families and selected configuration. It does not include the variability of another query author, development split or model-selection round. The frozen bootstrap resamples families without reproducing the ID-band stratification exactly. Secondary metrics, subgroups and supplements are exploratory and are not corrected for multiple comparisons.

The results, together

Only two families improved on the primary graph-versus-BM25 metric; none declined and 78 tied. One paraphrase moved its intended target from rank 9 to rank 8. One boundary question about topological space moved from rank 4 to rank 3. Neither change added a top-ten hit or a rank-one result.

The percentile interval’s lower bound is exactly zero. With two positive family differences and 78 zeros, resampling the observed differences cannot produce a negative one. The nonnegative interval is therefore not a guarantee that the method cannot hurt retrieval. The paired sign-flip result and the very sparse changes provide no reliable evidence of a benefit.

The application questions were more difficult for these methods: both BM25 and its graph blend found the intended concept in the first ten for 74 of 80 application questions, compared with 79 of 80 paraphrases and 79 of 80 boundary questions. This breakdown is descriptive; it does not independently validate any question category as a measure of understanding.

Two misses show what the average can conceal

The stopping-time question describes an analyst ending observation when a condition becomes detectable from data already seen, without consulting later observations. The intended definition concerns exactly that restriction on deciding whether stopping has occurred. Yet the intended concept ranked 424th under BM25 and 425th after the graph blend.

The oscillation question describes a quantity rising and falling around a middle level, with gradually shrinking swings, before a physical cause is known. Its intended concept moved from 48th to 46th. It was one hop from a top-twenty lexical seed, but that proximity was insufficient to bring it into the first ten.

Both are intended-target misses, not complete relevance judgments on every returned item. They show why a high aggregate score should not replace inspection of individual needs. The technical report and experiment kit preserve all eight misses and the underlying rankings. The questions and graph were not revised in response to these outcomes.

The controls do not rescue the hypothesis

An empty-edge ablation reproduced the lexical result. Ranking by degree alone produced no intended-target top-ten hits. Five randomized graphs preserved each node’s degree while swapping edges; their mean MRR@10 was 0.914917, with individual results from 0.914778 to 0.915126. The real graph’s result was only slightly higher at 0.915184.

Every rewired graph used the same locked parameters chosen for the real graph. They were not separately optimized. The recorded swap counts and graph diagnostics make the controls inspectable, but five draws do not characterize a complete graph-null distribution or establish statistical significance over graph structures.

The stored-direction ablation matched the main graph’s MRR@10. Stored direction can reflect publication structure and the order in which records were added. That exploratory result should not be described as a finding about the usefulness of semantic direction.

One dense model, reported as a supplement

The exploratory dense baseline used all-MiniLM-L6-v2, with its repository revision and files pinned before encoding. It encoded the same repeated name and definition text through mean pooling and normalized-vector cosine similarity. The configured 256-wordpiece maximum truncated none of the documents or questions. Encoding ran locally.

Its MRR@10 was 0.814563, an absolute difference of −0.100215 from BM25. The exploratory paired interval for that difference was [−0.161726, −0.041412]. This compact pretrained model performed worse on this benchmark; the comparison does not represent all dense retrievers. Training-data overlap was not audited, and definition-conditioned queries may favour lexical overlap. The experiment does not separate those possibilities.

Adding the same graph settings raised dense MRR@10 to 0.818413, a change of +0.003849. The exploratory interval was [−0.003889, 0.012698], with p = 0.35636. Rank-one hits increased from 176 to 177, while top-ten hits fell from 227 to 226. The graph settings were borrowed from lexical development, not optimized for dense retrieval. These mixed outcomes do not establish a reliable dense-graph benefit.

The reader pilot reached a ceiling

A separate exploratory exercise selected sixteen held-out application questions by a fixed hash-based rule that did not use retrieval success. Each question appeared with the top five definitions from BM25 and with the top five from the graph method. Scores, method labels and intended targets were withheld, and the resulting 32 contexts were shuffled.

One AI reader, who had not authored the queries, selected a concept, wrote a concise application answer and supplied an exact quote supporting a qualification. The intended target was available in all sixteen contexts from each method. The reader selected it in all sixteen from each method, and every quote matched its supplied definition exactly. No comparative benefit was demonstrated.

These were sixteen paired questions, not 32 independent questions. Repetition could create carry-over. The reader belonged to the same project and model family and had previous methodological and publication context; masking per-item methods did not provide external independence. Selecting the intended ID and matching a quotation are mechanical checks. They do not establish expert entailment of the whole answer, source truth, human understanding or independent semantic replication.

What can responsibly follow

The result supports retaining a strong lexical baseline and treating this graph blend as an inspectable experimental option. It does not justify a general claim that contextual links improve retrieval. The dense comparison and reader pilot add observations with their own limits; neither changes the primary conclusion.

A stronger next study would collect information needs before revealing intended definitions, include independent relevance judgments for plausible alternatives, and reserve fresh questions for evaluation. Graph designs, dense models and combinations could then be compared under a new frozen plan. Those are proposals for later work, not improvements demonstrated by R01.

This experiment also leaves the earlier USL1 message-recovery results in their own scope. R01 did not perform a new AI-to-AI transport experiment. Successful retrieval, correct source quotation, intact encoding and independent interpretation remain different things to test.

Records, sources and reuse

The experiment adapts the approved Carrier snapshot for retrieval by forming text documents and an undirected contextual graph. It preserves the source dictionary. Wikipedia-derived definitions retain their attributed CC BY-SA 4.0 terms; existing Carrier material is not retroactively relicensed. The optional model carries its own Apache 2.0 license. Relevant attribution and model notices accompany the reproducibility package.

The original protocol should be read with the preserved methodological addendum and the two supplement plans in the experiment kit. The results data and technical report keep the evidence and limits available together.

Original protocol Frozen questions Results JSON Technical report Experiment kit Optional model package