Can connections improve retrieval?

Abstract

Carrier R01 tests whether a simple use of Carrier’s contextual graph improves retrieval of intended concepts beyond text search. The corpus contains 6,257 checked records and 20,422 undirected contextual pairs. Two AI authors produced three queries for each of 120 sampled target concepts without being supplied the graph or retrieval outcomes. Forty families selected settings; eighty families supplied 240 held-out queries.

The selected BM25 baseline achieved MRR@10 0.914778; its graph-assisted variant achieved 0.915184. The paired difference was 0.000405, with a family-bootstrap 95% interval [0.000000, 0.001157] and two-sided sign-flip p=0.4944. Both found the intended concept in the top ten for 232/240 queries and at rank one for 214/240. This study did not demonstrate a reliable graph advantage.

An exploratory local MiniLM comparison scored lower overall on this definition-derived benchmark. A masked evidence-use exercise selected the intended concept and quoted a supplied qualification in all sixteen cases under each method; it also showed no comparative benefit. These are bounded synthetic retrieval and evidence-use results, not exhaustive relevance judgments or evidence of general human comprehension.

1. Question, estimand and prospective record

The primary estimand is the difference in mean reciprocal rank, truncated at ten, between the development-selected nonzero graph setting and the development-selected lexical baseline. “Semantic retrieval” names the investigation; it does not certify that a ranking score represents understanding. Application and boundary queries probe whether an intended definition is recovered under a changed expression of the need.

The protocol, benchmark and implementation hashes were frozen locally before held-out scoring. This was not an externally registered or externally timestamped trial. A pre-execution methods addendum records limited author blinding. Two defensive repairs and an empty-edge-control failure were corrected before the first test run, with initial evidence preserved. Development scores were unchanged by those repairs.

No relation, target or query was edited in response to held-out results. The implementation refuses to overwrite the first test rankings; reproduction takes place in a separate folder. Dense and reader supplements were specified before held-out outcomes and remain exploratory.

2. Corpus and graph

The frozen checked head is 398492317d139bd78ca599965f08603296daedd95f4fb088fde05393175bfe88. The exact checked JSONL is included, with its chain and file hash. It is the locally approved Wave III release; this experiment does not independently authenticate live publication or the truth of definitions. Three working records are excluded.

The primary graph is an undirected projection of contextual links. Stored direction can reflect record storage or publication order; it is not necessarily an authored semantic predicate. An edge does not imply equivalence, entailment, causation, endorsement or permission. Text retrieval indexes only names and definitions, excluding provenance, sources, IDs and relations.

3. Sampling, authorship and intended targets

Seed 1272026 selected twenty eligible concepts from each of six equal-width sequential-ID bands. Definitions needed at least twelve whitespace-separated words: 6,236 records were eligible and 21 short definitions were excluded. This distributes historical position; it is not subject-balanced or representative of user traffic. There was no exclusion by degree or retrieval success.

Two AI author processes each received sixty target names and definitions. Each wrote a paraphrase, an application question and a question about a qualification or boundary. Full-title phrases and IDs were excluded from queries; duplication and structure were checked. All 360 queries, rationales and intended IDs are published.

Authors also saw IDs, band labels and ordered family keys. Explicit split labels were withheld, but order made the split potentially inferable. The authors were not fully blinded and were not independent human judges. Queries are target-informed, definition-derived and share model/project context; this can favour lexical recoverability and introduces a ceiling effect.

There are 40 development families (120 queries) and 80 held-out families (240 queries). All three variants of a concept remain in the same split. The assigned intended target is one known relevant item, not an exhaustive relevance set. A different retrieved concept may be valid; it has not been adjudicated. The reported metrics must not be interpreted as precision over all valid answers.

4. Ranking systems

BM25: k₁=1.2, b=0.75, positive Robertson-style IDF ln(1+(N−df+0.5)/(df+0.5)); repeated query terms do not multiply their contribution. The name is repeated twice, then the definition is appended. Tokens are lowercase ASCII alphanumerics of length at least two, with a fixed published stop list and no stemming.

TF–IDF: term weight (1+ln tf)·(ln((N+1)/(df+1))+1), cosine normalized. RRF: BM25 and TF–IDF positive candidate ranks fused as 1/(60+rank). Zero-score tails do not become artificial candidates.

Graph: the twenty highest positive lexical scores seed a probability distribution s. Starting p=s, twenty iterations apply p←αs+(1−α)Pᵀp, returning dangling mass to s. The final score is (1−λ)·maxnorm(text)+λ·maxnorm(p). With no lexical evidence, no graph seed is invented. The development grid used α∈{0.5,0.8}, λ∈{0.1,0.25,0.5}; ties prefer lower λ then higher α. The selected setting is α=0.5, λ=0.1 with BM25.

Controls: five degree-preserving edge-swap graphs use exactly these settings. A degree-only ranking, an empty-edge ablation and a stored-direction walk are also recorded. Ten times the edge count swap attempts per control is an operational budget, not proof of uniform mixing. Control seeds, acceptance counts, retained-edge overlap and components are included. The five graphs are descriptive controls, not five independent test collections.

Only positive scores are ranked; exact ties use ascending checked ID. Nonfinite scores fail closed. An intended item outside the positive ranking has no rank and contributes zero.

5. Metrics and statistical analysis

The primary query score is 1/r for intended rank r≤10, and zero otherwise. Three scores are averaged per concept family, then the eighty family means are averaged. Secondary metrics are intended-target hit rates at 1,5,10,20, untruncated reciprocal rank, and 1/log₂(r+1) within ten. The latter is a single-target discounted rank, not a complete graded nDCG assessment.

Ten thousand paired bootstrap resamples and ten thousand paired sign flips operate on families, with seed127300. The sign-flip p-value is two-sided with a plus-one correction. It relies on exchangeability/symmetry under the null; this was not a randomized intervention on people. Bootstrap intervals are conditional on the sampled synthetic families, chosen split and settings. They do not capture query-author variation, parameter-selection uncertainty, or a population of real users. The empirical family bootstrap does not recreate the original stratified sampling design.

There is one primary contrast. All subgroup, control, dense and reader analyses are exploratory or descriptive. Secondary intervals are not multiplicity-corrected. A small p-value elsewhere cannot substitute for the primary result, and a nonsignificant result does not establish equivalence.

6. Held-out results

The primary improvement comes from two families; 78 are unchanged in family-mean MRR@10 and none declines on that truncated metric. Other ranks outside ten do move in both directions. “No losses” on this metric is not a general non-degradation guarantee. Hit rates at all reported cutoffs remain unchanged between BM25 and its graph variant.

The real-graph advantage over the mean rewired control is also small and uncertain. The similar controls do not support a strong attribution of improved retrieval to the particular authored links in this setup. An empty graph reproduces the lexical ranks as required.

7. Query types

The median overlap is 41.2% of a query’s distinct non-stopword tokens present in its intended title or definition. This diagnostic uses the frozen tokenizer; it is not a semantic similarity measurement. The high lexical baseline and target-informed authorship limit headroom. Future tests should collect information needs before exposing writers to the target vocabulary.

8. Every primary change and every lexical top-ten miss

These tables are rule-selected: every nonzero primary query-score change, followed by every BM25 top-ten miss. They are not a hand-picked set of successes.

The application probes expose a useful distinction between strong average recovery and particular failures. A search method can identify many definitions while still missing a concept expressed through a different task. The tables retain those failures without treating unjudged alternatives as automatically false.

9. Local dense-retrieval supplement

The official all-MiniLM-L6-v2 model was run locally with ONNX; no Carrier text was submitted to a remote inference API. Revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41, file hashes and model card are retained. Attention-mask mean pooling and L2 normalization produce 384-dimensional vectors, with at most 256 wordpieces. The representation uses the same repeated name and definition. Documents truncated: 0; queries truncated: 0.

Dense cosine MRR@10 was 0.814563. Borrowing the already-frozen graph setting produced 0.818413; its paired graph increment has a 95% interval [-0.003889, 0.012698]. No dense-specific tuning was performed. This model was weaker overall here, while individual misses can differ. It does not follow that all dense retrieval is inferior, or that graph augmentation is generally ineffective.

This is one pretrained model. Training overlap with Wikipedia, conventional terminology and the query-author model is unknown. The normalized vectors are included so rankings can be recalculated without re-running a model. Recomputing vectors requires the pinned model and local inference libraries; the model is included in the experiment kit and also offered as a separate download; numerical details can vary by runtime/hardware.

10. Exploratory evidence-use pilot

Sixteen held-out application families were selected by a fixed hash rule before inspecting success. Top-five contexts from each method produced 32 shuffled items with method labels and intended targets withheld. One AI reader that did not author queries wrote an interpretation and supplied a qualification quote. Its first completed response was retained before scoring.

The intended concept was available in all 16 contexts per method; the reader selected it in all 16 under each method, with exact quotation matches throughout. This tests evidence use within favourable supplied contexts and provides no observed comparative gain. A quotation match establishes correspondence with supplied text, not logical entailment, external truth or human understanding.

The reader retained earlier project/method context. Repeated questions could create recognition or carry-over, and algorithm identities were withheld operationally rather than cryptographically concealed. There was no second independent reader or human adjudication. Full inputs, mapping key, responses and mechanical assessment remain available for challenge.

11. Verification, failures and reproducibility

The independent scalar/toy suite checks BM25, cosine, positive ranking, ties, random-walk mass, dangling nodes, rewiring degrees, family weighting and zero/negative effects. All24 checks pass after repair. The first audit’s two empty-edge failures are retained; the ablation was repaired rather than dropped. Defensive nonfinite and single-edge cases are also preserved. A separate results auditor recalculates scores and uncertainty from recorded ranks without importing the original metric functions.

The JavaScript live-search implementation is checked against Python for all 240 held-out queries across BM25, TF–IDF, RRF and the graph:960 method-query comparisons. The live page performs local lexical/graph search; it does not silently run the dense model or report new queries as benchmark observations. The page has no automatic external data calls.

Download the experiment kit, verify its inventory, then run python tools/reproduce.py in a Python environment with NumPy. Reproduction writes to a separate folder and compares ranks and metrics with the retained first run. Dense score reproduction uses supplied vectors; full vector recomputation uses the optional model archive and documented inference dependencies. All figures have SVG, PNG and PDF versions. UI verification is code/DOM/parity checking, not a live-browser or physical-device claim.

12. What this result means

The tested contextual graph did not reliably improve intended-concept retrieval over a strong lexical baseline on this synthetic benchmark. It also did not improve the observed outcome of the small evidence-use exercise. Those findings answer a specific question about a specific representation and ranking rule.

They do not show that the relations are meaningless. They do show that a visible network and a plausible walk algorithm are not sufficient evidence of retrieval benefit. Carrier’s definitions were often recoverable directly. A relation designed to preserve context may be useful to inspect without being a useful numerical boost to a neighbouring result.

The next experiment should be more independent and more task-shaped: real information needs written before target exposure; pooled relevance judgments including alternative answers and hard negatives; multi-concept questions requiring explicit roles and conditions; relation-type-aware paths with direction and qualifications; and a randomized reader task that measures answers, not merely target position. New tuning should use a new development split and reserve a fresh final holdout. The current test set is now public and cannot remain a secret final benchmark.

A larger dictionary has become a measurable system. The obligation now is to let evidence determine how its connections are used.

References and rights

• Manning, Raghavan & Schütze — Information retrieval system evaluation
• Manning, Raghavan & Schütze — Evaluation of ranked retrieval results
• Manning, Raghavan & Schütze — Assessing relevance
• Robertson & Zaragoza — The Probabilistic Relevance Framework: BM25 and Beyond
• Lofgren — Personalized PageRank Estimation and Search
• Urbano, Lima & Hanjalic — Statistical Significance Testing in Information Retrieval

Carrier definitions retain their recorded provenance and source terms. Wikipedia-derived adaptations retain CC BY-SA4.0 attribution. The MiniLM model card declares Apache2.0; the pinned model card and notices accompany the optional model archive. This study does not relicense third-party material or authenticate authorship through hashes alone.