News / Retrieval Lab R01
News · Carrier Retrieval Lab R01
The graph was tested.
The gain remains unconfirmed.
Carrier’s first retrieval study found no reliable improvement from its tested graph blend over the selected lexical baseline. The complete comparison is prepared for publication in a new Retrieval Lab.
The experiment used a frozen snapshot of 6,257 checked definitions and 20,422 undirected contextual pairs. Two AI authors produced three questions for each of 120 intended concepts. Forty concept families were used for development; the remaining 80 families supplied 240 held-out questions.
A very small observed change
BM25 achieved mean reciprocal rank at ten of 0.914778; the locked graph blend achieved 0.915184. The paired change was +0.000405, with a 95% bootstrap interval of [0, 0.001157] and a two-sided sign-flip p-value of 0.49435. This result does not establish a reliable gain, and it does not show that contextual graphs are useless in other tasks or configurations.
The additional comparisons
An exploratory dense comparator using all-MiniLM-L6-v2 scored 0.814563, below BM25 on this benchmark. Adding the same graph settings raised it to 0.818413, but the graph-effect interval included zero. One compact model on definition-aware synthetic questions cannot settle the relative merits of dense and lexical retrieval in general.
In a separate exploratory evidence-use exercise, one AI reader chose the intended concept in 16 of 16 contexts from each method, and every supporting quote matched a supplied definition exactly. Both methods already supplied the intended concept in all those contexts. There was no demonstrated comparative reader benefit; repeated questions and the reader’s shared project context limit independence.
What is being published
The lab includes the frozen questions and results, a technical report, controls and error cases, and a reproducibility kit. The primary protocol was frozen locally before scoring, with later supplements identified separately. It was not externally preregistered. Query authors saw target definitions and identifying metadata, so their blinding was limited.
The next research step is a fresh evaluation using more varied needs and independent relevance judgments. The current result stays visible: the tested graph contribution made almost no difference to intended-target retrieval.