Carrier / Retrieval Lab R01
Carrier Retrieval Lab · Study R01
Find a concept.
Inspect the evidence.
Compare text search with a contribution from Carrier’s contextual graph, then see what happened when both were tested on the same questions.
The lab brings search, definitions and an inspectable experiment together. The primary test asked whether a small graph contribution improves retrieval of an intended concept beyond the best lexical method selected on development questions.
The measured result: no reliable graph gain. BM25 and BM25 plus graph both placed the intended concept first for 214 of 240 held-out questions and in the first ten for 232. The graph’s mean reciprocal-rank change was +0.000405, with a 95% paired bootstrap interval of [0, 0.001157] and a two-sided sign-flip p-value of 0.49435.
What to explore
The public search comparison uses lexical and graph methods. The learned dense model appears as a frozen experimental comparator, not a model running in the browser. In this test, its mean reciprocal rank was lower than BM25’s; that finding concerns one model and this particular benchmark.
A separate exploratory reader exercise selected the intended concept in 16 of 16 contexts from each method, with exact supporting quotes throughout. It showed no comparative benefit. The reader belonged to the same project, and repeated questions could carry information between items.
These are synthetic questions written with knowledge of the intended definitions. The benchmark measures retrieval of those intended targets; it does not exhaustively judge every returned concept or establish human comprehension. Contextual links remain connections to inspect, not assertions of entailment.
Original protocol Frozen questions Results JSON Technical report Experiment kit Optional model package
The study uses the approved Wave III snapshot without adding or changing definitions. Wikipedia-derived material retains its source attribution and CC BY-SA 4.0 terms; earlier Carrier material and the optional model retain their own terms. The experiment kit carries the relevant notices.