1. Results before interpretation
The typed layer did not establish a joint retrieval-and-answer advantage.
The study compares a new question-first benchmark against a frozen Carrier snapshot. The retrieval endpoint changes by -0.0034 pooled nDCG@5 (95% paired bootstrap [-0.0102, 0.0000]); the answer endpoint changes by 0.0139 on a 0–1 rubric (95% interval [-0.0035, 0.0313]). These estimates concern this particular task sample and model judgments.
A confidence interval or p-value does not establish semantic truth, domain expertise, or performance for future users. A null or negative effect is retained. The numerical endpoints were specified before judgments; neither was substituted after seeing the result.
Only one paired retrieval difference is nonzero. Flipping its sign leaves the absolute mean unchanged, which explains p=1 despite the small negative point estimate. The 35 ties remain in the declared analysis.
2. What changed from R01
R01 used questions written from assigned target concepts. R02 reverses that sequence: information needs and answer obligations are written first, without exposing the authors to Carrier. Candidate concepts are then retrieved and pooled. Multiple alternatives can receive credit. The answer trial evaluates what a reader can actually explain using retrieved evidence.
The corpus remains the same 6,257 checked concepts. This isolates a methodological change from dictionary growth, but the tasks, relevance definition and endpoint differ from R01. Their absolute scores should not be plotted as a simple performance trend.
Retrieval and answer endpoints. Exported from the recorded outcomes.
Paired effect estimates and uncertainty. Exported from the recorded outcomes.
Typed relation coverage and changed retrieval contexts. Exported from the recorded outcomes.
Need-first experimental design. Exported from the recorded outcomes.3. The questions and their provenance
Two separate author contexts collected 24 actual public questions from Stack Exchange question-only sources and wrote 24 additional scenario tasks. The public questions were adapted into explicit information needs; the adaptations and answer obligations are AI-authored, not verbatim statements by the original people. Original text, authors, URLs, dates and API-reported licenses are retained separately. The source selection is purposive and conceptual, not a random sample of search traffic.
There are eight tasks in each of mathematics, physics, statistics, philosophy, biology and economics: four public-origin and four scenario tasks. Within each domain/origin cell, a seeded shuffle assigns one task to development and three to final evaluation: 12 development and 36 final questions. Every final question retains its role and condition requirements. No R01 query or target is reused.
The collectors were instructed not to read Carrier, previous outcomes or accepted answers. One initial search returned unsolicited answer excerpts for excluded candidates; the collection log preserves that exposure. The selected questions were obtained from question-only endpoints. Public material may nevertheless occur in model pretraining. Neither process isolation nor a fresh context establishes independent human authorship or training-data blindness.
4. What counts as a relation
The original 20,787 stored links remain untyped contextual connections. Publication direction is not silently converted into semantic direction. A separate, source-only model curator reviewed explicit definition language without access to the new questions. The resulting research overlay contains 146 directed or symmetric assertions touching 236 concepts: 3.77% of the corpus.
Each assertion carries literal evidence offsets, source and target definitions, a reviewed facet, modality, qualifications and hashes. Classification does not establish formal subsumption; contrast does not mean contradiction; dependence does not mean necessary causation; support is weaker than enabling. Fifty typed pairs are newly extracted overlay connections, while 96 already have a contextual link. No checked record is changed.
Ambiguous senses, negation and unqualified generalization were rejected. For example, a definition saying enzymes are usually proteins cannot justify an unconditional enzyme-is-a-protein assertion. The reserved predicates requires, constrains and evidence-for remain empty when their conditions are not established. This is a sparse, model-curated layer, not a complete or expert-validated ontology.
5. Retrieval methods and qualification boundaries
Three non-graph baselines are compared: BM25 over repeated name plus definition; cosine similarity from a pinned local all-MiniLM-L6-v2 model; and reciprocal-rank fusion of those two. BM25 uses k₁=1.2 and b=.75. The fusion constant is 60. The development winner is dense.
The selected typed setting is typed:dense:1:0.1. Starting from the top 20 positive baseline scores, the method follows one or two qualified directed edges, normalizes outgoing predicate weights, discounts the second hop by .5 and blends the resulting signal at .1 or .3. Explicit query cues modify relative type weights. Cyclic revisits are excluded. With no reachable typed path, the baseline is returned exactly.
Paths are retrieval suggestions, not proofs: a two-edge path never creates an entailed third assertion. Full qualifications remain available in the overlay and public path viewer. The reader receives the retrieved definitions, not a claimed logical conclusion manufactured from the path.
Type weights cancel at a node with only one outgoing edge or identical outgoing weights. A type-label control therefore tests relative branching effects, not the value of naming a predicate in every situation. The untyped comparator uses a different graph and walk; its contrast changes topology and algorithm as well as typing.
6. Pooled relevance and difficult alternatives
Before judging, the union of every fixed setting’s top five results was collected, with lexical ranks six and ten and one seeded random candidate per question. Across 57 settings, this produced 666 unique task–candidate pairs. The two assessors saw task-local opaque codes and independently shuffled definitions. Method, rank, split, checked ID and graph degree were withheld.
Each assessor graded every pair: 0 irrelevant, 1 related background, 2 useful for an explicit obligation, or 3 directly supporting a central distinction. They also recorded which obligations the definition helps and a short reason. The declared combination is the arithmetic mean of the two ordinal grades, used as a soft relevance score; no later outcome-driven adjudication occurs.
Exact agreement was 81.4%; quadratic weighted κ was 0.9259. Agreement is not proof of correctness when assessors can share model biases. Individual-assessor sensitivity and every disagreement are retained. There are 221 pooled top-five appearances, counted as task–candidate pairs, that both assessors rate no higher than background; these are inspectable near-matches, not declarations of universal irrelevance.
nDCG@5 uses gain 2^g−1 and an ideal ranking over the judged pool only. Every compared top-five item is judged. Unjudged corpus records are not known negatives, and this study cannot estimate exhaustive recall. Precision at grade≥2 and agreed obligation coverage are secondary descriptions.
7. Development and final separation
Task text, obligations, overlay, scoring code and candidate pools were frozen before relevance judgments. Both first judgment files were retained. Selection scripts use only the 12 development tasks; the strongest baseline, typed hop/blend setting and untyped blend are then pinned in a parameter lock. Final metrics are computed only after that lock, and the final runner refuses overwrite.
The parent orchestrator has project continuity and filesystem access; this is not a security-enforced blind or external preregistration. Assessors and readers operate in separate instructed contexts. Final labels necessarily exist before pool evaluation, but final aggregate label summaries and endpoint scores are withheld until selection. Local hashes document sequence and consistency, not a trusted external timestamp.
8. Randomized answer trial
All 36 final questions are given to two separate reader contexts. Each reader encounters each question once. One receives the selected baseline’s top five definitions and the other the selected typed method’s top five, with complementary assignment balanced three versus three inside every domain. Method names and relevance judgments are hidden. Retrieval order is retained because ordering is part of the tested intervention.
Readers must answer the prewritten obligations, preserve roles and conditions, cite supplied labels, and state insufficiency. Explicit deductions and labeled hypothetical examples are allowed; unsupported empirical additions are not. First responses are frozen before scoring.
Two new graders score every answer independently against the question, obligations, context and response. Each obligation receives 0–2; role/condition fidelity and evidence discipline each receive 0–2. The normalized mean of both graders is the endpoint. A well-justified evidence gap earns partial substantive credit, while full credit requires a supported answer. Faithful abstention may preserve evidence discipline without completing the requested explanation.
This measures answer utility under supplied evidence, not merely exact quotation matching. It still relies on model judgments rather than independent expert fact-checking. Graders can encounter repeated questions in shuffled order, which may introduce comparison carry-over. Readers never receive a repeated question, but each answers 36 different items sequentially in one context. Previously seen concepts may carry over; current-item citations constrain the evidence without proving cognitive isolation.
Neither grader flagged an unsupported claim or false citation in these responses. That is an assessment outcome, not independently established error absence. Evidence discipline is at the rubric ceiling, and role/condition scores are nearly saturated. These easy-to-saturate components contribute one-third of the composite; the component scores should accompany the high overall means.
9. Statistical analysis
The two co-primary effects are mean final-query nDCG@5 difference and mean answer-rubric difference. A Holm correction controls the declared .05 familywise threshold across the two p-values. Retrieval uses a two-sided paired sign-flip test; the answer test samples allocations that preserve the actual three/three balance within all six domains, under a sharp no-effect null for the observed reader scores.
The retrieval sign-flip calculation assumes exchangeable paired signs under its null; retrieval method assignment itself is not randomized. Both endpoints include 10,000-replicate paired percentile bootstrap intervals; stratified-by-domain resampling is reported as a sensitivity analysis. The unit is the question, not an individual relevance label. Two reader contexts do not become 36 independent human participants. The randomization interpretation assumes no material cross-question interference; sequential reader contexts may weaken that assumption. Questions remain purposively selected and can share domain assumptions; the intervals do not represent uncertainty over every future user or every language model.
Subgroups, controls, individual-judge comparisons and diagnostic examples are secondary. They do not replace either primary result. An interval excluding zero in an exploratory comparison should not be reported as an independently confirmed discovery.
10. Controls and where the intervention acts
The selected typed layer reaches at least one path on 15/36 final queries. It changes the ordered top-five context on 2/36; the other 34 questions receive identical contexts under both methods. Answer differences on those identical-context questions cannot be attributed to changed current-item definitions. They may reflect reader variation, grading variation or cross-question carryover from earlier assigned contexts.
The controls reverse asymmetric edges, permute type labels while keeping endpoints, remove all typed edges, or use the full untyped contextual graph. Reversed and permuted relations are deliberately counterfactual numerical controls; they are never displayed as truthful semantic assertions.
11. Findings that can be inspected
The public explorer joins each question to its source/adaptation record, both retrieval contexts, graded alternatives, paths and reader responses. A reader can inspect whether a change brought in a useful distinction, displaced a necessary qualification, or simply moved an already adequate definition.
In A21, a question about repeated testing and multiple outcomes, a valid classification path promotes the general concept experiment above the more directly useful multiple comparisons problem. The definitions remain the same, but their order gets less useful under the declared relevance grades. A textually supported relationship does not guarantee a useful ranking intervention.
In B18, a lake-restoration task, food web replaces environmental monitoring. Both have a mean relevance grade of 3, so pooled nDCG does not change. Yet the agreed obligation coverage falls: a mechanism-related concept displaces context useful to the monitoring requirement. This is why an equally relevant alternative need not serve the same role.
Only one final question changes top-five membership; the other change is order alone. The two-reader trial is therefore dominated by 34 identical-context pairs. It provides useful evidence about answer construction and variability, but little exposure to substantively different retrieval content.
Differences in nDCG and differences in answer quality need not agree. A relevant extra definition may be redundant; a short definition can supply a crucial boundary; and a capable reader may still lack evidence for the requested mechanism. The declared obligations make these gaps visible without assuming that any one dictionary entry is the unique correct answer.
Dense retrieval narrowly wins development selection, but hybrid fusion has the best observed final baseline score. That is a descriptive finding, not permission to change the locked comparator. A 12-question development sample can select a method that does not remain best on new questions.
The untyped graph’s secondary improvement over the selected baseline is 0.0111. Its exploratory sign-flip p-value is 0.0743; its percentile bootstrap interval is [0.0003, 0.0241]. These procedures need not agree at a threshold in a small, discrete sample. Report both; do not promote the favorable interval into a confirmed graph benefit.
The two changed cases have higher typed-assignment answer scores: +0.0833 in A21 and +0.1667 in B18. Both typed contexts happened to be assigned to reader A. These two observations cannot isolate ordering or concept substitution from reader realization. The other 34, identical-current-context pairs average +0.0074. Those differences can include grading variation and cross-question carryover. None of these post-lock descriptions establishes a separate subgroup benefit.
12. Implementation checks and preserved failures
Independent synthetic tests check scalar retrieval calculations, directed and symmetric traversal, type-weight branches, two-hop limits, cycles, empty-graph identity, pooling coverage, gains, agreement, multiplicity and blocked assignment. Initial malformed-input checks found missing validation in the agreement statistic and acceptance of Boolean answer grades as integers. Both were repaired before the corresponding data freeze; first failures remain in the checks folder.
Scientific outputs are recomputed in a fresh directory using the archived questions, judgments and answers. Reproduction of those calculations is distinct from obtaining the same judgments from a new model. The downloadable audits state their exact scope. Source hashes, figure data, archive members and public links are checked; no physical-device or live-hosting guarantee is implied.
The first answer-scoring run rejected both assessor files because they numbered obligations 1–4 while the frozen evaluator expects 0–3. Both assessors separately confirmed the ordinal correspondence. Their first files and hashes are retained; a logged mechanical transformation subtracts one only from those index labels. An inverse comparison confirms that scores, reasons, flags and all other fields are unchanged. The evaluator and its checks were not weakened. Final scoring occurred only after this format correction.
The independent auditor also initially demanded zero-based indices in the raw reader response summaries, although the reader instructions did not specify an index base. That extra assertion was corrected to accept a complete ordinal sequence; the raw reader responses stayed unchanged. Both failed attempts, their precise scopes and the final passing audits are preserved. The final independent audit reproduces both endpoints and their uncertainty, and a separate audit verifies that grader normalization changed only index labels.
13. What this experiment does not settle
- It does not independently establish the truth or completeness of Carrier definitions.
- It does not provide expert human relevance or answer judgments.
- It does not isolate all aspects of relation semantics from topology, coverage and algorithm choice.
- It does not test a modern embedding model suite, unrestricted web research or a production conversation.
- It does not validate arbitrary path composition or prove meaning transfer solely from retrieval scores.
- Its public final tasks are now disclosed and cannot serve as a secret future holdout.
The practical next step is a human review of disagreement cases and evidence gaps, followed by a new held-out task set if algorithms or definitions change. Expanding the typed layer should preserve the same evidence and qualification discipline; a larger asserted graph is not automatically a better one.
14. Reproduce and extend
Download the complete experiment kit, read START-HERE.txt, validate the inventory and run the documented scripts. Core scoring requires Python and NumPy. Optional fresh embedding inference uses the pinned local model with ONNX Runtime and tokenizers. No remote inference API is required.
The reader challenge carries masked contexts and instructions without the grading key. Published questions and answers are already available elsewhere; a new reader must disclose prior exposure. Preserve new responses before scoring, and report their identity and tool access rather than claiming automatic independence.
15. Research sources and attribution
Pooling and ranked-retrieval evaluation follow established information-retrieval practice. Separate answer grading is motivated by the distinction between retrieved relevance and useful supported responses; model-judge and context-position biases remain relevant limitations.
Public task sources retain their individual attribution and API-reported CC BY-SA license versions in the benchmark. Carrier records preserve their existing provenance and terms. See license and attribution. The MiniLM model retains its repository license and pinned revision.
16. Public-question source ledger