R03 · PUBLIC-BENCHMARK EXPLORATION · POOL RESTRICTED
One fixed obligation-aware reranker reduced semantic redundancy but lowered pooled nDCG. Its small coverage gain across all 48 tasks did not persist in the original final36 subset. This result does not support replacing dense retrieval.
The unchanged R02 benchmark contains 6,257 corpus records, 48 tasks, 192 prewritten requirements and 666 judged task–document pairs. This experiment selects five documents from each existing judged pool. It adds no holdout, questions, judgments or corpus content. All effects below are exploratory; no significance tests were run.
Representation. Keep all 6257 R02 document vectors and all 48 question vectors unchanged. Encode each raw prewritten requirements string individually using the pinned local MiniLM ONNX model; no rewriting, task labels, grades, names, or generated requirement expansions. R02 mean attention-mask pooling, L2 normalization, max 256 wordpieces, batch 16, CPU threads 4/1.
Facets. Question followed by its m original requirements; every facet has weight 1/(m+1). Thus explicit requirements jointly have m/(m+1) weight.
Normalizer. For each facet, take positive cosine max(0, dot(facet,document)) for every member of the complete same task pool, then divide by its maximum over that complete pool. An all-zero facet remains zero. This normalizes similarity scale without relevance grades. Never normalize on selected documents.
Objective. F(S) = [sum_f max_{d in S} normalized_cosine(f,d) - sum_{unordered {d,e} subset S} max(0,dot(d,e))/C(5,2)]/(m+1), with max over empty S = 0.
Selection. Start empty. For exactly five steps choose the as-yet unselected pool member giving the largest F(S union {d}) - F(S). Exact score ties choose smaller checked ID. Retain selection order as ranking; no post-selection reranking.
Rationale. Equal facet weighting avoids inventing importance labels and grants most weight to the explicit requirements. Max coverage gives no reward for repeating a facet already covered equally well. The pair penalty has maximum total cost of one facet's weight at the context budget, making document duplication an explicit cost without introducing a tuned coefficient. These are design choices, not estimated optimal values.
Method, programs, source files, requirement vectors, complete score inputs and all label-free rankings were hashed before qrels content was opened. Preparation enforced a file-open barrier on judgment/outcome filenames. This records local ordering, not an external preregistration or a newly blinded public benchmark. The original R02 typed setting was already chosen using R02 development labels and is only a fixed historical comparator here.
Exact method · Pre-outcome freeze · Score inputs · Selections · All results · Reproduction instructions · Verification
| Selection | nDCG@5 | Agreed coverage@5 | Label redundancy | Mean positive pair cosine | Docs with no agreed obligation | Unique obligations lost / gained vs dense |
|---|---|---|---|---|---|---|
| Dense | 0.6927 | 0.7396 | 0.3734 | 0.3936 | 1.9167 | 0 / 0 |
| R02 typed | 0.6894 | 0.7344 | 0.3853 | 0.3936 | 1.9167 | 1 / 0 |
| Fixed coverage | 0.6048 | 0.7656 | 0.2688 | 0.1839 | 2.5417 | 12 / 17 |
| Labeled oracle — diagnostic | 0.9978 | 0.8854 | 0.5300 | 0.3524 | 0.4792 | 0 / 28 |
Coverage uses all prewritten requirements as denominator. Label redundancy is repeated agreed document–obligation assignments divided by total assignments; it is zero when there are no assignments. A lower redundancy value can therefore reflect missing evidence. “Lost / gained” sums task-specific distinct obligations, counted once per task.
| Exploratory paired effect | nDCG difference [95% bootstrap interval] | Coverage difference [95% bootstrap interval] |
|---|---|---|
| coverage-minus-dense | -0.0878 [-0.1407, -0.0354] | +0.0260 [-0.0365, +0.0938] |
| coverage-minus-typed | -0.0846 [-0.1387, -0.0310] | +0.0312 [-0.0312, +0.0990] |
| typed-minus-dense | -0.0033 [-0.0091, +0.0000] | -0.0052 [-0.0156, +0.0000] |
10,000 paired task bootstrap draws, seed 129300. Percentile intervals describe this selected, model-judged benchmark; they are not confirmatory tests. “Original final36” names the old R02 split, not a fresh R03 holdout.
| Selection | nDCG@5 | Agreed coverage@5 | Label redundancy | Mean positive pair cosine | Docs with no agreed obligation | Unique obligations lost / gained vs dense |
|---|---|---|---|---|---|---|
| Dense | 0.6741 | 0.7361 | 0.3485 | 0.3873 | 2.0000 | 0 / 0 |
| R02 typed | 0.6707 | 0.7292 | 0.3624 | 0.3887 | 2.0000 | 1 / 0 |
| Fixed coverage | 0.5931 | 0.7292 | 0.2800 | 0.1846 | 2.5556 | 10 / 9 |
| Labeled oracle — diagnostic | 0.9971 | 0.8750 | 0.5185 | 0.3517 | 0.5278 | 0 / 20 |
Coverage uses all prewritten requirements as denominator. Label redundancy is repeated agreed document–obligation assignments divided by total assignments; it is zero when there are no assignments. A lower redundancy value can therefore reflect missing evidence. “Lost / gained” sums task-specific distinct obligations, counted once per task.
| Exploratory paired effect | nDCG difference [95% bootstrap interval] | Coverage difference [95% bootstrap interval] |
|---|---|---|
| coverage-minus-dense | -0.0810 [-0.1372, -0.0253] | -0.0069 [-0.0625, +0.0486] |
| coverage-minus-typed | -0.0776 [-0.1350, -0.0207] | +0.0000 [-0.0557, +0.0556] |
| typed-minus-dense | -0.0034 [-0.0102, +0.0000] | -0.0069 [-0.0208, +0.0000] |
10,000 paired task bootstrap draws, seed 129300. Percentile intervals describe this selected, model-judged benchmark; they are not confirmatory tests. “Original final36” names the old R02 split, not a fresh R03 holdout.
| Selection | nDCG@5 | Agreed coverage@5 | Label redundancy | Mean positive pair cosine | Docs with no agreed obligation | Unique obligations lost / gained vs dense |
|---|---|---|---|---|---|---|
| Dense | 0.7484 | 0.7500 | 0.4480 | 0.4125 | 1.6667 | 0 / 0 |
| R02 typed | 0.7456 | 0.7500 | 0.4540 | 0.4081 | 1.6667 | 0 / 0 |
| Fixed coverage | 0.6401 | 0.8750 | 0.2353 | 0.1817 | 2.5000 | 2 / 8 |
| Labeled oracle — diagnostic | 1.0000 | 0.9167 | 0.5646 | 0.3547 | 0.3333 | 0 / 8 |
Coverage uses all prewritten requirements as denominator. Label redundancy is repeated agreed document–obligation assignments divided by total assignments; it is zero when there are no assignments. A lower redundancy value can therefore reflect missing evidence. “Lost / gained” sums task-specific distinct obligations, counted once per task.
| Exploratory paired effect | nDCG difference [95% bootstrap interval] | Coverage difference [95% bootstrap interval] |
|---|---|---|
| coverage-minus-dense | -0.1084 [-0.2307, +0.0229] | +0.1250 [-0.0417, +0.3333] |
| coverage-minus-typed | -0.1055 [-0.2279, +0.0245] | +0.1250 [-0.0417, +0.3333] |
| typed-minus-dense | -0.0028 [-0.0085, +0.0000] | +0.0000 [+0.0000, +0.0000] |
10,000 paired task bootstrap draws, seed 129300. Percentile intervals describe this selected, model-judged benchmark; they are not confirmatory tests. “Original final36” names the old R02 split, not a fresh R03 holdout.
All 121,036 five-document subsets were enumerated across the 48 pools. Agreed-label coverage reaches 0.8854; 22 of 192 obligations have no agreed supporting document anywhere in their pool. Five slots can represent every pool-attainable agreed obligation on every task, so no additional coverage is lost solely to this budget.
The oracle maximizes agreed coverage using labels, then breaks ties by label-derived nDCG and orders documents by mean grade. Its near-perfect nDCG is consequently built into its construction. It is a diagnostic ceiling, never a baseline, proposed retrieval method, or answer-quality result. Agreement means both judges labeled the same requirement on the same document.
You are reviewing a team's randomized website experiment. They compare two designs on three outcome measures, inspect results every day, and want to stop as soon as any conventional significance threshold is crossed. Explain how repeated looks and multiple outcomes differ, separate statistical evidence from practical effect size, and recommend a decision procedure that preserves an interpretable error guarantee.
The new policy puts sequential statistical analysis first, restores the sequential-stopping requirement, and retains multiple comparisons. Coverage rises from 2/4 to 3/4, but the effect-size requirement remains missed even though the pool contains agreed evidence for it. Two selected documents are grade 0: the binding of Isaac and Buddhist two-truths doctrine. This case illustrates both useful requirement targeting and irrelevant diversity.
| Rank / checked ID | Document | Mean grade | Agreed requirements (one-based) | New requirements at rank |
|---|---|---|---|---|
| 1 / 3449 | multiple comparisons problem | 3 | 1, 4 | 1, 4 |
| 2 / 3472 | experiment | 1 | None | None |
| 3 / 3475 | randomized controlled trial | 1 | None | None |
| 4 / 3478 | quasi-experiment | 0 | None | None |
| 5 / 4699 | experimental blocking | 1 | None | None |
| Rank / checked ID | Document | Mean grade | Agreed requirements (one-based) | New requirements at rank |
|---|---|---|---|---|
| 1 / 3472 | experiment | 1 | None | None |
| 2 / 3449 | multiple comparisons problem | 3 | 1, 4 | 1, 4 |
| 3 / 3475 | randomized controlled trial | 1 | None | None |
| 4 / 3478 | quasi-experiment | 0 | None | None |
| 5 / 4699 | experimental blocking | 1 | None | None |
| Rank / checked ID | Document | Mean grade | Agreed requirements (one-based) | New requirements at rank |
|---|---|---|---|---|
| 1 / 4684 | sequential statistical analysis | 3 | 1, 2, 4 | 1, 2, 4 |
| 2 / 3475 | randomized controlled trial | 1 | None | None |
| 3 / 3449 | multiple comparisons problem | 3 | 1, 4 | None |
| 4 / 147 | the binding of Isaac | 0 | None | None |
| 5 / 4288 | Buddhist two-truths doctrine | 0 | None | None |
| Rank / checked ID | Document | Mean grade | Agreed requirements (one-based) | New requirements at rank |
|---|---|---|---|---|
| 1 / 3429 | statistical decision errors | 3 | 4 | 4 |
| 2 / 3449 | multiple comparisons problem | 3 | 1, 4 | 1 |
| 3 / 4684 | sequential statistical analysis | 3 | 1, 2, 4 | 2 |
| 4 / 3428 | statistical power | 2.5 | 3, 4 | 3 |
| 5 / 3426 | statistical hypothesis test | 2 | 4 | None |
147: the binding of Isaac — grade 0, requirements []
The narrative of Abraham commanded to offer his son, brought to the point of sacrifice, and then stopped, with a ram supplied instead. It concentrates promise, obedience, danger, and substitution in a scene whose moral and theological implications remain contested.
2750: eventual consistency — grade 0, requirements []
A replication guarantee that, if further updates stop and the required communication and recovery conditions hold, replicas converge to a common resolved state. It permits temporary disagreement and does not by itself bound how soon convergence occurs or what intermediate reads observe.
3426: statistical hypothesis test — grade 2, requirements [4]
A decision procedure comparing observations with expectations under a specified statistical hypothesis. Rejection or non-rejection depends on assumptions and a decision rule; failing to reject does not establish that the hypothesis is true.
3428: statistical power — grade 2.5, requirements [3, 4]
The probability that a test rejects its null hypothesis for a specified alternative when that alternative holds. Power depends on effect, design, sample size, noise and decision threshold; it is not a property of sample size alone.
3429: statistical decision errors — grade 3, requirements [4]
A type I error rejects a true null hypothesis; a type II error fails to reject it under a specified alternative. Their probabilities belong to the chosen hypotheses and decision procedure, rather than a general measure of factual correctness.
3449: multiple comparisons problem — grade 3, requirements [1, 4]
The increased opportunity for misleading findings when many statistical comparisons are made and interpreted selectively. Error control depends on the family of questions and procedure, not only the threshold for each individual test.
3450: meta-analysis — grade 0, requirements []
Statistical synthesis of results from multiple studies addressing related questions. Validity depends on study compatibility, bias, uncertainty and selection; combining estimates does not automatically remove weaknesses shared by the studies.
3460: random measurement error — grade 1, requirements []
A component of measurement error varying unpredictably between repeated measurements under stated conditions. Statistical treatment can characterize its variability; randomness does not imply that the complete measurement is unbiased.
3472: experiment — grade 1, requirements []
An inquiry that deliberately controls or changes conditions to examine resulting observations against a question or hypothesis. Its inferential strength depends on design, comparison and measurement, not merely on intervening.
3475: randomized controlled trial — grade 1, requirements []
An experiment assigning eligible units to interventions by a random mechanism and comparing outcomes. Randomization limits certain confounding in expectation, but attrition, nonadherence, measurement and limited scope can still affect interpretation.
3478: quasi-experiment — grade 0, requirements []
An intervention study without full random assignment that uses design or analysis to support a causal comparison. Its identifying assumptions require explicit scrutiny because treatment groups may differ for other reasons.
4189: hierarchy of evidence — grade 1, requirements []
A heuristic ordering of study designs or evidence types by their expected strength for a specified question, often by susceptibility to bias. The ranking depends on the question, outcome and execution; a high-ranked design is not a guarantee that a particular result is sound.
4288: Buddhist two-truths doctrine — grade 0, requirements []
A distinction between conventional and ultimate truth or reality in Buddhist teaching. Schools differ about how the two relate; distinguishing levels does not automatically license any contradiction or make conventional concerns disposable.
4684: sequential statistical analysis — grade 3, requirements [1, 2, 4]
Evaluating accumulating data under a prespecified monitoring and stopping procedure, allowing sample size to depend on observations while accounting for the repeated decisions.
4699: experimental blocking — grade 1, requirements []
Grouping experimental units by relevant nuisance variation and arranging treatment comparisons within those groups. The purpose is to separate the treatment comparison from known sources of variability.
I am a lake-restoration officer choosing between reducing nutrient inflow, removing a common fish, or combining the two. The lake is persistently cloudy, but fish surveys and nutrient measurements vary strongly by season. Explain how food-web interactions, external inputs, and delayed ecosystem responses could support different choices, and propose monitoring that can separate treatment effects from ordinary seasonal variation.
The original typed selection replaces environmental monitoring with food web, duplicating requirements already covered and losing monitoring. The new policy places environmental monitoring first, restoring the dense set’s 3/4 coverage and improving its ordering. It nevertheless selects two grade-0 items: memoization and internal mutual support alongside external subjection. No document in this judged pool has agreed coverage for delayed responses or feedbacks.
| Rank / checked ID | Document | Mean grade | Agreed requirements (one-based) | New requirements at rank |
|---|---|---|---|---|
| 1 / 3111 | ecosystem service | 1 | None | None |
| 2 / 3090 | nutrient cycle | 3 | 1, 2 | 1, 2 |
| 3 / 3120 | net ecosystem production | 1 | None | None |
| 4 / 75 | closed-loop habitat | 1.5 | None | None |
| 5 / 3077 | environmental monitoring | 3 | 4 | 4 |
| Rank / checked ID | Document | Mean grade | Agreed requirements (one-based) | New requirements at rank |
|---|---|---|---|---|
| 1 / 3111 | ecosystem service | 1 | None | None |
| 2 / 3090 | nutrient cycle | 3 | 1, 2 | 1, 2 |
| 3 / 3120 | net ecosystem production | 1 | None | None |
| 4 / 75 | closed-loop habitat | 1.5 | None | None |
| 5 / 3121 | food web | 3 | 1, 2 | None |
| Rank / checked ID | Document | Mean grade | Agreed requirements (one-based) | New requirements at rank |
|---|---|---|---|---|
| 1 / 3077 | environmental monitoring | 3 | 4 | 4 |
| 2 / 3121 | food web | 3 | 1, 2 | 1, 2 |
| 3 / 3111 | ecosystem service | 1 | None | None |
| 4 / 2740 | memoization | 0 | None | None |
| 5 / 1970 | internal mutual support alongside external subjection | 0 | None | None |
| Rank / checked ID | Document | Mean grade | Agreed requirements (one-based) | New requirements at rank |
|---|---|---|---|---|
| 1 / 3077 | environmental monitoring | 3 | 4 | 4 |
| 2 / 3090 | nutrient cycle | 3 | 1, 2 | 1, 2 |
| 3 / 3121 | food web | 3 | 1, 2 | None |
| 4 / 4387 | trophic cascade | 3 | 1, 2 | None |
| 5 / 3117 | ecosystem | 2.5 | 2 | None |
75: closed-loop habitat — grade 1.5, requirements []
A habitat designed by treating air, water, food, waste, energy, organisms, and maintenance as coupled cycles whose inputs, outputs, losses, and dependencies are explicit.
1124: separate measures for infrastructure ecology and heritage harm — grade 2, requirements [4]
A draft monitoring agenda keeps power capacity loss, pollution, ecological damage and cultural-site damage as different kinds of impact. Each needs appropriate units, baselines and evidence; a shared dashboard must not make unlike measurements interchangeable or infer a common attribution from proximity.
1558: seasonal variation recorded in sediment layers — grade 2, requirements [4]
A proposed relation between recurring changes in water volume and transport and alternating layers of coarser and finer deposited material.
1970: internal mutual support alongside external subjection — grade 0, requirements []
An institution may support its own members while collectively dominating outsiders. Internal reciprocity and the treatment of nonmembers therefore require separate evaluation.
2442: historical description, current allocation — grade 0, requirements []
The distinction between an earlier description retained as history and the validated registry governing present use. A linked amendment identifies the changed assignment and its reason without silently rewriting the earlier wording. Historical references remain tied to their original snapshots; preserving a statement does not renew an obsolete allocation.
2740: memoization — grade 0, requirements []
Reusing stored results when the same function inputs recur, avoiding repeated computation at the cost of storage. The stored result must remain valid for all relevant dependencies; hidden state or side effects can make apparently identical calls require different treatment.
3077: environmental monitoring — grade 3, requirements [4]
Repeated or organized observation of environmental conditions using a design suited to a stated question. It can establish baselines and trends, while requiring attention to sampling, measurement and interpretation before change or its cause is inferred.
3090: nutrient cycle — grade 3, requirements [1, 2]
Movement and transformation of biologically relevant matter among organisms and environmental reservoirs. Materials can be reused through linked processes; this cycling differs from energy flowing through a system and becoming less available for work.
3110: ecological restoration — grade 2, requirements [1]
Assisting the recovery of an ecosystem that has been damaged, degraded or destroyed through interventions guided by ecological understanding and explicit goals. Recovery need not mean exact reproduction of one historical condition.
3111: ecosystem service — grade 1, requirements []
A benefit people obtain from ecosystems, such as food, water regulation, pollination or cultural experience. This human-benefit lens helps describe dependencies while leaving room for ecological and ethical values not captured by service accounting.
3117: ecosystem — grade 2.5, requirements [2]
Organisms and their physical environment considered as an interacting system, connected by transfers of energy and cycling of materials. Its boundary and scale depend on the question being studied; it is more than a list of species sharing a place.
3120: net ecosystem production — grade 1, requirements []
Gross primary production minus respiration by all organisms within the selected ecosystem and time interval. Other transfers and disturbances must be accounted for before interpreting this difference as a complete carbon-stock change.
3121: food web — grade 3, requirements [1, 2]
A network of feeding relationships that connects organisms through transfers of matter and energy. A participant may draw on several food sources and occupy more than one trophic level, so a single food chain captures only selected paths through the network.
4387: trophic cascade — grade 3, requirements [1, 2]
An indirect ecological effect transmitted across trophic levels, for example when changing predators changes herbivore pressure on producers. Direction and strength depend on the food web and conditions; restoring one species need not guarantee a desired ecosystem outcome.
5801: wetland — grade 0.5, requirements []
An ecosystem whose sustained or recurrent inundation or saturation shapes soil processes and biota, often producing oxygen-limited soils and water-adapted vegetation.
6188: snow line — grade 0, requirements []
A boundary between snow-covered and snow-free terrain under a specified spatial and temporal definition. Seasonal, end-of-melt-season and persistent snow lines answer different questions.
The frozen method often spends late slots on dissimilar evidence with no agreed obligation. The max-over-facets objective saturates, and its duplication cost can reward dissimilar but irrelevant candidates. Lower pair cosine therefore cannot be read as better context. Separately encoded requirement text can also omit the subject context needed to disambiguate a generic requirement. These are post-outcome interpretations, not newly tested ablations. The method was not retuned after seeing them.
A21 and B18 demonstrate particular ordering and displacement effects, not overall success. The larger result is mixed coverage and worse pooled nDCG. The model-generated labels, purposive tasks, existing incomplete method pool and public/pretraining exposure limit generalization. No answer reader experiment or corpus-wide performance claim is supplied by this module.
Foundation reuse: R02 local MiniLM tokenizer and mean-pooling convention; R02 declared nDCG definition. Source corpus, tasks, question/document embeddings and relevance labels remain unchanged. Requirement embeddings alone are new.