# USL Lexical Base message codec — runtime release 1.1.0 This file specifies `usl-lexical-message/1` JSON messages and the equivalent `USL1` binary transport. The dependency-free reference implementation is `usl-lexical.js`. It exports `USLLexical` in browsers and a CommonJS object in Node.js. Both need WebCrypto SHA-256; a current browser secure context or current Node.js supplies it. This is a lexical transport. A source form is not a lemma, sense, definition, or certified assertion. An imported English form and a Carrier concept are different types. No spelling match automatically creates a semantic relationship. The codec does not assign parts of speech or infer morphology, translation, synonymy, or equivalence. ## Runtime 1.1.0 and efficient whole-text APIs `VERSION` identifies the implementation (`1.1.0`). `LEXICAL_VERSION` identifies the supported frozen dictionary (`1.0.0`). Installing a dictionary must use its own version, not the implementation version. The binary version byte remains `1`: short packets are byte-for-byte compatible with the 1.0.0 runtime. Older runtimes retain their older size and token limits and may reject larger packets. ```js const encoded = await USLLexical.encodeTextBinary(text, { lexicon, concepts: [], previewLimit: 140 }); // {bytes, preview, concepts, stats} const decoded = await USLLexical.decodeTextBinary(encoded.bytes, { lexicon, snapshots: [], previewLimit: 140, matchingSnapshotsOnly: false }); // {text, utf8Bytes, resolvedConcepts, unresolvedConcepts, stats, preview, concepts} ``` The installed handle also offers `lexicon.encodeTextBinary(text, options)`. Both efficient APIs retain only the first `previewLimit` tokens and **all explicitly selected concept tokens**. The default preview limit is 140; the allowed range is 0–10,000. `stats` counts the full stream. Concept resolution details retain token indices from the full stream. `preview` has `{format,lexicon,utf8Bytes,tokens}` for display, with the full source byte length and a possibly truncated token list. It is not a complete message and must not be passed to `encodeBinary`, `validateMessage`, or `decodeMessage` as though it were one. Use `bytes` as the complete transport object. The `concepts` result is the complete list of concept tokens, including selections beyond the preview; it supports collecting every required historical snapshot for a sequence package. By default snapshot matching has the same strict behavior as `decodeMessage`. `matchingSnapshotsOnly:true` filters the supplied snapshots by the exact registry/tier/head references present in this binary before resolving them. Missing exact heads remain unresolved, and unrelated snapshots in a caller's historical cache are ignored. The option never substitutes a different head or changes recovered surface text. Invalid matching snapshots and missing IDs still fail. Callers remain responsible for verifying the Carrier chains before passing snapshots to the codec. The encoder visits exact source spans and writes token bytes into a buffer that starts at 4 KiB and grows as needed. It does not retain a full lexical/literal token object array. The decoder owns the supplied bytes before asynchronous hashing, verifies the full digest and metadata, checks all token framing and UTF-8, and verifies the fully expanded length **before allocating the output buffer**. It then reconstructs exact UTF-8 in a second pass. Output is exposed only after all validation succeeds. The complete binary and reconstructed source each have a 128 MiB limit. Token count may be at most 134,217,728 and cannot exceed the body byte length, because every token occupies at least one body byte. Metadata remains limited to 1 MiB. Header and metadata overhead count toward the complete packet limit. Peak memory still includes the installed lexicon, input/output text, binary buffers and SHA-256 input; these APIs do not promise constant memory or streaming network consumption. The compatibility APIs retain full token arrays and can require substantially more memory for large inputs. Prefer the efficient APIs for books and other long documents. ## Pinned lexical addresses When one message explicitly references several historical Carrier heads, each head resolves independently. An available head requested by another token is not a substitute for a missing head: the missing reference is reported as unresolved. Supplying an unrelated wrong head for a requested registry/tier still fails validation. The supported namespace is `https://www.unifiedstate.us/carrier/lexicon/en-moby`, release `1.0.0`. A lexical address consists of that namespace and version, the canonical forms SHA-256, the release manifest SHA-256, and an integer ID. All four pin fields must match the installed lexicon. Hashes are exactly 64 lowercase hexadecimal characters. The seed is a nonempty array of distinct, exact, case-sensitive NFC strings. ID is the zero-based array index. The forms hash is SHA-256 of UTF-8 `JSON.stringify(forms)` with no trailing newline. Forms must be nonempty and at most 4,096 UTF-8 bytes each; the canonical JSON file is at most 64 MiB. The maximum count is 16,777,216 forms. These are implementation resource limits, not claims about the number of English words. ID `0xRRGGBB` has address color `[RR,GG,BB]`. The global colour registry reserves the published English Base 1 range. This unchanged USL1 transport still carries lexical namespace and release pins; historical messages are never reinterpreted using a newer allocation snapshot. A colour appearance alone does not certify its owner, release, or meaning. These address colors are distinct from CVP1's calibrated transmission palette. Unchanged CVP1 encodes four payload bits per module; it does not transmit one arbitrary RGB24 word per module. Framing, metadata, and error correction add overhead. Compare actual complete binary and compressed CVP1 sizes against raw UTF-8 when evaluating compression. `install` verifies the canonical forms hash. When provided, `formsBytes` must be byte-for-byte canonical JSON and `manifestBytes` must match the manifest pin. Without `manifestBytes`, the supplied manifest pin must have been verified by the caller's release loader. The loader must obtain expected hashes from a trusted published release; matching a self-supplied hash does not authenticate a source. Even a verified manifest hash is integrity evidence, not a signature or proof that every source word is current or appropriate. ## API ```js const lexicon = await USLLexical.install({ namespace: 'https://www.unifiedstate.us/carrier/lexicon/en-moby', version: '1.0.0', forms, formsSHA256, manifestSHA256, formsBytes, // optional Uint8Array of the canonical downloaded file manifestBytes // optional Uint8Array of the exact manifest file }); lexicon.count; lexicon.lookup('bank'); // a scoped address object, or null; case-sensitive lexicon.search('ban', { limit: 20, mode: 'prefix' }); lexicon.getForm(3); // equivalent to lexicon.form(3) const message = lexicon.encodeText(sourceText, { concepts: [] }); const binary = await USLLexical.encodeBinary(message); const recoveredMessage = await USLLexical.decodeBinary(binary); const recovered = USLLexical.decodeMessage(recoveredMessage, { lexicon, snapshots: [{ registry, tier, head, entries }] }); // recovered: {text, utf8Bytes, resolvedConcepts, unresolvedConcepts, stats} ``` Static `lookup(lexicon, form)`, `search(lexicon, query, options)` and `encodeText(text, {lexicon, concepts})` are equivalent to handle methods. `idToRGB(id)` converts an ordinal for display; it does not resolve a reference. `validateMessage(packet)` validates JSON syntax and resource limits, but lexical IDs and actual decoded length require `decodeMessage` with an installed handle. `lookup` returns `{namespace,version,formsSHA256,manifestSHA256,id,form,rgb,anchor,trust}`. `anchor` is lowercase `#rrggbb`; `trust` is `source-membership-only`. Search supports `prefix` (default), `contains`, and `exact`, with limit 1–200. Search and lookup are case-sensitive and do not silently normalize input. Handles expose pin properties and `manifestBytesVerified`. The installed form array is privately copied and frozen. ## Exact text and explicit concept selections Text is valid Unicode encoded as UTF-8. JavaScript strings containing unpaired surrogates are rejected rather than silently replaced. Input is never normalized, case-folded, whitespace-collapsed, or reworded. Combining characters, a leading BOM, NUL, punctuation, line endings, and unknown forms survive exactly. The reference encoder first checks each entire non-whitespace chunk against the source array, so punctuation forms such as `&c`, `'em`, and `a.m.` can be lexical tokens. Otherwise it checks exact candidates consisting of Unicode letters/marks/numbers with internal apostrophe, right apostrophe, or hyphen. Separators and unknown candidates become literal text. Adjacent literals are merged. This tokenizer intentionally makes no completeness claim: every imported form is addressable via lookup, while some punctuation or compound contexts may use literals. Another encoder may select other exact form spans as long as it preserves the original text and never implies a concept relationship. Concepts are selected explicitly using UTF-16 source offsets: ```js const message = lexicon.encodeText('the visible carrier', { concepts: [{ start: 4, end: 19, text: 'visible carrier', registry: 'https://www.unifiedstate.us/carrier/', tier: 'checked', id: 38, head: checkedSnapshotHead }] }); ``` Every selection must include exact `text === sourceText.slice(start,end)`. Selections must have nonempty, valid Unicode surfaces and cannot overlap. The supplied surface remains the recovered text; the concept name never replaces it. The registry is an absolute HTTP(S) URL without credentials, query, or fragment. A checked ID is a nonnegative safe integer; a working ID is `w` followed by a canonical nonnegative safe integer. The caller must validate supplied Carrier snapshots using the Carrier chain loader before passing them here. The codec matches registry, tier, head, and ID. An exact matching snapshot resolves a reference and returns its entry separately. An absent snapshot yields `unresolvedConcepts` with `reason: 'snapshot-not-provided'` while preserving the surface text. A supplied registry/tier snapshot with an unrelated wrong head is rejected; a head explicitly requested by another token may resolve that other token while the missing head remains unresolved. A missing ID in a matching snapshot, duplicate ID, or ambiguous duplicate snapshot is rejected. A different registry is never silently substituted. Snapshot matching does not add an independent semantic endorsement or promote working entries to checked. ## JSON message ```json { "format": "usl-lexical-message/1", "lexicon": { "namespace": "https://www.unifiedstate.us/carrier/lexicon/en-moby", "version": "1.0.0", "formsSHA256": "<64 lowercase hex>", "manifestSHA256": "<64 lowercase hex>" }, "utf8Bytes": 8, "tokens": [ {"type":"lexical","id":7}, {"type":"literal","text":" "}, {"type":"lexical","id":3} ] } ``` The IDs in this illustrative packet depend on its pinned source array. The three exact token shapes are: * `{type:'literal', text}`: nonempty literal Unicode text. * `{type:'lexical', id}`: integer ID from 0 through 0xFFFFFF, resolved against the pinned installed array. * `{type:'concept', registry, tier, id, head, text}`: a fully pinned Carrier reference plus nonempty original surface text. No extra fields are accepted in message, pin, token, or concept reference objects. `utf8Bytes` is the exact length of the reconstructed UTF-8 text. An empty message has zero tokens and zero bytes. JSON syntax validation alone cannot verify lexical expansion length; successful `decodeMessage` is required. JSON has no built-in cryptographic integrity field. Use the binary format or an independently authenticated enclosing artifact when integrity matters. ## USL1 binary format All fixed-width integers are unsigned little-endian. The complete object is limited to 128 MiB, including its header and metadata. Fixed header length is 64 bytes: | Offset | Length | Field | | --- | --- | --- | | 0 | 4 | ASCII `USL1` | | 4 | 1 | Binary version, exactly 1 | | 5 | 1 | Flags, exactly 0 | | 6 | 2 | Fixed header length, exactly 64 | | 8 | 4 | Metadata byte length, 1 through 1,048,576 | | 12 | 4 | Token count, 0 through 134,217,728 | | 16 | 4 | Reconstructed text UTF-8 byte length, 0 through 134,217,728 | | 20 | 4 | Body byte length | | 24 | 8 | Reserved, all zero | | 32 | 32 | SHA-256 digest | | 64 | variable | Metadata UTF-8 JSON | | following metadata | variable | Token body | SHA-256 covers bytes `0..31` concatenated with bytes `64..end`. Thus every header field, lexical pin, concept reference, and token byte is integrity-bound. Only the digest's own slot is excluded. This hash detects modification relative to a supplied object; it is not an author signature. CVP1 adds its existing object integrity and error correction without any CVP1 wire change. Metadata is exactly the compact `JSON.stringify` form of the following ordered object, with its nested fields in the order shown: ```js { format: 'usl-lexical-message/1', lexicon: {namespace, version, formsSHA256, manifestSHA256}, concepts: [{registry, tier, id, head}, ...] } ``` The encoder deduplicates concept references in first-use order. Duplicate or unused references are rejected. The compatibility API `decodeBinary` returns a full validated JSON packet for `decodeMessage`. The efficient API `decodeTextBinary` validates, resolves and reconstructs text directly without retaining the full token array. Body integers are canonical unsigned LEB128, at most five bytes and no greater than 0xFFFFFFFF. Redundant high zero groups, overflow, or truncated integers are rejected. Each token begins with an operation integer; `tag = operation % 4`, `parameter = floor(operation / 4)`: | Tag | Meaning | Following bytes | | --- | --- | --- | | 0 | Literal with UTF-8 byte length `parameter`, which must be positive | Exactly that many UTF-8 bytes | | 1 | Lexical ID `parameter`, from 0 through 0xFFFFFF | None | | 2 | Index `parameter` into the metadata concept table | Positive UTF-8 surface byte length as ULEB128, then exact surface bytes | | 3 | One literal ASCII space, only when `parameter === 0` | None | All other tag-3 parameters are reserved and rejected. Concept surface text is carried explicitly. The literal-space form saves overhead for ordinary word boundaries; a general literal containing one space is also valid. The body must contain exactly the declared number of tokens and have no trailing bytes. UTF-8 decoding is fatal and preserves BOM characters. Header lengths, count limits, and digest are checked before token parsing. `decodeMessage` then checks exact lexical pins, actual IDs, concept snapshots, and the final reconstructed byte count. ## Bounds and validation Input and reconstructed text each have an 128 MiB UTF-8 limit, with at most 134,217,728 tokens. The complete binary has the same 128 MiB limit, so near-limit source text can require a smaller input or multiple independently framed objects when metadata overhead would exceed that bound. Binary metadata is limited to 1 MiB. Decoding fails before joining an output larger than its declared count or the output limit. Caller-owned binary input is copied before asynchronous hash verification to prevent mutation races. These limits make malformed messages bounded; they do not guarantee a browser can install an arbitrarily large future lexicon within available memory. Releases should measure memory and transport sizes. Do not promote a list of forms into Carrier checked concepts or treat a successful byte round-trip as proof of meaning. ## Conformance Run `node --test usl-lexical.test.cjs` in this workspace. Nine test groups cover independent runtime/dictionary versions; byte-exact comparison with the preserved 1.0.0 runtime; punctuation, case and Unicode fidelity; 80 deterministic mixed-input cases; explicit and historical concept references; bounded previews; malformed hashes, framing, lengths and UTF-8; caller-input mutation during hashing; and the complete `64/pg10.txt` source round-trip. The full-book test also passes with a 96 MiB Node old-generation heap cap; typed-array storage is outside that cap, so it is not a total-memory bound. The suite does not claim browser UI behavior, semantic correctness, camera robustness, or complete English coverage.