Introduction

Welcome to the Tribles Book. This first chapter is your map to Trible Space—the problems it tackles, the core ideas behind the triblespace crate, and the kinds of projects it was built to support. By the end of these opening pages you should be able to recognize when TribleSpace is the right tool, whether you are prototyping something new, extending an existing system, or adapting it for your own research purposes.

Why TribleSpace exists

TribleSpace exists because teams need to steward complex, diverse, and interconnected datasets without losing context. Research groups, startups, and digital libraries must pair large binary payloads with fine-grained facts, synchronize findings across laptops, servers, and mobile devices, and prove that derived results trace back to their inputs. Combining a conventional database, object store, and version-control workflow rarely delivers that outcome: blobs drift away from their provenance, merges become brittle, and auditing which observations justified a decision or model is tedious.

TribleSpace closes those gaps with a single substrate that stores heavyweight assets and the relationships that explain them. Think of it as a library catalog for blobs and a lab notebook for the annotations, measurements, and discussions that give those blobs meaning. Because facts and payloads travel together, features like reproducible versioning, verifiability, and provenance fall naturally out of the data model instead of bolting on as afterthoughts.

That same structure also lets you support offline edits, reconcile concurrent changes safely, and ship datasets to partners with the evidence needed to trust them.

To deliver those outcomes, TribleSpace blends ideas from databases, join semilattices, and content-addressed storage. Information is encoded as fixed-width tribles: 64-byte entity–attribute–value facts. Each trible stores two 16-byte extrinsic identifiers plus a 32-byte typed value. When a value exceeds the inline slot it becomes a schema-qualified hash pointing to an immutable content-addressed blob. The blob holds the heavyweight payload, while the trible remains a compact fact that fits neatly into indexes, caches, and query engines. Because both tribles and blobs are immutable, you can keep them in memory, on disk, or in remote object stores without transformation. Content hashes serve as identifiers, so every payload has a stable address and integrity is easy to verify when data is shared or synchronized.

This design unlocks capabilities that are difficult to achieve together in traditional stacks:

• Trustworthy collaboration – signed commits and immutable algebra records provide the audit trail needed to review contributions, union independent work, and reproduce results across teams.
• Content-addressed storage – values are stored and looked up by their contents, making caches and replicas safe even when they live on untrusted infrastructure.
• Flexible querying – the query engine blends indexes on the fly, letting a single query range across trible sets, succinct indexes, and familiar Rust collections such as hash maps in one pass.

Taken together, these traits make it feasible to build systems with rich histories, reproducible computations, and verifiable data exchange while keeping the developer experience approachable.

Who this book is for

If you are new to Tribles, the opening chapters build vocabulary and provide a guided tour of the core data structures. Developers who already understand the problem space can skim ahead to detailed sections on schema design, query semantics, and the Atreides join algorithm. The book also points to the API documentation whenever you are ready to explore the crate directly.

How to read this book

The book is organized so you can either read it front-to-back or jump straight to the material that answers your questions. Each chapter layers new ideas onto the previous ones:

1. Getting Started walks through installing the tooling, creating your first trible store, and issuing simple queries.
2. Architecture and Query Engine explain how the runtime is structured so you can reason about performance and extensibility.
3. Later sections explore schema design, incremental queries, collection workflows, and formal verification so you can grow from experiments to production systems.

Inline links lead to deeper resources and code samples. The Glossary offers quick refreshers on terminology, while Developing Locally covers how to set up a development environment and contribute back to the project. Whenever the book introduces a new concept, look for references to the crate documentation so you can inspect the corresponding APIs and examples.

By the end of this chapter you should have a mental model for why TribleSpace is structured the way it is. From there, head to the chapters that match your goals—whether that means learning to query data effectively or integrating Tribles into a larger system.

Philosophy

Triblespace was designed to feel approachable without sacrificing rigor. This chapter collects the guiding values that shape everything from the storage format to the public APIs. Treat it as a companion to the rest of the deep-dive sections: when you wonder "why does it work this way?", the answer usually traces back to one of these principles.

Clarity before cleverness

We favour predictable, mechanically simple components over opaque heuristics. Each subsystem should be understandable on its own, with behaviours that are obvious when composed with the rest of the stack. When a trade-off appears between a clever optimisation and debuggability, we err on the side of the latter and document the costs so future work can revisit the decision with better evidence.

Productive developer experience

APIs should read like regular Rust. Where backends demand asynchronous capabilities—such as remote object stores—we wrap them in blocking entry points so typical workflows stay synchronous while still supporting advanced integrations. Well-documented patterns and composable macros let readers experiment in a REPL or test harness without extra scaffolding, and examples in the book mirror the crates users import so copy-and-paste snippets behave as advertised.

Soundness and data integrity

The engine must reject malformed data early, surface explicit error paths, and make invariants easy to audit. Safety checks live close to the data structures that rely on them, and proofs or tests accompany subtle invariants when feasible. Correctness remains the baseline for every optimisation.

Performance with headroom

Efficient data structures keep the core fast, but we prioritise predictable latency over micro-benchmarks that complicate maintenance. Hot paths receive focused tuning backed by benchmarks so we can understand the impact of each change.

Practical implications

These principles surface in day-to-day workflows:

• Documentation favours runnable snippets and end-to-end walkthroughs, lowering the barrier to experimentation.
• Internal abstractions expose minimal, intentional APIs, reducing the amount of context a contributor needs before making a change.
• Tooling—such as the preflight.sh convenience script, targeted Kani verification harnesses, and runnable doc tests—keeps quality checks accessible so they are run regularly rather than only in CI.

Taken together, the philosophy is simple: build a reliable system whose pieces are easy to reason about, teach, and extend.

Getting Started

This chapter publishes and queries a small native collection. It assumes Rust is installed and you are comfortable running cargo commands.

1. Add dependencies

cargo new tribles-demo
cd tribles-demo
cargo add triblespace ed25519-dalek rand

triblespace supplies the data model, stores, collection operations, and query macros. ed25519-dalek and rand create the publishing identity used in this example.

2. Declare attributes

Attributes carry the encoding of their value. Shared attributes should use a stable explicit anchor; the encoding participates in the resulting identity. Omit the anchor only for local prototypes whose identity may follow their name.

mod literature {
    use triblespace::prelude::*;
    use triblespace::prelude::blobencodings::UTF8String;
    use triblespace::prelude::inlineencodings::{GenId, Handle};

    attributes! {
        "A74AA63539354CDA47F387A4C3A8D54C" as pub title: Handle<UTF8String>;
        "6A03BAF6CFB822F04DA164ADAAEB53F6" as pub quote: Handle<UTF8String>;
        "8F180883F9FD5F787E9E0AF0DF5866B9" as pub author: GenId;
        "0DBB530B37B966D137C50B943700EDB2" as pub firstname: Handle<UTF8String>;
        "6BAA463FD4EAF45F6A103DB9433E4545" as pub lastname: Handle<UTF8String>;
        "D2D1B857AC92CEAA45C0737147CA417E" as pub alias: Handle<UTF8String>;
    }
}

Use trible genid when minting a new published anchor. The literal-pinning "HEX_ID" unsafe as ... spelling is only for preserving an already-published attribute's exact historical bytes when its old identity cannot be re-derived.

3. Register a collection

A root collection is identified by the content handle of its descriptor. The descriptor carries its UTF-8 name, member encoding, and independent READ and WRITE admission policies. The encoding itself owns member validation and defines one canonical member-join operation; a root needs no mapping. The descriptor is an ordinary self-contained Fragment, but applications ask the store to construct and register it rather than assembling its facts manually. Its canonical content handle is the collection identity:

use ed25519_dalek::SigningKey;
use rand::rngs::OsRng;
use triblespace::core::collection::{AdmissionPolicy, CollectionPolicy};
use triblespace::prelude::*;

let key = SigningKey::generate(&mut OsRng);
let mut storage = MemoryRepo::default();
let root = key.verifying_key();
let library = storage.collection(
    "library",
    CollectionPolicy::new(
        AdmissionPolicy::direct(root),
        AdmissionPolicy::direct(root),
    ),
)?;

Each policy is either Open or a canonical threshold over capability roots. AdmissionPolicy::direct(root) records one such root. Each signed proof edge then says whether its recipient may invoke, delegate, or do both. The two policies are deliberately independent: a collection can have many writers but few readers, or public READ with tightly held WRITE. Both policies participate in the descriptor handle, so there is no ambient team scope or separate reach flag.

Local publication is unconditional: any process may append a structurally valid self-signed commit to its own store. Reading applies the descriptor's WRITE policy. Commits by a policy root are admitted directly; other authors become visible only when the same store snapshot contains enough valid proof paths for exact ACTION_WRITE on this descriptor. Invalid resident evidence grants nothing.

4. Build a self-contained fragment

With no explicit subject prefix, entity! derives the entity ID from the canonical set of emitted fields. Fragment::root() returns that exported ID, which can be referenced by another entity. Text becomes UTF8String blobs that remain attached to the fragment automatically; the fact holds the blob's content handle.

let author = entity! {
    literature::firstname: "Frank",
    literature::lastname: "Herbert",
};
let author_id = author.root().expect("intrinsic author id");

let book = entity! {
    literature::title: "Dune",
    literature::author: &author_id,
    literature::quote: "I must not fear. Fear is the mind-killer.",
};

let mut import = author;
import += book;

A fragment has four coordinated channels: facts, descriptive metafacts, exported IDs, and one shared attachment store. += unions all four, so no parallel manifest or manual blob-staging step is needed.

5. Publish independent commits

let first = storage.commit(library, &key, import)?;

// A later fact about the same entity is another independent member.
let second = storage.commit(library, &key, entity! {
    &author_id @ literature::alias: "Francis",
})?;

assert_ne!(first, second);

Publication loads and structurally validates the registered descriptor, then writes fragment attachments, data, and metadata before inserting the signed COMMIT record. It performs no permission check and no implicit flush. The commit is the atomic assertion. There is no mutable head to advance: both records remain members and the collection value is their union.

Repeating byte-identical input produces the same exact canonical commit and is idempotent. Distinct input produces another coexisting member. Application-level supersession or versioning is represented in the facts when a domain needs it; append order is never an implicit winner.

6. Read one coherent snapshot

let snapshot = storage.snapshot()?;
let admitted = library.admitted(&snapshot)?;
let facts: TribleSet = library.read(&snapshot)?;
let title = "Dune";

for (first, last, quote) in find!(
    (first, last, quote),
    pattern!(&facts, [
        { _?author @
            literature::firstname: ?first,
            literature::lastname: ?last
        },
        { _?book @
            literature::title: title,
            literature::author: _?author,
            literature::quote: ?quote
        }
    ])
) {
    let first: View<str> = snapshot.get(first)?;
    let last: View<str> = snapshot.get(last)?;
    let quote: View<str> = snapshot.get(quote)?;
    println!(
        "'{}'\n - from {title} by {} {}.",
        quote.as_ref(),
        first.as_ref(),
        last.as_ref()
    );
}

storage.snapshot() freezes blobs, collection records, capability proofs, and backend state at one coherent known prefix. library.admitted(&snapshot) reads the coverage index in that same observation and returns the exact semantic payload cover: the union of what the collection's frontier stands for, whether or not every byte is here. library.read(&snapshot) attaches that frontier from the same index and constructs the logical value from the nodes whose bytes are resident, through the same immutable snapshot; a node whose bytes are elsewhere is descended through the MERGE that produced it, so the value stands on whatever is local and the support says exactly what that is. Snapshots carry no clock: later proof or content arrivals require a new observation, while an application evaluates its own time-sensitive rules at an explicit instant.

Provenance for an exact cover is available through cover.commits(&snapshot). These are strictly verified attestations over its payloads, not necessarily the authorized claims which admitted them. Duplicate signed claims for one payload collapse to one cover member; authors, signatures, and metadata remain provenance rather than payload identity.

7. Choose durability explicitly

store.commit performs no implicit flush. For a memory store that makes no difference. For a pile or remote backend, choose the durability boundary that matches the application:

storage.commit(library, &key, batch_a)?;
storage.commit(library, &key, batch_b)?;
storage.flush()?;

Amortizing one flush over several commits does not weaken their logical identity or change merge semantics. Flushing and closing remain operations of the chosen backend rather than collection policy.

What to remember

• entity! builds intrinsic entities and carries required blobs.
• Fragment is the self-contained publication value.
• Collection<E> is a descriptor handle statically bound to the CollectionEncoding which owns its member bytes, validation, and canonical physical join operation; the store owns all I/O.
• store.commit publishes one signed, independent member without conflating local storage with network authorization.
• store.snapshot freezes one coherent known-prefix store observation; collection admission and materialization are pure reads over it.
• Replicas converge by unioning records; they never elect a branch head.
• Derived indexes are reproducible collection images, not alternate authority.

Continue with Collection Workflows for the native record algebra, exact covers, migration from legacy piles, and derived collection maintenance.

Developing Locally

Tribles is developed with stable Rust tooling and a small collection of helper scripts. This chapter walks through preparing a development environment, running tests, and rebuilding the documentation so you can iterate with confidence.

Prerequisites

1. Install a recent Rust toolchain from rustup.rs. The default stable channel is what the project targets.

2. Clone the repository and switch into it:

   git clone https://github.com/TribleSpace/triblespace-rs.git
   cd triblespace-rs
   

3. (Optional) Install mdbook with cargo install mdbook if you would like to preview the rendered documentation locally.

Everyday workflows

The repository includes several scripts that keep formatting, tests, and book builds in sync:

• ./scripts/devtest.sh executes the most relevant unit and integration tests for fast feedback while you are iterating.
• ./scripts/preflight.sh runs formatting, the full test suite, and rebuilds this book. Run it before committing to ensure your branch is in good shape.
• ./scripts/build_book.sh regenerates the documentation after you modify Markdown chapters or code snippets.

You can always fall back to the standard Cargo commands if you prefer to run specific tools by hand, but use cargo fmt --all and cargo test --workspace. The repository root is both the workspace root and its own package (the triblespace facade), so the bare forms select only that package and silently skip triblespace-core and the other workspace members.

Rebuilding the book

Once mdbook is installed you can rebuild the documentation with:

./scripts/build_book.sh

This script compiles the chapters into book/book, allowing you to open the HTML output in a browser. Rebuilding regularly helps catch stale examples and keeps the published copy aligned with the codebase.

Identifiers for Distributed Systems

Distributed systems are assembled from independently authored pieces of data. Keeping those pieces addressable requires names that survive replication, concurrent edits, and local conventions. We have found it useful to categorize identifier schemes along two axes:

UFOID = Universal Forgettable Ordered ID — a 128-bit identifier built from a 32-bit time prefix (the low 32 bits of the UNIX time in milliseconds) plus 96 random bits. The prefix gives recently-minted ids high locality: they sort roughly by mint time, so a system can range-scan "everything created in the last N hours" directly off the id. Forgettable refers to that prefix wrapping around about every 50 days — the high bits of absolute time are intentionally dropped, leaving a rolling window rather than a monotonic-forever clock. Within the window the ordering is exact; outside it the prefix repeats. That's the right trade for systems that only reason over a bounded recent horizon and garbage-collect older data — caches, robotics pipelines, event streams — where the time-locality is worth far more than a globally unique timestamp. (Use timestamp_distance to compare two prefixes correctly across the rollover.)

FUCID = Fast Unsafe Compressible ID — a 128-bit identifier generated by XOR-ing a per-source salt with a monotonic counter, so consecutive ids from the same source compress extremely well at the cost of being predictable per source. See the Comparison of Identifier Types table below for when to reach for which.

• Rows — derivability. An intrinsic identifier can be recomputed from the entity alone: anyone holding the bytes can produce the same id independently. An extrinsic identifier is assigned separately from the entity — the entity carries no hint of what its id should be.
• Columns — content encoding. An abstract identifier is opaque: its bits carry no readable meaning about the entity. A semantic identifier encodes meaning (words, codes, URL paths) that humans or machines can consume as signal without a lookup.

The axes are independent — every cell is populated — but the quadrants have different structural properties (see Quadrant Properties below). Classifying an identifier along both axes makes its trade-offs explicit and clarifies when a workflow needs to combine multiple schemes.

Abstract vs. Semantic Identifiers

Semantic identifiers

Semantic identifiers (names, URLs, descriptive labels, embeddings) carry meaning about the thing they reference. Humans don't work well with opaque bit patterns — we think in names. Any usable system has to expose some semantic layer, which is why semantic identifiers are structurally essential rather than merely convenient.

The power of the semantic layer scales inversely with its scope:

• Locally, semantic names are exactly what you want. A bibliography's [Herbert1965], a codebase's firstname, a paper's "the agent" — each is unique enough within its scope, collides with nothing outside it, and can evolve as your understanding improves.
• Globally, the same names get expensive. Semantic content is low-entropy (there are only so many sensible names for a thing), so two parties naming independently will collide. Avoiding collisions requires coordination — an authority, or an unbounded scope prefix whose root is still an authority (see Quadrant Properties).

Distributed systems get the best of both worlds by pairing a shared abstract identifier for global identity with many local semantic names on top. The same attribute id can be timestamp in my codebase, legacy_timestamp in yours, and created_at in a third — all binding to the same underlying id, each name authoritative within its scope. Three concrete benefits fall out:

• Decoupling. Groups don't have to agree on vocabulary to share data. The id is the shared ontology; names are each group's business.
• Evolution. You can rename your local label when the meaning clarifies ("height" → "stature") without breaking anyone else's system, because the global binding is unchanged.
• No bike-shedding. The hardest coordination problem in any shared system is naming. Making names purely local reduces that cost to zero.

Embeddings deserve a special mention. They encode meaning in a machine-friendly form that can be compared for similarity instead of exact equality. That makes them great for recommendations and clustering but still unsuitable as primary identifiers: two distinct entities can legitimately share similar embeddings, and embeddings can change whenever the underlying model is retrained.

Abstract identifiers

Abstract identifiers (UUIDs, UFOIDs, FUCIDs, hashes, signatures) strip all meaning away in favor of uniqueness. They can be minted without coordination, usually by drawing from a high-entropy space and trusting probability to keep collisions effectively impossible. Abstract identifiers shine when you need:

• Stable handles that survive across replicas and through refactors.
• Globally unique names without a centralized registrar.
• Cheap, constant-time generation so every component can allocate identifiers on demand.

Because they carry no inherent semantics, abstract identifiers are almost always paired with richer metadata. They provide the skeleton that keeps references consistent while semantic identifiers supply the narrative that humans consume.

Intrinsic vs. Extrinsic Identifiers

The intrinsic/extrinsic axis captures whether an identifier can be recomputed from the entity itself or whether it is assigned externally.

Intrinsic identifiers

Intrinsic identifiers (cryptographic hashes, digital signatures, content-based addresses) are derived from the bytes they describe. They function as fingerprints: if two values share the same intrinsic identifier then they are bit-for-bit identical. This property gives us:

• Immutability. Changing the content produces a different identifier, which immediately signals tampering or corruption.
• Self-validation. Replicas can verify received data locally instead of trusting a third party.
• Stronger adversarial guarantees. Because an attacker must find collisions deliberately, intrinsic identifiers rely on cryptographic strength rather than purely statistical rarity.

These are guarantees of cryptographic content addresses, such as blob handles. They are not validity rules for the 128-bit entity ids produced by entity!: those remain opaque subject names, and their derivation serves idempotence, not authentication. An entity can acquire annotations without changing its id.

Extrinsic identifiers

Extrinsic identifiers (names, URLs, DOIs, UUIDs, UFOIDs, FUCIDs) are assigned by policy instead of by content. They track a conceptual entity as it evolves through versions, formats, or migrations. In other words, extrinsic identifiers carry the "story" of a thing while intrinsic identifiers nail down individual revisions.

Thinking about the classic ship of Theseus thought experiment makes the distinction concrete: the restored ship and the reconstructed ship share the same extrinsic identity (they are both "Theseus' ship") but have different intrinsic identities because their planks differ.

A note on signatures and public keys

Signatures are a special form of hash: a digest over (content, private_key). Used as identifiers they sit in the intrinsic-abstract cell — anyone holding the content and the author's public key can recompute and verify.

Public keys used as actor identities (identifying a person, agent, or service) are a different animal: a pubkey doesn't derive from the person it identifies — it's assigned by whoever generated the keypair. That makes pubkey-as-identity extrinsic abstract, essentially a high-entropy UUID with a bonus cryptographic capability.

Quadrant Properties

Classifying along two independent axes leaves four quadrants, and each has a structural property worth calling out because it constrains system design.

Extrinsic + Semantic (global) ⇒ an authority. If an identifier carries meaning, can't be derived from the entity, and has to be unique across all actors, someone had to assign that meaning globally. Semantic content is low-entropy by nature — the space of sensible names is small, so two parties minting independently will often collide — and avoiding collisions requires coordination. That coordinator is the authority, whether explicit (DOI registrar, ICANN, ISBN agency) or implicit. You can postpone this by adding scope prefix ("global tree of local ids" — DNS, Java packages, DOIs with publisher/journal structure) but the tree has a root, and the root is still an authority. The identifier also grows unboundedly as scopes stack: com.triblespace.core.repo.branch::metadata is already longer than some of the values it names.

Extrinsic + Semantic (local) is essential and free. The same property reverses once scope is bounded. A bibliography's [Herbert1965] is unique within the paper; a codebase's firstname is unique within its namespace. Local authority is trivial because it is the scope. Most of the human-facing vocabulary in any working system lives here, riding on top of abstract identifiers for global identity.

Extrinsic + Abstract can be fully decentralized. A 128-bit random identifier collides only statistically. Two parties minting UUIDs (or UFOIDs, or pubkeys) independently will never see each other's values in practice, so no coordinator is needed — the decentralization is paid for in entropy, not consensus.

Intrinsic + anything is decentralized by construction. The entity is the authority: any two parties with the same bytes produce the same id (for hashes) or the same neighborhood in embedding space (for embeddings). No registrar exists, because none is needed.

The upshot: if you want a decentralized naming system, you have two moves: use extrinsic abstract ids globally, and extrinsic semantic names locally. Don't promote semantic identifiers to global scope. Every centralized naming system in practice (domains, ISBNs, DOIs, academic affiliations) exists because it violated that rule.

Picking entity ids in code

TribleSpace lets you mix intrinsic and extrinsic identifiers depending on what you are modeling. The entity! macro mirrors that split:

#![allow(unused)]
fn main() {
use triblespace::examples::literature;
use triblespace::prelude::*;

// Intrinsic identity (default): the id is derived deterministically from the
// canonical fact rows, so identical record literals unify.
let record = entity! {
    literature::firstname: "Frank",
    literature::lastname: "Herbert",
};

// Extrinsic identity: supply an id expression when you want a stable subject
// whose facts can evolve across edits and commits.
let alice = ufoid();
let subject = entity! { &alice @
    literature::firstname: "Frank",
};

// `_ @` is an explicit synonym for the intrinsic form.
let also_intrinsic = entity! { _ @ literature::firstname: "Frank" };

// Optional facts: use `?:` with an `Option<T>` value to omit missing data
// without resorting to branching.
let maybe_alias: Option<&str> = None;
let with_optional = entity! { _ @
    literature::firstname: "Frank",
    literature::alias?: maybe_alias,
};

// Repeated facts: use `*:` with an `IntoIterator<Item = T>` to emit multiple
// facts for the same attribute.
let aliases = ["Frank", "F.H."];
let with_repeated = entity! { _ @
    literature::firstname: "Frank",
    literature::alias*: aliases,
};
}

For the intrinsic form, entity! first represents every encoded fact as NIL || attribute || value, using the ordinary 64-byte trible layout with an empty entity column. It sorts and deduplicates those rows, hashes the complete contiguous sequence with BLAKE3, takes the final 16 digest bytes as the entity id, and fills that id into every row. Identity therefore depends on the raw projected facts, not source order or Rust conversion history. An explicit entity id bypasses this derivation.

These two forms intentionally have different ownership types. An explicit subject must be supplied as an ExclusiveId, which is the capability to add facts to that extrinsic identity. An intrinsic entity instead exports its reproducible root as a plain Id.

The rows that participated in the hash are the entity's core: the minimal subset of facts that determines its identity. Everything else attached to it is annotation — facts that describe without defining. Annotating a core-derived entity is the ordinary case, not a violation: core fields determine the id, but the id does not determine the entity's facts. An entity that acquires facts over time is normal and needs no special treatment.

What the core must not become is a channel. Derivation exists to make idempotence easy, and for nothing else. It must never encode meaning. The test is mechanical:

Replace a derived id with a fresh genid. Everything must still work, except idempotence.

If anything else breaks, meaning has leaked into the id and the code is reading it back out — recomputing an id to look a thing up, or comparing a stored id against a re-derivation to check it. Both are hash joins wearing an Id, and both mean the entity they name was never written down. Write the entity, query it by its facts, and let the id go back to being a convenience.

Derivation is a producer-side naming convention, not a reader-side validity check. Re-deriving an id does not validate the surrounding facts, and readers must not reject a subject because its id differs from a locally recomputed value.

The canonical-row algorithm produces different new ids from the historical hash of concatenated attribute || value pairs. This changes cross-version idempotence: constructing the same core under different algorithms may produce two entities. Existing subjects and references to them remain valid opaque ids. Old, new, and randomly minted entity ids can coexist without re-ID or re-ingestion; queries find the stored entities by their facts, not by predicting their ids.

Other persistence boundaries still matter. A change to an attribute identity or a value/blob encoding may independently require additive migration or readers which understand both forms. Preserve those contracts explicitly; a changed entity-id derivation algorithm alone does not require rewriting source identities.

Embeddings as Semantic Intrinsic Identifiers

Embeddings blur our neat taxonomy. They are intrinsic because they are computed from the underlying data, yet they are overtly semantic because similar content produces nearby points in the embedding space. That duality makes them powerful for discovery:

• Systems can exchange embeddings as a "lingua franca" without exposing raw documents.
• Expensive feature extraction can happen once and power many downstream indexes, decentralizing search infrastructure.
• Embeddings let us compare otherwise incomparable artifacts (for example, a caption and an illustration) by projecting them into a shared space.

Despite those advantages, embeddings should still point at a durable abstract identifier rather than act as the identifier. Collisions are expected, model updates can shift the space, and floating-point representations can lose determinism across hardware.

High-Entropy Identifiers

For a truly distributed system, the creation of identifiers must avoid the bottlenecks and overhead associated with a central coordinating authority. At the same time, we must ensure that these identifiers are unique.

To guarantee uniqueness, we use abstract identifiers containing a large amount of entropy, making collisions statistically irrelevant. However, the entropy requirements differ based on the type of identifier:

• Extrinsic abstract identifiers need enough entropy to prevent accidental collisions in normal operation.
• Intrinsic abstract identifiers must also resist adversarial forging attempts, requiring significantly higher entropy.

From an information-theoretic perspective, the length of an identifier determines the maximum amount of entropy it can encode. For example, a 128-bit identifier can represent ( 2^{128} ) unique values, which is sufficient to make collisions statistically negligible even for large-scale systems.

For intrinsic identifiers, 256 bits is widely considered sufficient when modern cryptographic hash functions (e.g., SHA-256) are used. These hash functions provide strong guarantees of collision resistance, preimage resistance, and second-preimage resistance. Even in the event of weaknesses being discovered in a specific algorithm, it is more practical to adopt a new hash function than to increase the bit size of identifiers.

Additionally, future advances such as quantum computing are unlikely to undermine this length. Grover's algorithm would halve the effective security of a 256-bit hash, reducing it to ( 2^{128} ) operations—still infeasible with current or theoretical technology. As a result, 256 bits remains a future-proof choice for intrinsic identifiers.

Such 256-bit intrinsic identifiers are represented by the types Hash and Handle.

Not every workflow needs cryptographic strength. We therefore ship three high-entropy abstract identifier families—RNGID, UFOID, and FUCID—that keep 128 bits of global uniqueness while trading off locality, compressibility, and predictability to suit different scenarios.

Comparison of Identifier Types

"Predictability" here is a tradeoff axis, not a quality: higher predictability enables tighter compression and cache-friendly scans, but reveals mint metadata (time or source) and is therefore unsuitable whenever adversarial unpredictability matters. For those cases prefer RNGID's fully random bits, or step up to a 256-bit cryptographic Hash.

Example: Scientific Publishing

A published paper fuses several distinct identifier roles into a single centralized name (the DOI) — and a single centralized registry for authors (the ORCID). The quadrant framing suggests pulling these apart so each lives where it fits:

• Artifact identity → intrinsic-abstract. Identify each .html/.pdf by a cryptographic hash of its bytes. Any two parties referencing the same digest look at bit-for-bit identical content; verification is self-contained.
• Revision-group identity ("the same paper across revisions") → extrinsic-abstract. Mint a UFOID/FUCID at first publication and attach each later revision's content hash to it. Stable across rewrites, decentralizable, no registrar needed.
• Author identity → extrinsic-abstract, specifically the author's public key. A pubkey is extrinsic (you assign it by generating a keypair), abstract (the bits carry no meaning about the person), high-entropy (no collisions), and bundles a cryptographic capability: the author can sign the paper and anyone can verify it against their pubkey without a registrar. ORCID solves the same problem with central authority; pubkeys solve it with entropy plus cryptography.
• Human-readable labels → extrinsic-semantic, kept local. Citation keys ([Herbert1965]), reading-list tags, display titles, abbreviations in a bibliography — all semantic, all scoped to the document or reader that uses them. They're the human layer riding on top of the three abstract ids above, free to evolve and diverge across contexts without coordination.

DOIs land in the extrinsic-semantic, global-scope quadrant: a publisher prefix, a journal stem, often a human-recognizable slug — all assigned by a registrar and meant to be unique across every paper in the world. By the quadrant properties above that quadrant structurally requires a central authority, for collision avoidance (low semantic entropy across a global namespace) and for resolution (translating the id to a concrete artifact). DOIs-as-centralized isn't a design flaw; it's the price of asking one identifier to play the artifact-identity, revision-grouping, and human-label roles at global scope.

What triblespace recommends isn't "replace DOIs with something better in the same quadrant" — it's to decompose the role DOIs try to play into identifiers that each live in a quadrant they fit: content hashes for artifacts, abstract extrinsic ids (UFOID for revision groups, pubkeys for authors) for anything that needs global identity, and semantic labels (including DOIs and ORCIDs, when you need citation-compatibility with the outside world) as per-context references on top. Each role lands where decentralization is cheap or free.

ID Ownership

In distributed systems, consistency requires monotonicity due to the CALM principle ("Consistency As Logical Monotonicity" — any program that only grows its state can be eventually consistent without coordination; anything that can retract state requires coordination). However, this is not necessary for single-writer systems. By assigning each ID an owner, we ensure that only the current owner can write new information about an entity associated with that ID. This allows for fine-grained synchronization and concurrency control.

To create a transaction, you can uniquely own all entities involved and write new data for them simultaneously. Since there can only be one owner for each ID at any given time, you can be confident that no other information has been written about the entities in question.

By default, all minted ExclusiveIds are associated with the thread they are dropped from. These IDs can be found in queries via the local_ids function.

Once the IDs are back in scope you can either work with them directly as ExclusiveIds or move them into an explicit IdOwner for a longer lived transaction. The example below shows both approaches in action:

#![allow(unused)]
fn main() {
use triblespace::examples::literature;
use triblespace::prelude::*;

let mut kb = TribleSet::new();
{
    let isaac = ufoid();
    let jules = ufoid();
    kb += entity! { &isaac @
        literature::firstname: "Isaac",
        literature::lastname: "Asimov",
    };
    kb += entity! { &jules @
        literature::firstname: "Jules",
        literature::lastname: "Verne",
    };
} // `isaac` and `jules` fall back to this thread's implicit IdOwner here.

let mut txn_owner = IdOwner::new();
let mut updates = TribleSet::new();

for (author, name) in find!(
    (author: ExclusiveId, name: String),
    and!(
        local_ids(author),
        pattern!(&kb, [{
            ?author @ literature::firstname: ?name
        }])
    )
) {
    // `author` is an ExclusiveId borrowed from the implicit thread owner.
    let author_id = txn_owner.insert(author);

    {
        let borrowed = txn_owner
            .borrow(&author_id)
            .expect("the ID was inserted above");
        updates += entity! { &borrowed @ literature::lastname: name.clone() };
    } // `borrowed` drops here and returns the ID to `txn_owner`.
}
}

The entity! macro accepts ExclusiveIds by value or reference, so you can pass either an owned guard or a borrowed one.

Sometimes you want to compare two attributes without exposing the comparison variable outside the pattern. Prefixing the binding with _?, such as _?name, allocates a scoped variable local to the macro invocation. Both pattern! and pattern_changes! will reuse the same generated query variable whenever the _? form appears again, letting you express equality constraints inline without touching the outer find! signature.

Binding the variable as an ExclusiveId means the closure that find! installs will run the TryFromInline implementation for ExclusiveId. The conversion invokes Id::acquire and would silently skip the row if the current thread did not own the identifier (filter semantics). The local_ids constraint keeps the query safe by only enumerating IDs already owned by this thread, so no rows are filtered in practice. In the example we immediately move the acquired guard into txn_owner, enabling subsequent calls to IdOwner::borrow that yield OwnedIds. Dropping an OwnedId automatically returns the identifier to its owner so you can borrow it again later. If you only need the ID for a quick update you can skip the explicit owner entirely, bind the variable as a plain Id, and call Id::acquire when exclusive access is required.

Ownership and Eventual Consistency

While a simple grow set (like the signed members of a native collection) already constitutes a conflict-free replicated data type (CRDT), it is also limited in expressiveness. To provide richer semantics while guaranteeing conflict-free mergeability we allow only "owned" IDs to be used in the entity position of newly generated triples. As owned IDs are [Send] but not [Sync] owning a set of them essentially constitutes a single-writer transaction domain, allowing for some non-monotonic operations like if-does-not-exist over the set of contained entities. Note that this does not make operations that would break CALM (consistency as logical monotonicity) safe — e.g. delete.

The trible module defines the Trible struct, the smallest unit of knowledge the system stores. Instances of Tribles live inside TribleSets, which index each fact in several complementary ways so that queries can be answered with as little work as possible.

┌────────────────────────────64 byte───────────────────────────┐
┌──────────────┐┌──────────────┐┌──────────────────────────────┐
│  entity-id   ││ attribute-id ││        inlined value         │
└──────────────┘└──────────────┘└──────────────────────────────┘
└────16 byte───┘└────16 byte───┘└────────────32 byte───────────┘
─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─▶

At a high level a trible is a three-tuple consisting of an entity, an attribute, and a value. The entity and attribute are both 128‑bit abstract extrinsic identifiers as described in [triblespace::core::id], while the value is an arbitrary 256‑bit [triblespace::core::inline::Inline]. The value width deliberately matches the size of an intrinsic identifier so larger payloads can be referenced via blobs without inflating the inlined representation.

Abstract identifiers

Entities and attributes are purely extrinsic; their identifiers do not encode any meaning beyond uniqueness. An entity may accrue additional tribles over time and attributes simply name relationships without prescribing a schema. This keeps the format agnostic to external ontologies and minimises accidental coupling between datasets.

The value slot can carry any 256‑bit payload. Its size is dictated by the need to embed an intrinsic identifier for out‑of‑line data. When a fact exceeds this space the value typically stores a blob handle pointing to the larger payload.

Tribles are stored as a contiguous 64‑byte array with the entity occupying the first 16 bytes, the attribute the next 16, and the value the final 32 bytes. The name "trible" is a portmanteau of triple and byte and is pronounced like "tribble" from Star Trek – hence the project's mascot, Robert the tribble. This rigid layout keeps the representation friendly to SIMD optimisations and allows the storage layer to compute sizes deterministically.

Index permutations

TribleSets index each fact under all six permutations of entity (E), attribute (A) and value (V) so any combination of bound variables can be resolved efficiently. Regardless of which columns a query fixes the search can reach matching leaves with a handful of comparisons:

┌─────┐  ┌─────┐  ┌─────┐  ┌─────┐  ┌─────┐  ┌─────┐
│ EAV │  │ EVA │  │ AEV │  │ AVE │  │ VEA │  │ VAE │
└──┬──┘  └──┬──┘  └──┬──┘  └──┬──┘  └──┬──┘  └──┬──┘
   │        │        │        │        │        │
┌───────────────────────────────────────────────────────┐
│            order-specific inner nodes                 │
└───────────────────────────────────────────────────────┘ 
   │        │        │        │        │        │
   ▼        ▼        ▼        ▼        ▼        ▼

┌───────────────────────────────────────────────────────┐
│                   SHARED LEAVES                       │
│     single canonical E–A–V tribles used by all        │
└───────────────────────────────────────────────────────┘

Each permutation maintains its own inner nodes, but all six share leaf nodes containing the 64‑byte trible. This avoids a naïve six‑fold memory cost while still letting the search pick the most selective ordering, keeping joins resistant to skew even when cardinalities vary widely.

Advantages

• A total order over tribles enables efficient storage and canonicalisation.
• Simple byte‑wise segmentation supports indexing and querying without an interning mechanism, keeping memory usage low and parallelisation easy while avoiding the need for garbage collection.
• Schemas describe the value portion directly, making serialisation and deserialisation straightforward.
• The fixed 64‑byte layout makes it easy to estimate the physical size of a dataset as a function of the number of tribles stored.
• The minimalistic design aims to minimise entropy while retaining collision resistance, making it likely that a similar format would emerge through convergent evolution and could serve as a universal data interchange format.

Set operations and monotonic semantics

TribleSets provide familiar set-theoretic helpers such as TribleSet::union, TribleSet::intersect and TribleSet::difference. union consumes the right-hand operand and merges its contents into the receiver in place, while intersect and difference each produce a fresh TribleSet without mutating their inputs. Together these helpers make it straightforward to merge datasets, locate their overlap or identify the facts that still need to propagate between replicas while keeping the original sources intact.

This design reflects the crate's commitment to CALM-friendly, monotonic semantics. New information can be added freely, but existing facts are never destroyed. Consequently, difference is intended for comparing snapshots (e.g. "which facts are present in the remote set that I have not yet indexed?") rather than for destructive deletion. This keeps workflows declarative and convergent: sets can be combined in any order without introducing conflicts, and subtraction simply reports the gaps that remain to be filled.

Direction and consistency

In many triple stores the direction of an edge is chosen incidentally—there is no intrinsic preference for hasColor over colorOf. This ambiguity often leads to confusion, duplication, or both as different writers pick different conventions. Common mitigations either mirror every edge automatically (as done by OWL and RDF through inverseOf, doubling storage or demanding runtime inference) or devolve into bikeshedding about the "correct" orientation.

tribles avoids that trap by giving edge direction explicit semantics: the arrow points from the entity making the claim to the entity being described. The observer owns the identifier and is responsible for the consistency of the facts it asserts—see ID Ownership. This rule naturally fits the distributed setting where each entity has a single authoritative writer.

Viewed another way, edges always flow from describing to described entities, while cycles represent consensus between the parties involved. For example, hasColor must point from the object that exhibits the colour to the entity representing that colour. The orientation is therefore a consequence of the statement's meaning, not an arbitrary modelling choice.

Query Language

This chapter introduces the core query facilities provided by triblespace. A query is described in a small declarative language that states which values should match instead of spelling out the iteration strategy. When you read a query, you are effectively looking at a logical statement about the data: if the constraints can all be satisfied, then the variable bindings are produced as results. The declarative style gives the engine freedom to reorder work and choose efficient execution strategies.

Every macro shown here is a convenience wrapper around a concrete Constraint implementation. When you need finer control—or want to assemble constraints manually outside the provided macros—reach for the corresponding builder types in triblespace::core::query.

Declaring a query

The find! macro builds a Query by declaring variables and a constraint expression. The macro mirrors Datalog syntax: the head ((...)) lists the variables you want back, and the body describes the conditions they must meet. A minimal invocation looks like this:

let results = find!((a), a.is(1.into())).collect::<Vec<_>>();

find! returns an Iterator over the bound variables. Matches can be consumed lazily or collected into common collections.

The head is an ordered projection with BAG semantics. The engine emits one row every time a complete binding is found, and the head selects which of its variables you get back. Hidden variables therefore multiply: an assignment proved by eight different witnesses is emitted eight times. Deduplication is the consumer's job — collect into a HashSet, or ask the question with exists! so the fan-out is never enumerated. The Query Engine chapter explains why the engine does not deduplicate for you.

When the head declares a single variable, omit the parentheses to get bare values instead of 1-tuples:

for a in find!(a, a.is(1.into())) {
    println!("match: {a}");
}

When the head declares multiple variables, wrap them in parentheses to get tuples:

for (a, b) in find!((a, b), and!(a.is(1.into()), b.is(2.into()))) {
    println!("{a}, {b}");
}

Adding more variables is as simple as expanding the list: find!((a, b, c), ...) yields (a, b, c) tuples. Variables declared in the head can be reused multiple times inside the constraint to express joins. When a variable appears in several clauses the engine ensures every occurrence binds to the same value. Repeating a variable in two patterns, for example, restricts the result set to entities that satisfy both attribute assignments simultaneously. The order of declarations defines the shape of the tuple in the iterator, so reorganising the head changes how you destructure results.

Typed variables

Variables optionally include a concrete type to convert the underlying value. The constraint phase still works with untyped Inline instances; conversion happens when results are emitted. These conversions use TryFromInline.

By default, if a conversion fails the entire row is silently skipped — like a constraint that doesn't match. For types whose TryFromInline::Error is Infallible the error branch is dead code and no rows can ever be accidentally filtered.

Append ? to a variable to receive the raw Result<T, E> instead. Both Ok and Err values pass through without filtering, matching Rust's ? semantics of "bubble the error to the caller."

// `x` is filtered (rows where conversion fails are skipped).
// `y` is passed through as Result (no filtering).
find!((x: i32, y: Inline<Handle<UTF8String>>?),
      and!(x.is(1.into()), y.is("foo".to_inline())))

The query engine explores assignments that satisfy the constraint and yields the declared variables in head order, one row per satisfying assignment. Variables omitted from the head still participate in the search — they decide whether an assignment exists, and each distinct value they take is a separate row. A repeated variable in the head is rejected because it would not add a new projected column.

The empty head find!((), constraint) therefore yields one () per satisfying assignment, which makes find!((), ...).count() a way to count them. When you only want to know whether any assignment exists, use exists!: it stops at the first one instead of draining the fan-out.

Collecting results

Any type that implements FromIterator can collect the results of a query. Vec<_> is common for tests and examples, while HashSet<_> is useful when the match order is irrelevant. When you only need the first result, call iterator adapters such as next, find, or try_fold to avoid materializing the full result set.

Built-in constraints

find! queries combine a small set of constraint operators to form a declarative language for matching tribles. Each operator implements Constraint and can therefore be mixed and nested freely.

Any data structure that can iterate its contents, test membership, and report its size can implement ContainsConstraint. Membership constraints are particularly handy for single-column collections such as sets or map key views, while multi-position sources like TribleSet rely on pattern! to keep entity, attribute, and value bindings aligned.

Constant matches (is)

Call Variable::is when you need a binding to equal a specific value. The method returns a ConstantConstraint that checks whether the solver can assign the variable to the provided Inline. Constant constraints behave like any other clause: combine them with and! to narrow a variable after other constraints have proposed candidates, or place them inside or! branches to accept multiple literals.

find!((title: Inline<_>),
      and!(dataset.has(title), title.is("Dune".to_inline())));

The snippet above keeps only the rows where title equals "Dune". Because is constrains the variable's value rather than projecting a new binding, it is also handy for helpers such as temp! when you want to filter hidden bindings without exposing them in the result tuple.

pattern! and pattern_changes! fold literal values (and attribute constants) directly into the pattern constraint as constant Terms — no variable is allocated for them — so you often get the same behaviour simply by writing the desired value in the pattern:

find!((friend: Inline<_>),
      pattern!(&dataset,
               [{ _?person @ social::friend: ?friend,
                  social::city: "Caladan" }]));

Repeating .is(...) on the same variable with different values causes the query to fail—just as conflicting pattern! clauses would—so prefer or! (or switch to a membership helper such as .has(...)) when you want to accept several constants.

Intersections (and!)

and! combines multiple constraints that must all hold simultaneously. Each sub-clause can introduce new bindings or further narrow existing ones, and the solver is free to reorder the work to reduce the search space. When a sub-constraint fails to produce a candidate that is compatible with the current bindings, the whole conjunction rejects that branch and moves on. The macro accepts any number of arguments, so and!(...) is often a convenient way to keep related clauses together without nesting additional find! calls:

let favourites = favourite_titles(); // e.g. a HashSet<Id> built elsewhere
find!((book: Inline<_>, author: Inline<_>),
      and!(favourites.has(book),
           pattern!(&dataset,
                    [{ ?book @ literature::title: "Dune",
                       literature::author: ?author }])));

Here the membership test over favourites and the attribute pattern from dataset run as part of the same conjunction. The solver joins them on their shared bindings (book and author) so only tuples that satisfy every clause make it into the result set. Because and! simply returns a constraint, you can nest it inside other combinators such as temp! or or! to structure queries however you like.

Alternatives (or!)

Use or! to express alternatives. Each branch behaves like an independent constraint and may introduce additional bindings that participate in the surrounding query, provided every branch mentions the same set of variables:

find!((alias: Inline<_>),
      temp!((entity),
            or!(pattern!(&dataset,
                         [{ ?entity @ profile::nickname: ?alias }]),
                pattern!(&dataset,
                         [{ ?entity @ profile::display_name: ?alias }]))));

Each branch contributes every match it can produce given the current bindings. Results are a bag of complete bindings: one row per witness of the declared variables (here entity and alias), so a nickname and display name with different raw values contribute two rows, equal values for the same entity collapse (the union is a set per binding step), and the same alias witnessed by different entities yields one row per entity — dedup of projected columns belongs to the consumer. Branches that cannot match simply contribute nothing.

All branches of an or! must bind exactly the same set of variables; branch-local variables are not supported. This is a consequence of the engine's flat result schema — every result row binds the same variable set exactly once, so there is no way to represent a variable that only exists in some alternatives. (It is not a semantic or monotonicity limitation: the union itself is monotonic.) Attribute constants and literal values do not count towards a branch's variable set: pattern! folds them into the constraint as constant Terms rather than allocating hidden variables, which is what lets the branches above use different attributes (nickname vs. display_name) while still declaring the identical set {entity, alias}. Only genuine query variables must align: if two branches reference different variables the constraint panics at construction time, naming the mismatched sets. Note that an anonymous entity ({ attr: ?v } without an ?entity @ id) introduces a fresh variable scoped to its own pattern!, so bind entities explicitly — as the example does with temp!((entity), ...) — when combining such patterns with or!.

Temporary variables (temp!)

Real queries often need helper bindings that participate in the joins but do not show up in the result tuple. Wrap the relevant constraint with temp!((...vars...), expr) to mint hidden variables and evaluate expr with them in scope:

find!((person: Inline<_>),
      temp!((friend),
            and!(pattern!(&dataset,
                          [{ _?p @ social::person: ?person, social::friend: ?friend }]),
                 pattern!(&dataset,
                          [{ ?friend @ social::city: "Caladan" }]))));

The helper binding friend links the two patterns, ensuring the same entity is used across both clauses without expanding the result tuple. temp! can create multiple variables at once (temp!((street, city), ...)). You always wrap the hidden bindings in a tuple, so each invocation reads temp!((...vars...), ...). Here social would be a namespace module exporting the person, friend, and city attributes. The variables adopt the value schemas implied by the constraints they appear in, so no extra annotations are required. When working outside the query macros, call VariableContext::next_variable directly instead.

Temporary variables are hidden from the result tuple, but they are not hidden from the search: several friends that prove the same projected person produce that person several times. Collect into a set when you want each person once, or restructure the query so the inner condition is an exists! check and the fan-out is never enumerated at all.

When the helper variable lives entirely within a single pattern, consider using _?alias instead of temp!. Both pattern! and pattern_changes! support _?ident placeholders that mint fresh bindings scoped to that one macro invocation. They behave like non-projected variables: you can reuse the _?ident multiple times inside the pattern to relate attributes, but the binding vanishes once control leaves the macro. Reach for temp! when the helper must span several constraints or when you need to reuse the same hidden binding across multiple patterns.

Example

use triblespace::prelude::*;
use triblespace::core::examples::{self, literature};

let dataset = examples::dataset();

for (title,) in find!((title: Inline<_>),
                     and!(dataset.has(title), title.is("Dune".to_inline()))) {
    println!("Found {}", title.from_inline::<&str>());
}

This query searches the example dataset for the book titled "Dune". The variables and constraint can be adapted to express more complex joins and filters. For instance, you can introduce additional variables to retrieve both the title and the author while sharing the same dataset predicate:

for (title, author) in find!((title: Inline<_>, author: Inline<_>),
                             and!(title.is("Dune".to_inline()),
                                  pattern!(&dataset,
                                           [{ _?book @ literature::title: ?title,
                                              literature::author: ?author }]))) {
    println!("{title} was written by {}", author.from_inline::<&str>());
}

The extra variables participate in the join automatically; no explicit loop nesting or indexing is required.

Attribute patterns (pattern!)

The pattern! macro provides a concise way to match entities by attribute assignments. It expands to a constraint that can be used directly inside find!.

Important: in pattern! values prefixed with ? refer to variables declared in the surrounding find! head while string/number literals and more complex expressions are treated as literal values. Use _?name when you need a fresh variable that is scoped to a single macro invocation; you can reference it multiple times within the same pattern without adding it to the find! head. Parenthesised expressions remain supported for explicit literals.

#![allow(unused)]
fn main() {
use triblespace::prelude::*;
mod literature {
    use triblespace::prelude::*;
    attributes! {
        "0DBB530B37B966D137C50B943700EDB2" as firstname: inlineencodings::Handle<blobencodings::UTF8String>;
        "6BAA463FD4EAF45F6A103DB9433E4545" as lastname: inlineencodings::Handle<blobencodings::UTF8String>;
    }
}
let mut kb = TribleSet::new();
let e = ufoid();
kb += entity! { &e @ literature::firstname: "William", literature::lastname: "Shakespeare" };

let results: Vec<_> = find!((ee: Id), pattern!(&kb, [{ ?ee @ literature::firstname: "William" }])).collect();
assert_eq!(results.len(), 1);
}

Patterns may contain multiple clauses and reuse _? bindings to relate attributes without introducing extra columns in the result set. A single _?person variable can connect several attribute/value pairs while staying scoped to the pattern:

#![allow(unused)]
fn main() {
use triblespace::prelude::*;
mod literature {
    use triblespace::prelude::*;
    attributes! {
        "0DBB530B37B966D137C50B943700EDB2" as firstname: inlineencodings::Handle<blobencodings::UTF8String>;
        "6BAA463FD4EAF45F6A103DB9433E4545" as lastname: inlineencodings::Handle<blobencodings::UTF8String>;
    }
}
let mut kb = TribleSet::new();
let e = ufoid();
kb += entity! { &e @ literature::firstname: "Frank", literature::lastname: "Herbert" };

let author_last_names: Vec<_> = find!((last: Inline<_>),
    pattern!(&kb, [{ _?person @ literature::firstname: "Frank", literature::lastname: ?last }])
).collect();
}

Here _?person remains scoped to the pattern while ensuring both attributes are drawn from the same entity. When a pattern references collections other than a TribleSet, ensure the collection implements TriblePattern so that the macro can materialize the requested triples.

To share a hidden binding across multiple patterns, declare it once with temp! and reference it with ?name from each clause:

#![allow(unused)]
fn main() {
use triblespace::prelude::*;
mod social {
    use triblespace::prelude::*;
    attributes! {
        "A19EC1D9DD534BA9896223A457A6B9C9" as name: inlineencodings::Handle<blobencodings::UTF8String>;
        "C21DE0AA5BA3446AB886C9640BA60244" as friend: inlineencodings::GenId;
    }
}
let mut kb = TribleSet::new();
let alice = ufoid();
let bob = ufoid();
kb += entity! { &alice @ social::name: "Alice", social::friend: &bob };
kb += entity! { &bob @ social::name: "Bob" };

let results: Vec<_> = find!((friend_name: Inline<_>),
    temp!((friend),
          and!(pattern!(&kb, [{ _?person @ social::friend: ?friend,
                                  social::name: ?friend_name }]),
               pattern!(&kb, [{ ?friend @ social::name: "Bob" }]))))
.collect();
}

The _?person variable is still local to the first pattern, while friend joins the two constraints without changing the projected results. As above, social denotes a namespace that defines the name and friend attributes.

exists!

Sometimes you only want to check whether a constraint has any solutions. The exists! macro mirrors the find! syntax but returns a boolean:

use triblespace::prelude::*;

assert!(exists!((x), x.is(1.into())));
assert!(!exists!((x), and!(x.is(1.into()), x.is(2.into()))));

Internally, exists! stops as soon as the first result is found. It is a lightweight alternative to find! when the mere existence of a match matters more than the actual bindings.

Custom constraints

Every building block implements the Constraint trait. You can implement this trait on your own types to integrate custom data sources or query operators with the solver. Collections that want to power pattern! implement TriblePattern so they can materialize the entity/attribute/value triples a pattern asks for. Membership-style helpers such as has(...) work with anything that implements ContainsConstraint, making it easy to join against pre-existing indexes, caches, or service clients without copying data into a TribleSet.

use std::collections::HashSet;

use triblespace::prelude::*;
use triblespace::prelude::blobencodings::UTF8String;
use triblespace::prelude::inlineencodings::Handle;
use triblespace::core::query::hashsetconstraint::SetConstraint;

type Tag = Handle<UTF8String>;

struct ExternalTags<'a> {
    tags: &'a HashSet<Inline<Tag>>,
}

impl<'a> ContainsConstraint<'a, Tag> for ExternalTags<'a> {
    type Constraint = SetConstraint<Tag, &'a HashSet<Inline<Tag>>, Inline<Tag>>;

    fn has(self, variable: Variable<Tag>) -> Self::Constraint {
        SetConstraint::new(variable, self.tags)
    }
}

let tags: HashSet<Inline<Tag>> = ["rust", "datalog"].into_iter().map(|tag| tag.to_inline()).collect();
let external = ExternalTags { tags: &tags };
let matches: Vec<_> =
    find!((tag: Inline<Tag>), external.has(tag)).collect();

The example wraps an external HashSet so it can be queried directly. A TriblePattern implementation follows the same shape: create a constraint type that reads from your backing store and return it from pattern. The query engine drives both traits through Constraint, so any data source that speaks the protocol can participate in find!. Four methods are required:

propose and confirm take a Frontier — a batch of parent bindings, of which a single binding is the width-1 case. Loop over Frontier::rows, calling ProposalBuffer::open(row) before each row's candidates; on the confirm side, ignore the parent tags if your verdict does not depend on the binding, or walk the region with Candidates::for_each_parent if it does.

Two more methods have defaults you can override: satisfied (defaulting to true) and influence (defaulting to "every variable I touch except this one").

The rules a custom constraint has to respect are short:

• estimate is a cost quote. It steers variable ordering and nothing else. A wrong estimate makes the search slower, never incorrect. None means "not my variable", not "no candidates".
• propose only appends. Entries already in the buffer belong to a sibling constraint in an enclosing composite; leave them alone. Within one chunked enumeration, never deliver the same value twice — a duplicate inflates row multiplicity.
• confirm only kills. It may never add a candidate or revive a dead one, and it may skip entries that are already dead. This is what lets several confirmers write into the same region in any order, or in parallel, and still compute their conjunction.
• satisfied may be optimistic, but only upward. Returning true while a relevant variable is unbound is fine; returning false must mean there is genuinely no completion. Once every variable the constraint touches is bound, the answer must be exact — or! relies on that to discard dead alternatives, and a fully constant constraint is settled by a single satisfied call at construction with no search to correct it later.

The Query Engine chapter explains the protocol, the search that drives it, and the reasoning behind these rules in detail.

Frontiers (latest)

A recurring modelling shape is the same thing, changing over time: a set of immutable states, each naming the states it observed, with reads wanting the ones nothing has moved past yet. Because the observer owns the identifier it writes under (see Direction and consistency), the edge always runs successor-to-predecessor — "I observed that" is a claim about your own new entity, whereas "I replace that" would be a claim about someone else's. metadata::supersedes is the published attribute for it.

The pure utility answers "which of these candidates has no known successor" with reverse-index probes. It lives in the query layer as latest:

use triblespace::prelude::*;
use triblespace::core::metadata;

let heads = latest(&facts, metadata::supersedes.id(), candidates);

Three things are worth stating explicitly, because each one is a place where designs usually go wrong:

• There is no global "current". There is a current state for a given set of commits, which is exactly what a collection view is. Two readers holding different commit sets legitimately disagree, and that is frame-relativity, not a consistency bug. The facts argument is that frame.
• A maintained head set needs its evidence. For a fixed transitive partial order, antichains join by taking maxima of their union. Here the order evidence arrives with the source facts: bare opaque head ids do not retain discarded ancestry. The maintained form below therefore retains historical superseded targets beside known live heads, and is monotone under that pair's join order.
• The predicate is local. s is maximal in C exactly when no state in C observes s — note "in C", not "in the frontier". Anything that observes s already dominates it, so immediate edges suffice: no transitive closure, no reachability query, and no vector clock. Each candidate costs one short-circuited reverse-index probe, and the answer cannot depend on the order states arrived in.

The answer is a set. Concurrent states are a genuine fork, and collapsing them to one would invent an order the data does not have.

Registers: the general form

latest hard-codes two choices — order by the observation edge, report the whole antichain — and both are parameters in query::register. latest is now just the multi-value reading of that substrate over an observation DAG.

The useful thing to see first is that a resolution policy is not a second knob beside the order. It is a choice of order:

There is one operation — take the maximal elements — and sole is a check that the order left exactly one, never a tie-break that invents one.

Two axes are parameterised, and the second is smaller than it looks:

• Order. ObservationOrder reads the DAG, as latest does. StatedOrder reads an identity attribute and an order attribute — a wall-clock timestamp, a counter, a version number, plus the thing it is a measurement of. Order values are compared as raw inline bytes, which is value order exactly when the encoding is order-preserving (NsTAIInterval, I256BE, ROrd256 — deliberately not R256); this is the same contract [value_range] carries, and the wide half of every comparison is pushed into the engine through it.
• End. .first() resolves to the minimum. min is the join of the opposite order, so first-write-wins is as lawful a derivation as last-write-wins.

There is no scope axis, and there was briefly one, which is worth saying plainly because it is an easy mistake to make twice.

A register is a set of states that are versions of the same thing, ordered. That is two facts. An observation edge asserts both at once — "I observed that" says same-thing and later in one breath — which is why the DAG order takes a single attribute and needs no scope. A timestamp asserts only one of them: it says when, never of what. So a stated order must be told the identity, and if it is not, the missing half comes back as a filter.

Compass hangs status events, notes, and priority events off a goal through the same board::task edge, all timestamped. board::task means belongs to this goal — not an identity: a note is not a later version of a status event. Read as one, a note at t=20 dominated a status event at t=10 and the goal reported no status at all, on 778 of 2939 live goals. The patch was an .among(tag, STATUS) knob narrowing who may dominate — which is (goal, status-kind), a composite identity reconstructed at the call site out of a grouping plus a type filter, because the real one was not in the data.

The cure is to put it in the data: an attribute meaning the status of goal G. Then nothing needs narrowing, because a note is not in that register to begin with.

Which attribute carries identity and which carries order is a property of the register, not of the reader's question. Both are carried as tribles on one concrete mapping entity embedded in the derived collection descriptor. That entity separately names the stable mapping algorithm, so its content-derived identity includes the concrete attribute pair without making the pair opaque. Two registers over one dataset are therefore distinct mappings and distinct collections -- their descriptors differ in content, so they differ in handle -- and cannot share a cache. What is left at the call site is the frame, and nothing else.

An earlier design folded concrete parameters into an otherwise opaque algorithm id. The digest became their only carrier, so no reader could recover which attributes a register was over. The current shape stores parameters on the mapping entity itself; latest, lww_register, and other derived collections all use the same content-derived mechanism.

An order composes into a query directly. maximal is a filter-only constraint: it estimates usize::MAX so the planner always sorts it last and a pattern! proposes the scope, which is what relieves the caller of materialising candidates.

use triblespace::prelude::*;

// The register: identified by `status_of`, ordered by `created_at`.
let order = StatedOrder::<_, inlineencodings::NsTAIInterval>::new(
    &facts, board::status_of.id(), metadata::created_at.id(),
)
.tiebreak_by_id();

let current: Vec<Id> = find!(
    event: Id,
    and!(
        pattern!(&facts, [{ ?event @ board::status_of: &goal }]),
        maximal(event, &order),
    )
).collect();

When an exact cardinality is wanted instead — so the planner can order around resolution rather than after it — materialise with resolve and propose from a SortedSlice, whose estimate is the resolved count itself.

The maintained form

collection::latest maintains an ordinary latest-state lattice. LatestBlob stores (H, D): known live states and all historically superseded targets, including targets whose own facts have not arrived. Its canonical join is

(H₁, D₁) ⊔ (H₂, D₂) = ((H₁ ∪ H₂) ∖ (D₁ ∪ D₂), D₁ ∪ D₂)

Leaf derivation projects every source subject into S and every well-formed target of the configured observation attribute into D, then stores (S ∖ D, D). Both projections preserve source union. They do not require a multi-field state shape, whose fields might arrive in separate commits. derive(C₁ ∪ C₂) = derive(C₁) ⊔ derive(C₂) therefore holds byte-exactly.

Historical D is necessary: after b → a, c → b leaves only c live, a later arrival of a must still be suppressed. Cycles are deterministic too: every observed cycle member is retired; no DAG validation is imposed. The head projection is neither monotone nor antitone under inclusion. The maintained pair is monotone under its own join order.

LatestIndex exposes H as ordinary positive membership, with exact cardinality, complete proposals, and binary-search confirmation:

let index: LatestIndex = snapshot.collection(latest_collection)?.view()?;
let heads: Vec<Id> = find!(state: Id, index.has(state)).collect();
let current: Vec<Id> = find!(state: Id, and!(
    index.has(state),
    pattern!(&facts, [{ ?state @ metadata::tag: revision_kind }]),
)).collect();

Unknown candidates are excluded, unlike the pure negative register utilities. Facts and latest may advance independently: an unobserved new state stays invisible, while an already-known old state can remain visible until latest advances. There is no shared-support requirement or global-current promise. Queries are pure reads of frozen observations; ordinary ensure/maintain advances the collection outside the query.

collection::lww_register is the maintained counterpart for a stated last-write-wins register. Its target element keeps the identity and order fact halves in separate canonical row sets. That detail is essential: the two facts for one state may arrive in different source commits, so deriving only already-complete coordinates would not commute with source union. LwwIndex attaches those row sets without copying them. Its explicit query() step pairs them across the cover and returns a LwwQuery selecting the greatest (order, state-id) coordinate for every register. The prepared query's .has(state) constraint positively proposes and confirms only known complete winners, excluding unknown and incomplete states. Its existing RegisterOrder implementation remains available for pure order-based reads.

The maintained LWW form makes one validity contract explicit: within an exact cover, a state which has both halves has at most one well-formed identity and at most one order value under the descriptor's attributes. Incomplete states remain incomparable even if the present half is multivalued. The row sets retain those values so a later counterpart exposes rather than hides a conflict, including one split across source commits. Order bytes must use an order-preserving encoding, just as they must for the live StatedOrder.

Recursive traversal

Queries in this chapter all have a fixed number of clauses, which means a fixed number of hops. Genuinely recursive questions — "everyone reachable through a chain of follows", "all ancestors via repeated parent" — cannot be written this way.

Earlier versions of the crate answered them with a path! macro that evaluated a regular expression over edge attributes inside the query engine. That macro and its evaluator have been removed: query-time traversal needs per-activation state, which the stateless constraint protocol has no place for, and keeping it inside the protocol meant every constraint paid for machinery only paths used.

The stable replacement is the standalone triblespace-paths crate. Its PathExpr builder describes sequence, alternatives, repetition, optionality, and inverse steps, then compiles them to a fixed epsilon-free automaton. The crate combines that automaton with graph edges to materialize an accepted endpoint relation, exposed through an ordinary two-variable constraint. That constraint composes directly with find!, and!, and pattern!; see Regular Path Indexes for expression construction, the low-level automaton escape hatch, and exact collection maintenance.

For a fixed small number of hops, explicit pattern clauses joined on temp! variables remain simpler. For unbounded traversal, use the path index or drive a one-off frontier search from application code when materializing a potentially dense endpoint relation would not pay for itself.

Macro Cookbook

This chapter is a quick map of the macro surface. The goal is not to replace the deeper chapters, but to make the everyday question easy:

"I know roughly what I want to do. Which macro should I reach for?"

The macros fall into three layers:

• Encoding definition: attributes!
• Fact construction: entity!
• Query construction: find!, exists!, pattern!, pattern_changes!, and!, or!, temp!

Define attributes with attributes!

Use attributes! to declare typed attributes once, then reuse them everywhere else.

#![allow(unused)]
fn main() {
use triblespace::prelude::*;

mod social {
    use triblespace::prelude::*;
    use triblespace::prelude::inlineencodings::{GenId, Handle};

    attributes! {
        /// A person's display name.
        "A74AA63539354CDA47F387A4C3A8D54C" as pub name: Handle<blobencodings::UTF8String>;

        /// Another person this person knows.
        pub friend: GenId;
    }
}

assert_ne!(social::name.id(), social::friend.id());
}

The three declaration forms differ in what determines identity:

• "HEX_ANCHOR" as name: Encoding derives the id from the stable anchor and encoding. This is the safe default for shared schemas: renaming the Rust binding is free, while changing the encoding creates a different attribute.
• name: Encoding derives the id from the name and encoding. It is convenient when the source name itself is the shared identifier.
• "HEX_ID" unsafe as name: Encoding uses the literal id without incorporating the encoding. Reserve it for compatibility with an already-published id; the caller must ensure the type still matches the stored rows.

Reach for this macro when:

• you are defining a namespace or encoding module
• you want attributes with stable ids and inline encodings
• you want doc comments to become attribute metadata

If you already have attributes, you usually do not need attributes! in the rest of the code you are writing.

Build facts with entity!

Use entity! when you want to create tribles for one entity.

#![allow(unused)]
fn main() {
use triblespace::prelude::*;
mod social {
    use triblespace::prelude::*;
    use triblespace::prelude::inlineencodings::Handle;
    attributes! {
        "A74AA63539354CDA47F387A4C3A8D54C" as pub name: Handle<blobencodings::UTF8String>;
    }
}
let alice = fucid();
let facts = entity! { &alice @
    social::name: "Alice",
};

assert_eq!(facts.root(), Some(alice.id));
assert_eq!(facts.len(), 1);
}

If you omit the entity id, entity! derives one deterministically from the facts. It encodes each fact as a 64-byte NIL || attribute || value row, sorts and deduplicates the rows, hashes their complete contiguous byte sequence with BLAKE3, and uses the final 16 digest bytes as the entity id. It then fills that id into every row before constructing the TribleSet. Field order and duplicate repeated values therefore do not affect identity; an absent optional field contributes no row.

#![allow(unused)]
fn main() {
use triblespace::prelude::*;
mod social {
    use triblespace::prelude::*;
    use triblespace::prelude::inlineencodings::Handle;
    attributes! {
        "A74AA63539354CDA47F387A4C3A8D54C" as pub name: Handle<blobencodings::UTF8String>;
    }
}
let facts = entity! { _
    @ social::name: "Alice"
};

assert!(facts.root().is_some());
assert_eq!(facts.len(), 1);
}

The macro also supports optional and repeated fields:

let aliases = ["Al", "A."];
let maybe_nickname = Some("Ace");

let facts = entity! { &alice @
    social::name: "Alice",
    social::nickname?: maybe_nickname,
    social::alias*: aliases,
};

Reach for this macro when:

• you are constructing new data
• you want optional/repeated attribute ergonomics
• you want a deterministic derived id for a value object

Match facts with pattern!

Use pattern! to turn trible-shaped structure into a query constraint.

#![allow(unused)]
fn main() {
use triblespace::prelude::*;
mod social {
    use triblespace::prelude::*;
    use triblespace::prelude::inlineencodings::{GenId, Handle};
    attributes! {
        "A74AA63539354CDA47F387A4C3A8D54C" as pub name: Handle<blobencodings::UTF8String>;
        "B74AA63539354CDA47F387A4C3A8D54C" as pub friend: GenId;
    }
}
let mut kb = TribleSet::new();
let alice = fucid();
let bob = fucid();
kb += entity! { &alice @ social::friend: &bob };
kb += entity! { &bob @ social::name: "Bob" };
let results: Vec<Id> = find!(
    friend: Id,
    pattern!(&kb, [
        { alice.id @ social::friend: ?friend },
        { ?friend @ social::name: "Bob" }
    ])
).collect();

assert_eq!(results, vec![bob.id]);
}

Inside a pattern:

• ?name refers to a query variable from the surrounding query
• _?name introduces a local helper variable scoped to that pattern
• literal expressions become equality constraints automatically

Use pattern! when you are querying the current contents of a TribleSet, Checkout, or another pattern-capable source.

Query for results with find!

Use find! when you want rows back.

#![allow(unused)]
fn main() {
use triblespace::prelude::*;
mod social {
    use triblespace::prelude::*;
    use triblespace::prelude::inlineencodings::Handle;
    attributes! {
        "A74AA63539354CDA47F387A4C3A8D54C" as pub name: Handle<blobencodings::UTF8String>;
    }
}
let mut kb = TribleSet::new();
let alice = fucid();
kb += entity! { &alice @ social::name: "Alice" };
let names: Vec<Inline<_>> = find!(
    name: Inline<_>,
    pattern!(&kb, [{ _?person @ social::name: ?name }])
).collect();

// A text value is the content handle of its blob: compare it with the
// handle of a string, or read the text from the blob store that holds it.
assert_eq!(names[0], "Alice".to_inline());
}

There are three common shapes:

• find!(value, constraint) for one projected variable as a bare value
• find!((a, b), constraint) for tuples
• find!((), constraint) when you want to count satisfying assignments without projecting any of them (reach for exists! if you only need yes/no)

find! heads have BAG semantics: one row per complete binding, with the head selecting which variables come back. Hidden variables introduced by temp! or _?name still multiply the result — an entity proved by eight witnesses is emitted eight times, and the empty head returns one () per satisfying assignment. Deduplicate on your side (collect::<HashSet<_>>()), or use exists! for the inner condition so the fan-out is never enumerated.

Typed projections happen in the head:

#![allow(unused)]
fn main() {
use triblespace::prelude::*;
mod social {
    use triblespace::prelude::*;
    use triblespace::prelude::inlineencodings::Handle;
    attributes! {
        "A74AA63539354CDA47F387A4C3A8D54C" as pub name: Handle<blobencodings::UTF8String>;
    }
}
let mut kb = TribleSet::new();
let alice = fucid();
kb += entity! { &alice @ social::name: "Alice" };
let ids: Vec<_> = find!(
    person: Id,
    pattern!(&kb, [{ ?person @ social::name: "Alice" }])
).collect();

assert_eq!(ids, vec![alice.id]);
}

Use ? on a projected variable when you want conversion failures as Result<T, E> instead of dropping the row.

Ask existence questions with exists!

Use exists! when you only need yes/no.

#![allow(unused)]
fn main() {
use triblespace::prelude::*;
mod social {
    use triblespace::prelude::*;
    use triblespace::prelude::inlineencodings::Handle;
    attributes! {
        "A74AA63539354CDA47F387A4C3A8D54C" as pub name: Handle<blobencodings::UTF8String>;
    }
}
let mut kb = TribleSet::new();
let bob = fucid();
kb += entity! { &bob @ social::name: "Bob" };
let has_bob = exists!(
    pattern!(&kb, [{ _?person @ social::name: "Bob" }])
);

assert!(has_bob);
}

You can also keep the typed-head form when the projection itself matters to the check:

let has_name = exists!(
    (name: Inline<_>),
    pattern!(&kb, [{ ?person @ social::name: ?name }])
);

Use exists!(constraint) for pure existence checks instead of find!((), constraint).next().is_some().

Match only new results with pattern_changes!

Use pattern_changes! for incremental queries: matches are allowed to join against the full current state, but at least one contributing trible must come from the change set.

for (work,) in find!(
    (work: Inline<_>),
    pattern_changes!(&full, &delta, [
        { ?work @ literature::author: &shakespeare }
    ])
) {
    // process only newly introduced matches
}

Reach for this macro when:

• you already have full and delta
• you want monotonic incremental processing
• pattern! would re-emit old matches every time

See Incremental Queries for the full workflow.

Combine constraints with and! and or!

Use and! when every clause must hold:

find!(
    (friend: Inline<_>),
    and!(
        pattern!(&kb, [{ ?person @ social::name: "Alice" }]),
        pattern!(&kb, [{ ?person @ social::friend: ?friend }])
    )
)

Use or! when any branch may hold:

find!(
    (alias: Inline<_>),
    or!(
        pattern!(&kb, [{ ?person @ social::nickname: ?alias }]),
        pattern!(&kb, [{ ?person @ social::name: ?alias }])
    )
)

or! branches must mention the same variable set.

Introduce helpers with temp!

Use temp! when you need a fresh variable only inside a sub-expression.

find!(
    (person: Inline<_>),
    temp!((friend), and!(
        pattern!(&kb, [{ ?person @ social::friend: ?friend }]),
        pattern!(&kb, [{ ?friend @ social::name: "Bob" }])
    ))
)

This is useful when the helper participates in joins but should not be projected. When the helper lives entirely within a single pattern!, prefer a _?var placeholder: it enforces equality across its occurrences inside that pattern without being projected.

Which macro should I use?

If you are:

• defining encodings: use attributes!
• building facts for one entity: use entity!
• matching trible structure: use pattern!
• matching only newly added results: use pattern_changes!
• asking for rows back: use find!
• asking for a boolean: use exists!
• requiring all clauses: use and!
• allowing alternatives: use or!
• introducing a fresh helper variable: use temp! (or _?var inside a single pattern!)

From here, the best next stops are:

• Query Language for the execution model
• Patterns & Recipes for modeling patterns
• Incremental Queries for pattern_changes!

Encodings

TribleSpace stores data in strongly typed values and blobs. An encoding describes the language‑agnostic byte layout for these types: [Inline]s always occupy exactly 32 bytes while [Blob]s may be any length. Encodings translate those raw bytes to concrete application types and decouple persisted data from a particular implementation. This separation lets you refactor to new libraries or frameworks without rewriting what's already stored or coordinating live migrations. The crate ships with a collection of ready‑made encodings located in triblespace::core::inline::encodings and triblespace::core::blob::encodings.

When data crosses the FFI boundary or is consumed by a different language, the encoding is the contract both sides agree on. Consumers only need to understand the byte layout and identifier to read the data—they never have to link against your Rust types. Likewise, the Rust side can evolve its internal representations—add helper methods, change struct layouts, or introduce new types—without invalidating existing datasets.

Why 32 bytes?

Storing arbitrary Rust types requires a portable representation. Instead of human‑readable identifiers like RDF's URIs, Tribles uses a fixed 32‑byte array for all values. This size provides enough entropy to embed intrinsic identifiers—typically cryptographic hashes—when a value references data stored elsewhere in a blob. Keeping the width constant avoids platform‑specific encoding concerns and makes it easy to reason about memory usage.

Conversion traits

Conversion goes through the Encodes<Source> trait, which lives on the encoding (the encoding is the impl target; the source is the trait parameter). This is the same direction as std's From<T> — and for the same reason: it trivially satisfies Rust's orphan rule, so you can write impl Encodes<SomeForeignType> for MyLocalEncoding without any "trait position 0" gymnastics.

The ergonomic source-side methods .to_inline() / .to_blob() / .into_encoded() are auto-derived blanket implementations — users never implement them directly, the same way you never implement Into<T> in Rust:

User implements:     Auto-derived via blanket:
  Encodes<T> for S     IntoEncoded<S> for T  (+ IntoInline / IntoBlob aliases)

For fallible conversions where the error type is part of the contract (parsing a hex string into a hash, validating a timestamp range, rejecting reserved bits), use TryToInline / TryFromInline — kept as separate traits because the error type is per‑source.

#![allow(unused)]
fn main() {
use triblespace::core::inline::encodings::iu256::U256BE;
use triblespace::core::inline::{Inline, IntoInline, TryFromInline, TryToInline};

struct Percent(u8);

impl TryToInline<U256BE> for Percent {
    type Error = &'static str;

    fn try_to_inline(self) -> Result<Inline<U256BE>, Self::Error> {
        if self.0 > 100 {
            Err("a percentage is at most 100")
        } else {
            Ok(u64::from(self.0).to_inline())
        }
    }
}

impl TryFromInline<'_, U256BE> for Percent {
    type Error = &'static str;

    fn try_from_inline(value: &Inline<U256BE>) -> Result<Self, Self::Error> {
        let n = <u64 as TryFromInline<U256BE>>::try_from_inline(value)
            .map_err(|_| "not a percentage")?;
        u8::try_from(n)
            .ok()
            .filter(|n| *n <= 100)
            .map(Percent)
            .ok_or("not a percentage")
    }
}
}

Encoding identifiers

Every encoding declares a unique 128‑bit identifier, accessible via the MetaDescribe::id method (for example, U256BE::id()). Persisting these IDs keeps serialized data self describing so other tooling can make sense of the payload without linking against your Rust types. Dynamic language bindings (like the Python crate) inspect the stored encoding identifier to choose the correct decoder, while internal metadata stored inside Trible Space can use the same IDs to describe which encoding governs a value, blob, or hash protocol.

Identifiers also make it possible to derive deterministic attribute IDs when you ingest external formats. Wrap the source field name in an entity-core fragment — Attribute::<S>::from(entity!{ metadata::name: <name handle>, metadata::value_encoding: <S as MetaDescribe>::id() }) — to combine the encoding ID with the source field name and produce a stable attribute so re-importing the same data always targets the same column. The attributes! macro offers three identity origins. Omitting the literal derives identity from (name, encoding), which is useful for quick experiments or source-shaped internal attributes. "HEX_ANCHOR" as name: Encoding derives identity from (anchor, encoding), which is the preferred form for attributes shared across binaries or languages: the Rust name can change freely, while a type change truthfully creates a different column. The exceptional "HEX_ID" unsafe as name: Encoding form uses the literal bytes verbatim. It is for preserving an already-published identity and carries the unchecked obligation that the encoding still agrees with all rows under that id.

Built‑in inline encodings

The crate provides the following inline encodings out of the box:

• GenId – an abstract 128 bit identifier.
• ShortString – a UTF-8 string up to 32 bytes. Deprecated: declare text attributes as Handle<UTF8String>; the type remains only so old rows stay decodable while downstream crates migrate.
• U256BE / U256LE – 256-bit unsigned integers.
• I256BE / I256LE – 256-bit signed integers.
• R256BE / R256LE – 256-bit rational numbers.
• ROrd256 – exact rationals whose bytes sort in numeric order.
• F64 – IEEE-754 double-precision floating point number (little-endian).
• F256BE / F256LE – 256-bit floating point numbers.
• Currency<C> – an exact monetary amount in the currency C, stored as an exact rational (see Monetary amounts).
• Hash and Handle – cryptographic digests and blob handles (see hash.rs).
• ED25519RComponent, ED25519SComponent and ED25519PublicKey – signature fields and keys.
• NsTAIInterval to encode time intervals.
• Boolean – all-zero for false, all-0xFF for true.
• LineLocation – a (start_line, start_col, end_line, end_col) span encoded as four big-endian u64 values.
• RangeU128 – a half-open (start, end) range of two big-endian u128 values.
• RangeInclusiveU128 – an inclusive (start, end) range of two big-endian u128 values.
• SocketAddress – an IP address and port: the IPv6 address (IPv4 in its IPv4-mapped form), then the big-endian port and IPv6 scope id, then ten zero bytes. It records where a sync daemon listens.
• UnknownInline as a fallback when no specific encoding is known.

#![allow(unused)]
fn main() {
use triblespace::prelude::*;
use triblespace::core::metadata::MetaDescribe;
use triblespace::core::inline::encodings::iu256::U256BE;
use triblespace::core::inline::{IntoInline, InlineEncoding};

let v: Inline<U256BE> = 42u64.to_inline();
let raw_bytes = v.raw; // Persist alongside the encoding's metadata id.
let encoding_id = U256BE::id(); // derived via describe(&mut scratch).root()
}

Built‑in blob encodings

The crate also ships with these blob encodings:

• UTF8String for arbitrarily long UTF‑8 strings.
• RawBytes for opaque file-backed byte payloads.
• EntityIdSetBlob for a grow-only set of opaque entity IDs: strictly sorted, unique, nonnil 16-byte IDs with no header or padding; empty is zero bytes. EntityIdSet retains typed views of a cover's members and performs membership and ordered set iteration without serializing a temporary union.
• SimpleArchive which stores a raw sequence of tribles.
• SuccinctArchiveBlob which stores the portable deterministic Ring/wavelet data and EOF metadata for a SuccinctArchive set. It has no native query indexes. SuccinctArchiveBlob::build_from_simple_archive derives this raw member without constructing a query runtime, and SuccinctArchiveBlob::merge computes its canonical set union.
• Rank9AcceleratedSuccinctArchiveBlob, an ABI-qualified Merkle root encoding for query-ready SuccinctArchive members. Its first 32 bytes name the exact portable SuccinctArchiveBlob child; the remaining bytes carry canonical native Rank9/select data for that child. Separate encoding and mapping ids pin pointer width, byte order, the accelerated-root format, and the builder epoch. A change that can alter canonical bytes requires a newly minted id. This is a complete source-bound accelerated encoding, not a sidecar. Member validation follows the embedded handle and requires the raw child to be resident. A cover-aware query view then loads that child through its store snapshot and attaches the persisted sections with safe bounds and alignment checks. It does not reconstruct the raw archive or prove the Rank9 bits against it; those are explicit audit operations. A raw archive uses unaccelerated query access unless an accelerated member is requested, rather than constructing Rank9 merely because someone opened it. The raw and accelerated encodings each own a canonical union. An accelerated join publishes only the accelerated result, but requires the matching raw union to be resident because its header names that exact child. It may consume an already-resident immutable raw dependency, but downstream maintenance never creates the raw blob or its MERGE; callers maintain each mapping hop explicitly with the same foundational Support. If the dependency is absent, target maintenance retains a finer accelerated cover. Ordinary support-equivalent cover resolution may reuse any already-derived or joined accelerated cover with the same foundation. Resolution first excludes an image whose named raw child is unavailable and falls back to a complete finer cover; direct typed-view construction retains the same check defensively.
• WasmCode for WebAssembly bytecode stored as a blob.
• UnknownBlob for data of unknown type.

BlobEncoding says what bytes mean. CollectionEncoding is the stronger contract for formats used as collection members: every member is an ordinary typed Blob, and the encoding owns canonical member validation and one canonical member-join operation. When one blob hits a deterministic geometry limit, a multi-member Cover<E> retains the same logical value, so every collection remains a full join-semilattice. "Derived" describes provenance across a join-preserving mapping, not weaker algebraic capability; covers are lazy physical decompositions of the same join. Explicit validation and joining share one immutable store snapshot, so Merkle-shaped encodings can resolve children named by their members without consulting ambient mutable state. The maintained collection encodings implement the same contract directly; there is no separate public artifact or lattice wrapper to pair with them.

Attaching a maintained index is not an audit of its producer. Ordinary readers retain typed views of the persisted bytes and perform the framing, bounds and alignment checks needed for safe access. Canonical ordering checks, count reproofs and reconstruction comparisons belong to explicit audits. Nor should a reader serialize and hash a new member just to interpret a multi-member cover. Query work remains query work: for example, a portable BM25 index stores no document lengths, so scoring needs one pass over its postings to derive them. Its query() method makes that preparation explicit; attaching the index itself only retains the original bytes. No wire-format or collection identity change is needed to keep these operations separate.

#![allow(unused)]
fn main() {
use triblespace::core::metadata::MetaDescribe;
use triblespace::core::blob::encodings::utf8string::UTF8String;
use triblespace::core::blob::{Blob, BlobEncoding, IntoBlob};

let b: Blob<UTF8String> = "example".to_blob();
let encoding_id = UTF8String::id(); // derived via describe(&mut scratch).root()
}

Both value and blob encodings can emit optional discovery metadata. Calling MetaDescribe::describe returns a rooted Fragment (exporting the encoding id) whose facts tag the encoding entity with metadata::KIND_INLINE_ENCODING or metadata::KIND_BLOB_ENCODING and may attach a metadata::name and metadata::description (UTF8String handles). Persist the description blobs alongside the metadata tribles if you want the text to remain readable.

Grow-only entity-ID sets

blob::encodings::entity_id_set::encode sorts and deduplicates authored Id values into the portable EntityIdSetBlob format. join merges two sorted members into their canonical union. It checks fixed-width framing while doing new producer work, without running a second canonical audit of its inputs. validate_element is the explicit linear check for framing, nonnil IDs, strict ordering, and uniqueness.

Ordinary TryFromBlob attachment retains an AnyBytes View<[[u8; 16]]> and checks only that the length contains complete rows. TryFromCover retains one such view per resident member. No read rebuilds, hashes, or copies a physical set. contains binary-searches each member; iter merges and deduplicates sorted member streams lazily. Exact len is constant-time for one member, but overlaps require enumerating the logical union. These operations rely on the canonical ordering contract; they are not replacement audits. Typed iteration never constructs an invalid nil Rust Id from unaudited bytes.

The IDs are opaque set elements, not blob handles or edges that cause dependency acquisition. The encoding differs from a flat typed array, whose order and duplicates are meaningful, and from LatestBlob, whose join also tracks retired states.

EntityIdSetBlob also implements CollectionDerivation from SimpleArchive. Its Id argument names one GenId-valued attribute; the concrete mapping stores that argument under the existing metadata::attribute convention. The source collection handle is already part of the ordinary derived descriptor. Each well-formed, nonnil value under the selected attribute becomes a set element; the assertion's subject does not. Nil and malformed GenId values are not typed matches. Source row framing/layout is checked without re-auditing canonical ordering, and the produced set is sorted and deduplicated.

This projection commutes with source union, including facts split across members, multiple values on one assertion, and duplicate values asserted by different entities. No type/persona/timestamp join qualifies a value. For example, a zooid-scoped receipt source can keep its ordinary SimpleArchive COMMITs, with optional created_at, and derive only the event attribute's values. A source mixing several observers projects their union; observer scope belongs to the selected source collection, not an implicit filter in this mapping. The target uses ordinary collection maintenance and foundational support; no generic authored-root or COMMIT API change is involved.

The encoding ID is 0BF639287590CFC9CE0E2B83D9FBC1E3, minted with the installed trible genid on 2026-09-15. Its bytes and identity are unchanged by adding the projection. The mapping algorithm ID is 7E4257A14880E8B4855B6282B337B4B5, minted with installed trible genid on the same date. There is no new pile record kind or configuration attribute.

Choosing the right encoding

When defining an attribute, the encoding determines how the 32-byte value slot is interpreted. Use this decision tree to pick the right one:

What are you storing?
│
├─ A reference to another entity?
│  └─ GenId
│
├─ A tag, category, or enum-like classifier?
│  └─ metadata::tag (GenId) — tags are entities with their own ID.
│     Use metadata::name to give them a human-readable label.
│     ⚠ Do NOT define a separate text tag attribute —
│     use the canonical metadata::tag and mint tag IDs.
│
├─ Text of any length (a label, a name, prose)?
│  └─ Handle<UTF8String>  (blob; the handle is its content
│     address, so equal text is an equal value and pattern!
│     can match it against a plain string)
│
├─ Money?
│  └─ Currency<C> — an exact rational, one encoding per
│     currency, in ROrd256's order-preserving layout.
│     No decimal scale anywhere: 1.50 is 3/2, so there is
│     no constant to pick now and regret later, and a VAT
│     rate applies without rounding. Bytes sort numerically,
│     so index ranges work. See "Monetary amounts".
│     ⚠ Never F64/F256: binary floats cannot represent 0.1.
│     ⚠ Not a bare rational: the currency belongs in the
│     encoding, so EUR and USD cannot meet in a query.
│
├─ A number?
│  ├─ Integer
│  │  ├─ Fits in 64 bits? → U256BE (zero-extended) or custom u64 encoding
│  │  └─ Needs full 256 bits? → U256BE / I256BE
│  ├─ Floating point
│  │  ├─ Standard double? → F64
│  │  └─ Extended precision? → F256BE
│  └─ Rational? → R256
│     ⚠ Canonical (reduced) but NOT numerically ordered:
│     comparison hits the numerator first, so 1/1 sorts
│     before 2/3. Use a fixed-scale integer if the index
│     has to answer "greater than x" — or ROrd256 if you
│     need exact division AND numeric range queries.
│
├─ A timestamp or time range?
│  └─ NsTAIInterval
│
├─ A cryptographic value?
│  ├─ Content hash? → Hash<Blake3>
│  ├─ Reference to a blob? → Handle<BlobEncoding>
│  └─ Signature? → ED25519RComponent / ED25519SComponent / ED25519PublicKey
│
├─ A file or binary payload?
│  └─ Handle<RawBytes>  (blob)
│
├─ A large structured dataset?
│  └─ Handle<SimpleArchive>  (blob, stores a TribleSet)
│
└─ Something else?
   ├─ Fits in 32 bytes? → define a custom InlineEncoding
   └─ Larger? → define a custom BlobEncoding + use Handle

Rules of thumb:

• If two values should be joinable (appear in the same query variable), they must share an encoding. Choose the most specific encoding that covers both uses.
• Store text as Handle<UTF8String>. A query can constrain it with a plain string, because the handle is the text's content address; reading the text back is a blob lookup.
• Use GenId for relationships between entities. Never store entity references as strings.
• When in doubt between an inline encoding and a blob, ask: "will I ever want to query or join on this directly?" If yes, it should be inline. If it's opaque content you just retrieve, use a blob handle.

Exact rationals: R256 vs ROrd256

Indexes compare the 32 stored bytes, so an index range is only a value range when the encoding's byte order matches its numeric order. R256 does not have that property. It stores the numerator in the first 16 bytes and the denominator in the last 16, so bytewise comparison reads the numerator first and 1/1 sorts below 2/3 even though 1 > 2/3. Its big-endian variant gives a stable portable layout, not a numerically meaningful one.

ROrd256 is the sibling encoding that does sort numerically, while staying exact and canonical.

How. The Stern–Brocot tree is a binary search tree over the rationals, so its in-order traversal is numerically sorted and a root path (L/R) with a terminator sorting between L and R compares lexicographically in numeric order. A raw path is not width-bounded — 1/1000000 is 999999 left branches deep — but the continued fraction [a0; a1, a2, …] is exactly the run-length encoding of that path, so it carries the identical order in O(log min(p,q)) terms. Comparison of continued fractions alternates (a0 ascending, a1 descending, …), so each term is written in a prefix-free order-preserving code and bit-complemented at odd positions, which turns the alternation back into plain lexicographic comparison. A terminated fraction behaves like a +∞ term at the next position, so the terminator is a run of the pad value that sorts above every code — and complementing it at odd positions correctly turns that +∞ into −∞.

Canonical. The Euclidean algorithm never emits the alternative […, n-1, 1] spelling, so exactly one byte string exists per value and intrinsic ids stay stable. Bytes that claim a trailing 1 are rejected by validate.

Representable subset. Encoding costs roughly 2·log2(max(|p|, q)) bits and fails with a typed OrderedRatioError::OutOfDomain rather than rounding:

The guaranteed bound is set by long continued fractions of small-but-not-one terms; the smallest value that does not fit needs max(|p|, q) > 2^104.7. Counter-intuitively, Fibonacci ratios — the longest continued fractions — are the cheapest, because a term of 1 costs a single bit, so every Fibonacci ratio representable in i128 encodes comfortably.

Cost. Ordering is free at query time (it is memcmp on bytes the index already compares); you pay for it on write. Encoding runs a Euclidean expansion and decoding a continuant recurrence, both O(number of terms):

R256LE's encode is two to_le_bytes and nothing else, so relative to a raw two-limb store ROrd256 is two orders of magnitude slower to encode. Against R256BE — which canonicalizes with a gcd — it is under 2× for typical values, and decoding is comparable either way because R256's own canonicality check also runs a gcd. Run cargo bench -p triblespace-core --bench ordered_rational to reproduce.

Choose ROrd256 when you need exactness and numeric range queries on the same column: exact empirical rates k/n where the denominator varies, exact probabilities or thresholds, ratios that arise from division and are then filtered by magnitude. The alternative — an R256 column plus a parallel F64 sort key — needs two attributes that can drift apart, and its range answers are inexact at the boundary, because the bound itself (1/3, say) is not a float. ROrd256 makes the index answer the exact answer.

Choose R256 for everything else. It is simpler, encodes in nanoseconds, and covers the full i128 × i128 box rather than a subset of it.

Choose neither for money. Amounts have a fixed scale, are not divided at storage, and already sort numerically as scaled integers. Use Currency<C>, which is exactly that.

Monetary amounts

Money gets its own encoding family — Currency<C>, one encoding per currency — but the value it stores is not a money-specific layout at all. It is an exact rational in ROrd256's encoding.

#![allow(unused)]
fn main() {
use triblespace::core::attribute::Attribute;
use triblespace::core::id::Id;
use triblespace::core::id_hex;
use triblespace::core::inline::TryToInline;
use triblespace::core::inline::encodings::money::{Amount, Currency, Euro, UsDollar};

let price = Amount::<Euro>::from_minor(150).expect("valid scale"); // €1.50, from cents
assert_eq!(price.to_string(), "1.50 EUR");
let value = price.try_to_inline().expect("in domain");

// One anchor, one attribute name, one id per currency.
const TOTAL: Id = id_hex!("251C2B673AE7F49F7374866925D4F7D7");
let eur_total = Attribute::<Currency<Euro>>::anchored(TOTAL);
let usd_total = Attribute::<Currency<UsDollar>>::anchored(TOTAL);
assert_ne!(eur_total.id(), usd_total.id());
}

A rational, because then there is no scale to get wrong

The obvious encoding for money is a fixed-point integer: pick a scale, count units of 10⁻ˢᶜᵃˡᵉ. It works, and it hides a decision. Whatever scale you pick becomes a constant that every stored value depends on. It has to be right for every currency and every future use — sub-cent unit prices, an eighteen-decimal crypto denomination — and it cannot be revised without rewriting every amount ever written, because the same figure at a different scale is a different byte string and therefore a different intrinsic ID.

A rational has no such constant. €1.50 is 3/2. €1.505 is 301/200. A third of a euro is 1/3, which no fixed-point encoding can hold at all. The question "is 18 places enough, or 4, or 36?" simply does not arise, and there is no migration hiding behind a later answer to it.

Canonical form — the property a content-addressed store actually needs — comes along for free. Every rational is stored reduced, so one amount has exactly one byte string, and merge, equality and intrinsic IDs all agree. That is the same guarantee a fixed global scale was buying, obtained from the number itself rather than from a convention about it.

It also makes rates exact. Applying 19% VAT to €19.99 gives 37981/10000 exactly — not 3.79, not 3.80, not a float. The intermediate keeps its full value, so rounding happens once, where the invoice is produced, which is the only place it belongs.

…and it still sorts

Using ROrd256 rather than R256 is what makes the choice free. R256 stores numerator and denominator side by side, so its bytes sort by numerator first and 1/1 lands below 2/3; an index range over such a column is not a value range. ROrd256 stores the canonical continued fraction in an order-preserving code, so plain bytewise comparison — which is what the indexes do — is numeric comparison. "Invoices over €10,000" is an index range, and the answer is exact rather than float-keyed.

Worth being honest about: ordering is not why this design won. At the scale of a working accounting database a linear filter over a hundred thousand rows is milliseconds, and index acceleration only begins to matter in the millions. If R256 were the only exact rational available, exactness would still be the right trade and the scan would be fine. ROrd256 simply means there is no trade.

What it costs

Encode time. A continued-fraction expansion, not a byte copy. Measured on two-decimal amounts across a ±500,000 EUR range (release build, Apple M-series):

A 133,000-record ingest therefore spends about 64 ms encoding money — far below the cost of reading those rows out of the database. Comparison, which is what queries do, is a plain byte compare and costs nothing extra.

A bounded, data-dependent domain. Encoding costs roughly 2·log2(max(|p|, q)) bits, so every p/q with max(|p|, q) ≤ 2^104 is guaranteed to fit, and wider values are rejected with a typed OrderedRatioError::OutOfDomain, never rounded silently. Unlike a fixed-scale integer, whose limit is a fixed magnitude, this limit depends on the shape of the number — so it deserves a real check rather than a shrug.

The check that matters is on the source column, not on today's values. Revolver, the accounting database this was built for, stores every monetary field as a PostgreSQL bigint at three decimal places (verified against the schema in the dump; measured earlier: 316,375 non-zero values across 31 currency columns, none with a digit below the second decimal). The widest value such a column can hold is ±(2⁶³−1) at scale 3, which reduces to a numerator below 2⁶⁰ over a denominator dividing 100 — against a guarantee that runs to 2¹⁰⁴. That is 44 bits of margin, and it cannot be eroded by new data, only by a schema change. Decimal money is friendly to this encoding in general: a two-decimal amount is p/100 at worst, and the numerator would have to reach 10³¹ before the guarantee stopped applying.

Ingest code should still surface the error rather than unwrap it.

The currency is in the encoding, not in the value

Currency<Euro> and Currency<UsDollar> are different encodings with different IDs. Since an attribute's identity is derived from (anchor, value_encoding), one anchored attribute name yields a different attribute per currency, with nothing minted per currency.

The objection — that a value should be self-describing — does not survive contact with the data model. A trible is (entity, attribute, value); the attribute is always present when the value is. Putting the currency in the attribute is exactly as self-describing as putting it in the value, and it buys three things:

1. Currency confusion becomes structurally impossible. You cannot sum EUR and USD by accident, because they are not the same attribute and do not meet in a query — and in Rust, Amount<Euro> + Amount<UsDollar> does not compile.
2. The value stays a plain number, which is what lets it be an ROrd256 payload unchanged rather than a money-specific layout with a prefix.
3. No combinatorial minting. Adding a currency is a four-line marker type, not a new attribute ID per (field × currency).

The cost is real: a query that spans currencies has to name each one, with an or! across the per-currency attributes. Summing across currencies without a conversion rate is meaningless anyway, so making it explicit is a feature — but it is still something you have to write out.

Declaring a currency is the whole extension mechanism:

#![allow(unused)]
fn main() {
use triblespace::core::inline::encodings::money::{Currency, CurrencyUnit};

pub struct NorwegianKrone;
impl CurrencyUnit for NorwegianKrone {
    const CODE: &'static str = "NOK";
    const MINOR_UNITS: u32 = 2;
}
type Nok = Currency<NorwegianKrone>;
}

The encoding's ID is derived from CODE, so two codebases that independently declare NOK land on the same encoding ID and the same attribute IDs, and their data merges — no registry, no coordination. MINOR_UNITS is deliberately an annotation rather than part of that identity: it is a presentation convention, and a disagreement about one should leave two claims about one currency rather than fork it into two currencies whose amounts no longer meet. It is what Display pads to and never truncates to: 1.50 EUR, 5 JPY, but 1.505 EUR when the value has more digits, and 1/3 EUR when it has no finite decimal at all.

An unknown currency is a compile-time absence, not a runtime error: there is no Currency<C> for a currency nobody declared, so no attribute ID exists to write it under. Reading a pile written by someone else, the code is recoverable from the encoding entity's code fact rather than from the 32 bytes.

Ingesting a runtime currency column means dispatching on the code to pick the monomorphisation:

match currency_code {
    "EUR" => set += entity!{ &doc @ total: Amount::<Euro>::from_minor(cents)?.try_to_inline()? },
    "USD" => set += entity!{ &doc @ total_usd: Amount::<UsDollar>::from_minor(cents)?.try_to_inline()? },
    other => return Err(UnknownCurrency(other.to_owned())),
}

Other things this encoding deliberately is not

• Not a float, at any width. Binary floats cannot represent 0.1; extra width only moves the discrepancy.
• Not an interval. Money that has been through a currency conversion is genuinely bounded rather than exact and deserves [low, high]. But an ingested figure is exact, and a VAT split or an allocation is now exact too — a rational times a rate is still a rational. What is left for an interval is narrow, and it belongs in an additive sibling encoding under its own anchor.
• Not a rounding policy. A rational can carry 19/300 of a euro exactly, but the figure legally owed is the rounded one on the invoice. Which way at the half, at which scale, on the line or on the total, is a business decision a byte layout has no standing to make. The encoding's job is to keep the intermediate exact so that decision happens once, deliberately.
• No ISO-4217 table. CODE is not checked against the registry. That registry is mutable (SSP in 2011, VES in 2018, ZWG in 2024), and a table baked into the crate would make the validity of already-stored data depend on which version of the library reads it — and would make adding a currency a release of this crate rather than four lines in yours.

Defining new encodings

Custom formats implement [InlineEncoding] or [BlobEncoding]. A unique identifier serves as the encoding ID. The example below defines a little-endian u64 inline encoding and a simple blob encoding for arbitrary bytes.

pub struct U64LE;

impl MetaDescribe for U64LE {
    fn describe() -> triblespace::core::trible::Fragment {
        let id: Id = id_hex!("0A0A0A0A0A0A0A0A0A0A0A0A0A0A0A0A");
        entity! { ExclusiveId::force_ref(&id) @
            metadata::name: "u64le",
            metadata::tag:  metadata::KIND_INLINE_ENCODING,
        }
    }
}

impl InlineEncoding for U64LE {
    type ValidationError = Infallible;
    type Encoding = Self;
}

impl Encodes<u64> for U64LE {
    type Output = Inline<U64LE>;
    fn encode(source: u64) -> Inline<U64LE> {
        let mut raw = [0u8; INLINE_LEN];
        raw[..8].copy_from_slice(&source.to_le_bytes());
        Inline::new(raw)
    }
}

impl TryFromInline<'_, U64LE> for u64 {
    type Error = std::convert::Infallible;
    fn try_from_inline(v: &Inline<U64LE>) -> Result<Self, std::convert::Infallible> {
        Ok(u64::from_le_bytes(v.raw[..8].try_into().unwrap()))
    }
}

pub struct BytesBlob;

impl MetaDescribe for BytesBlob {
    fn describe() -> triblespace::core::trible::Fragment {
        let id: Id = id_hex!("B0B0B0B0B0B0B0B0B0B0B0B0B0B0B0B0");
        entity! { ExclusiveId::force_ref(&id) @
            metadata::name: "bytesblob",
            metadata::tag:  metadata::KIND_BLOB_ENCODING,
        }
    }
}

impl BlobEncoding for BytesBlob {}

impl Encodes<Bytes> for BytesBlob {
    type Output = Blob<BytesBlob>;
    fn encode(source: Bytes) -> Blob<BytesBlob> {
        Blob::new(source)
    }
}

impl TryFromBlob<BytesBlob> for Bytes {
    type Error = Infallible;
    fn try_from_blob(b: Blob<BytesBlob>) -> Result<Self, Self::Error> {
        Ok(b.bytes)
    }
}

See examples/custom_schema.rs for the full source.

Versioning and evolution

Schemas form part of your persistence contract. When evolving them consider the following guidelines:

1. Prefer additive changes. Introduce a new encoding identifier when breaking compatibility. Consumers can continue to read the legacy data while new writers use the replacement ID.
2. Annotate data with migration paths. Store both the encoding ID and a logical version number if the consumer needs to know which rules to apply. UnknownInline/UnknownBlob allow you to safely defer decoding until a newer binary is available.
3. Keep validation centralized. Place invariants in your encoding conversions so migrations cannot accidentally create invalid values.

By keeping encoding identifiers alongside stored values and blobs you can roll out new representations incrementally: ship readers that understand both IDs, update your import pipelines, and finally switch writers once everything recognizes the replacement encoding.

Inline formatters (WASM)

Binary formats are great for portability and performance, but they can be painful to inspect if you don’t know the encoding ahead of time. TribleSpace supports an optional encoding-level formatter mechanism: an inline encoding can point to a small sandboxed WebAssembly module that turns its raw 32 bytes into a human-readable string.

The formatter is stored as a blob (blobencodings::WasmCode) and referenced from the encoding identifier entity via the metadata attribute metadata::value_formatter.

The built-in runner lives behind the wasm feature flag (enabled by default in the triblespace facade crate) and uses wasmi with tight limits (fuel, memory pages, output size). Modules must not import anything and use the following minimal ABI:

• memory (linear memory)
• format(w0: i64, w1: i64, w2: i64, w3: i64) -> i64

The format arguments are the raw 32 bytes split into 4×8-byte chunks (little-endian). The return value packs the output pointer and output length:

• Success returns (output_len << 32) | output_ptr with output_ptr != 0.
• Failure returns (error_code << 32) | 0 (i.e. output_ptr == 0).

The core crate can optionally ship built-in formatters for its built-in value encodings. Enable the wasm feature to have MetaDescribe::describe (which is fallible) attach metadata::value_formatter entries for the standard encodings. This feature requires the wasm32-unknown-unknown Rust target at build time because the bundled formatters are compiled to WebAssembly via the #[value_formatter] proc macro.

Incremental Queries

The query engine normally evaluates a pattern over one complete relation. Continuous ingest pipelines often need only the solutions which involve at least one newly added fact. pattern_changes! implements that semi-naive delta evaluation without tying the query engine to any storage or history model.

Supply full facts and one delta

Maintain two sets:

• full, which contains every fact visible for the current observation; and
• changed, the subset added since the previous observation.

full must include changed before the query runs:

let mut full = initial;
let changed = entity! {
    &messiah @
    literature::title: "Dune Messiah",
    literature::author: &herbert,
}.into_facts();
full += &changed;

let new_titles: Vec<String> = find!(
    title: String,
    pattern_changes!(&full, &changed, [
        { _?author @ literature::firstname: "Frank" },
        { _?book @
            literature::author: _?author,
            literature::title: ?title
        }
    ])
)
.collect();

The complete runnable version is deliberately storage-independent:

    // `pattern_changes!` needs only the complete known set and the newly
    // observed delta. How those sets arrived is deliberately orthogonal.
    let mut initial = entity! { literature::firstname: "Frank", literature::lastname: "Herbert" };
    let herbert = initial.root().expect("intrinsic author identity");
    initial += entity! { literature::title: "Dune", literature::author: &herbert };

    // On first observation, the whole set is new.
    let mut changed = initial.into_facts();
    let mut full = changed.clone();

    // Text attributes hold the content handle of their text, so a title
    // compares with the handle of a string without reading any blob.
    let all_titles: Vec<Inline<_>> = find!(
        title: Inline<_>,
        pattern_changes!(&full, &changed, [
            { _?author @ literature::firstname: "Frank" },
            { _?book @ literature::author: _?author, literature::title: ?title }
        ])
    )
    .collect();
    assert_eq!(all_titles, vec!["Dune".to_inline()]);

    // A later collection observation contributes one monotonic delta.
    changed = entity! {
        literature::title: "Dune Messiah",
        literature::author: &herbert,
    }
    .into_facts();
    full += changed.clone();

    // Only Dune Messiah shows up — Dune was in the previous observation.
    let new_titles: Vec<Inline<_>> = find!(
        title: Inline<_>,
        pattern_changes!(&full, &changed, [
            { _?author @ literature::firstname: "Frank" },
            { _?book @ literature::author: _?author, literature::title: ?title }
        ])
    )
    .collect();
    assert_eq!(new_titles, vec!["Dune Messiah".to_inline()]);
    println!("{} new title(s)", new_titles.len());

Applications may obtain changed directly from an importer, an event batch, a collection-cover difference, or any other source. The query algorithm only needs the two fact sets.

How semi-naive evaluation works

For a pattern with several trible constraints, the macro evaluates one variant per constraint:

1. that constraint reads from changed;
2. every other constraint reads from full; and
3. the variants are combined with or!.

Every returned binding therefore has at least one witness in the delta. Work scales with the changed set and the number of trible constraints instead of requiring each constraint to scan the complete dataset.

The union constraint deduplicates proposed candidate values at each search level, so the same complete binding supported by several variants is enumerated once per invocation. The usual find! projection semantics still apply: hidden variables can create multiple rows when genuinely distinct complete bindings project to the same tuple. Collect projected results into a set when the application wants projection-level uniqueness.

Nothing is remembered between invocations. A later delta may return a tuple seen earlier when a new fact supplies another proof. Applications that require global once-only delivery retain the projected tuples they have consumed; applications interested in witness events should project the witness identity.

Use absorbed source payloads as the continuation token

A Support is a Cover of one collection's own foundations: a root's distinct committed payload handles, a derived collection's leaf images. Its PATCH-backed membership makes a natural storage-level continuation token, and Cover::additions_since verifies that the earlier token remains a subset of the later one and returns only newly observed members.

The token has to name what the queried view answers for, and a derived view answers for the source payloads that have a leaf in it, not for everything the source holds. Every writer derives the payloads it wrote, so another writer's payload is in the source before it is in the view. A token taken from the source's own support would advance past that payload and never deliver it. So the example asks each hop's freshness instead: the source payloads the first view has a leaf for (the source's support minus raw.missing_from(&source)), and only while the second view has caught up with the first (accelerated.missing_from(&raw) is empty). A view that lags holds the token back:

fn observe(
    store: &mut MemoryRepo,
    signing_key: &SigningKey,
    raw: Collection<SuccinctArchiveBlob>,
    accelerated: Collection<Rank9AcceleratedSuccinctArchiveBlob>,
    checkpoint: &mut Option<Support>,
    mut consume: impl FnMut(&str) -> Result<(), Box<dyn Error>>,
) -> Result<Vec<String>, Box<dyn Error>> {
    // Maintain both attachments -- the Succinct index, then the Rank9
    // accelerator that reads it -- and read from the snapshot that observes
    // all of that work.
    block_on(store.maintain_attached(raw, signing_key))?;
    let snapshot = block_on(store.maintain_attached(accelerated, signing_key))?;
    let attached = snapshot.attached(accelerated)?;

    // The continuation token is the set of source payloads the attached cover
    // stands for, not what the source holds: a payload without an attachment
    // yet is the cover's residual, and a token that advanced past it would
    // never deliver it.
    let current = attached.support().clone();
    let changed = match checkpoint.as_ref() {
        Some(previous) => {
            if previous == &current {
                return Ok(Vec::new());
            }
            match current.additions_since(previous) {
                Ok(additions) => {
                    // The delta is a set of source payloads: read them from
                    // the source through the same snapshot.
                    let mut changed = TribleSet::new();
                    for member in additions.members() {
                        let payload: TribleSet = snapshot.get(member)?;
                        changed.union(payload);
                    }
                    Some(changed)
                }
                Err(CoverAdvanceError::ResetRequired { .. }) => None,
                Err(error) => return Err(error.into()),
            }
        }
        None => None,
    };

    // The attached cover answers the full side: it stands for everything the
    // token names.
    let full: UnionArchive<OrderedUniverse> = attached.view()?;
    let titles = match changed {
        Some(changed) => changes(&full, &changed, &snapshot, &mut consume)?,
        None => rebuild(&full, &snapshot, &mut consume)?,
    };

    // Adopt only after the complete fold succeeds. A failed consumer retries
    // the same delta, so external effects must be transactional or idempotent
    // when exactly-once delivery matters.
    *checkpoint = Some(current);
    Ok(titles)
}

This computes set difference over immutable payload identities; it does not walk a parent chain or ask an ambient head what changed. A second signer or metadata claim over an existing payload is provenance and produces no data delta. If a previous member is absent, the helper returns CoverAdvanceError::ResetRequired: additions-only maintenance is no longer sound, so rebuild the accumulated application state from current. Advance the saved token only after the complete fallible fold succeeds, as the example does, to make a failed fold retry the same delta.

The two pattern inputs need not share a representation. The delta is a set of source payloads, so it is read from the source by handle through the same snapshot -- snapshot.get(member) for each new member, unioned -- and the maintained target's view answers the full side. Supports of two collections are never compared: a view's support is its own leaf images, and whether it has caught up with its source is missing_from. There is no way to attach a target for only the delta; the delta is read where it lives. Persisted DERIVE and MERGE equations make repeated maintenance idempotent.

Keep the previous token until the complete fallible fold succeeds. A failed consumer therefore retries the same delta, while already completed lattice work is merely rediscovered. Because both inputs come from one snapshot, payloads first observed later cannot leak into either query input merely because their blobs are resident by then.

Payload support is deliberately not an exact fact difference. A new payload may repeat a fact already present, and that new witness may legitimately make a projected result recur. Consumers requiring global once-only delivery retain their consumed result identities independently.

When an ingestion API already returns its newly produced fragment, querying that fragment may still be cheaper than maintaining a changed derived cover. The collection snapshot path is useful across process boundaries, after reopening storage, and whenever the derived representation is the normal query substrate.

The incremental_collection_queries benchmark measures this complete maintenance loop against a full re-query over the same evolving source data:

cargo bench --bench incremental_collection_queries -- \
  --commits 64 --books-per-commit 256 --warmup 1 --iters 4

Each observation includes functional Succinct snapshot advancement and maintaining the application's result set. Publication, cover discovery, and fixture construction remain outside the timer. The benchmark checks raw row counts, projection-level set equality, consumer checkpoints, and exact source covers at every commit; geometric checkpoints affect reporting only, not observation. Its fixture projects every query variable and gives every book a unique binding, so those equality checks do not imply projected-set semantics for arbitrary queries. The between-observation checks deliberately favor strong invariants over pristine cache state; read the output as warm-trace maintenance latency rather than isolated cold-call latency.

Monotonicity and CALM

Removed results are not tracked. Facts and collection commits are monotone: new input can add witnesses but does not invalidate a previous conclusion. This is the CALM principle in executable form—monotone results can be distributed without consensus.

When a domain needs versions or supersession, represent those relationships as facts and query the explicit DAG. Do not infer a winner from insertion order.

Trade-offs

• The caller supplies the changed set; the query engine keeps no hidden cursor.
• Each trible constraint adds one query variant, so selective constraints keep delta evaluation efficient.
• A changed set which grows without bound loses its advantage; advance the continuation after successful consumption.
• A result may recur in a later invocation when the later delta provides a new witness.

Regular Path Indexes

Regular path queries ask whether two graph terms are connected by a path whose edge labels are accepted by a finite automaton. TribleSpace keeps that recursive work outside the core query solver: the standalone triblespace-paths crate materializes the accepted endpoint relation, then exposes it as an ordinary two-variable Constraint.

This separation keeps query-time constraints stateless. The expensive fixpoint is built once for one exact collection cover; find!, and!, constants, and normal dynamic variable ordering then treat the result like any other relation.

Add the companion crate alongside the facade crate:

[dependencies]
triblespace = "0.47"
triblespace-paths = "0.47"

Describe the path, then materialize it

Most callers describe a regular path with PathExpr. Each leaf is a graph property Step; the expression builders add concatenation, alternatives, and repetition. compile freezes that description into the fixed, epsilon-free automaton consumed by PathIndex and embedded in the descriptor of the corresponding PathSummaryBlob collection.

For example, friend+ means one or more forward friend edges:

use triblespace::prelude::*;
use triblespace::prelude::inlineencodings::GenId;
use triblespace_paths::{PathExpr, PathIndex, Step};

mod social {
    use triblespace::prelude::*;
    use triblespace::prelude::inlineencodings::{GenId, Handle};

    attributes! {
        "A19EC1D9DD534BA9896223A457A6B9C9" as pub name: Handle<blobencodings::UTF8String>;
        "C21DE0AA5BA3446AB886C9640BA60244" as pub friend: GenId;
    }
}

let friend = social::friend.id().into();
let friend_automaton = PathExpr::from(Step::Forward(friend)).plus().compile();

let alice = fucid();
let bob = fucid();
let carol = fucid();
let mut graph = TribleSet::new();
graph += entity! { &alice @ social::friend: &bob };
graph += entity! { &bob @ social::friend: &carol, social::name: "Bob" };
graph += entity! { &carol @ social::name: "Carol" };
let paths = PathIndex::from_tribles(friend_automaton.clone(), graph.iter())?;

Every trible is viewed as a directed graph edge from its entity to its inline value, labeled by its attribute id. Values do not have to encode entity ids, although a path endpoint must use a compatible inline encoding when it shares a query variable with another constraint.

The expression operations are regular-language operations:

• a.then(b) matches a followed by b;
• a.or(b) matches either expression;
• star, plus, and optional mean zero-or-more, one-or-more, and zero-or-one repetitions; and
• inverse reverses the complete path. It flips each atomic step, reverses sequence order, distributes over alternatives, and preserves repetition.

Step::Forward and Step::Reverse match one exact attribute in either graph direction. ForwardExcept and ReverseExcept match every attribute except a provided list; an empty exclusion list is a wildcard, available as Step::forward_any() or Step::reverse_any(). Inverting an exclusion or wildcard changes only its direction.

Canonical expressions and compilation

PathExpr canonicalizes structure as it is assembled. Nested sequences are flattened while retaining their order. Nested alternatives are flattened, sorted by a stable explicit order, and deduplicated. Exclusion lists are also sorted and deduplicated. Thus independently assembled expressions that differ only by alternative ordering, duplicate alternatives, sequence association, or exclusion ordering compile to the same canonical automaton and fingerprint.

This is structural canonicalization, not regular-language minimization. Distributively equivalent expressions, or identities such as a nested star, may still compile to different language-equivalent automata. Do not use automaton equality to decide arbitrary regular-language equivalence.

Compilation uses the Glushkov position construction. State zero is the sole initial state, and each atomic Step occurrence contributes one additional state. First-position and follow-position relations become transitions; nullable expressions make state zero accepting. The result is a fixed NFA with no epsilon transitions and no determinization pass. Repetition changes the finite follow relation rather than unrolling an unbounded machine.

The current high-level surface is the Rust builder API, not a string parser or path! macro. Every PathExpr contains at least one atomic step, so a pure epsilon language or the empty language must be expressed with a manual automaton. State ids are u32; compile panics if the expression contains u32::MAX atomic occurrences, because no valid Automaton can represent it.

Manual automata are the low-level escape hatch

Construct Automaton directly when importing another compiler's output or when a deliberately shared state topology matters, or when the language has no atomic step. The friend+ expression above is equivalent to this explicit two-state NFA:

use triblespace_paths::{Automaton, Step, Transition};

let friend_automaton = Automaton::new(
    2,
    [0],
    [1],
    [
        Transition::new(0, 1, Step::Forward(friend)),
        Transition::new(1, 1, Step::Forward(friend)),
    ],
)?;

Automaton::new validates state numbers and canonicalizes duplicate or out-of-order transitions and exclusion lists. Its input must already be epsilon-free. Represent nullability by making an initial state accepting; for example, one initial-and-accepting state with no transitions accepts only the empty path.

Join paths with ordinary constraints

PathIndex::constraint(start, end) creates the two-column relation. Either term may be a query variable or an inline constant, and using the same variable twice asks for the accepted diagonal. The relation composes directly with pattern! and every other constraint:

let alice_value: Inline<GenId> = (&alice).to_inline();

let reachable_people: Vec<(Id, String)> = find!(
    (person: Id, name: String),
    and!(
        paths.constraint(alice_value, person),
        pattern!(&graph, [{ ?person @ social::name: ?name }]),
    )
)
.collect();

The index also has direct read methods when no join is needed: contains, reachable_from, reaching, accepted_pairs, starts, ends, and diagonal. All endpoint fibers are sorted and duplicate-free. The path relation therefore contains one pair per accepted (start, end), not one row per distinct route between them; ordinary query joins can still introduce bag multiplicity through their other witnesses.

Persist through the native collection algebra

A durable path index is identified by the exact source descriptor, one automaton, and the independent READ/WRITE policies of its derived descriptor:

use std::sync::Arc;
use triblespace::core::collection::{
    AdmissionPolicy, CollectionPolicy, CollectionSnapshotExt, CollectionStoreExt,
};
use triblespace_paths::{PathIndex, PathSummaryBlob};

let source_policy = CollectionPolicy::new(
    AdmissionPolicy::direct(source_reader),
    AdmissionPolicy::direct(source_writer),
);
let source = store.collection("social", source_policy)?;
// Attached to the root: no policy of its own, built by the store's host.
let paths = store.attach::<PathSummaryBlob>(source, friend_automaton)?;

let after = store.maintain_attached(paths, &host).await?;
let observed = after.attached(paths)?;
let support = observed.support();
let index: Arc<PathIndex> = observed.view()?;

The foundational support replaces ambient heads, commit-chain traversal, manifests, registered hooks, and range planning.

The attached read selects a Cover<PathSummaryBlob> of attachments whose support is a Cover<SimpleArchive> of the root's foundations. Any mixture of attachments at merged and unmerged nodes with that support answers the same paths, because summaries are joined before closure, so route choice is not encoded as a cover mode.

The opaque cover is the value boundary. It must name the canonical policy-bearing SimpleArchive source descriptor, and every member names one exact source payload identity. Source bytes are needed only when missing target work must actually be computed.

Distinct signed claims that name identical data collapse to one cover member and one unit of derivation work. Their authorship, signatures, and metadata are queryable, possibly absent provenance through cover.commits(&snapshot) and are intentionally unnecessary for replay or path semantics.

snapshot.collection(paths) never writes or executes collection algebra. It follows existing source MERGE, path-summary MERGE, and source-to-target DERIVE equations and returns the maximal complete resident target cover; support() on that observation is exactly the foundational support it represents. After maintain, that is what the source's frontier stands on. There is no form which takes a requested support: a support is read back from a target, never asked of one.

view::<PathSummaryView>() attaches the selected summaries: it keeps them as stored, checks that each names the descriptor's automaton by the handle in its header, and decodes that automaton once from its blob. Nothing is re-encoded or hashed to believe a stored equation. Closing the relation is an explicit step, PathSummaryView::prepare, which decodes the members structurally, merges the decoded summaries once and closes the union into the endpoint relation; view::<Arc<PathIndex>>() is that attach followed by prepare. For the empty cover, the descriptor names the same blob so the canonical bottom remains available. The mapping descriptor carries the automaton facts for source-to-target derivation and retains its blob in the descriptor closure; interpretation otherwise uses the immutable content-addressed representation dependency rather than mutable state retained in a lifecycle facade. The canonical audit of a member (PathSummaryBlob::audit, PathAutomatonBlob::audit) is explicit: validate_member and the producer's join_members run it, a warm read does not.

CollectionStoreExt::ensure asynchronously acquires exact missing dependencies and publishes missing DERIVE work only; maintain additionally performs deterministic size-tiered target MERGE work. Both return a fresh store snapshot rather than pretending that mutation itself selected one final physical cover. Every successful artifact is persisted before its producer-signed equation, no implicit durability flush or durable WANT is performed, and an unchanged warm call executes no maps or joins.

The explicit writer key must satisfy the derived descriptor's WRITE policy to publish missing derivations. Reusing an admitted result needs no new grant; optional compaction may leave the existing finer cover in place.

An empty cover returns the automaton-indexed bottom relation locally and appends nothing.

What a persisted summary means

The automaton fingerprint participates in the target collection descriptor. Two automata over the same source scope therefore inhabit different collections.

Each PathSummaryBlob contains only:

• the sorted endpoint domain required by the fixed automaton; and
• the sorted direct arcs of the graph × automaton product.

These summaries are sparse constructional data, not independently closed path relations. Their join is associative, commutative, and idempotent. Closure runs only after the selected summaries have been joined. This order is essential: a path may take its first edge from source fragment A, its next edge from fragment B, and later re-enter A. Unioning closures built independently per fragment would miss such paths. Merge order remains irrelevant because closure is derived only after the canonical semilattice join.

The low-level path_summary_union module exposes the concrete law directly. store.attach::<PathSummaryBlob>(source, automaton) identifies one attached collection by the handle of the root it summarises, the PathSummaryBlob representation and the canonical automaton fingerprint; it carries no policy. derive_element lowers one canonical SimpleArchive into direct product arcs, and the encoding's join unions two summaries:

paths(∅) = ⊥
paths(a ∪ b) = paths(a) ⊔ paths(b)

The same derivation is available from the command line without writing Rust. trible pile collection attach <pile> <source> path --expr '<expression>' registers the path summary collection; the expression names attributes by their 32-hex-digit ids, juxtaposition is sequence, | alternation, *, + and ? the usual repetitions, ^ before an atom or group follows those edges in reverse, and parentheses group:

--expr '2B3A… (^4C5D… | 6E7F…)+ 8091…?'

trible pile collection maintain then folds the summary like any other derived collection; the automaton travels in the descriptor, so a later maintainer needs nothing but the pile.

The same grammar is a macro in Rust, with attributes named by their paths instead of their ids (and { expression } for one computed elsewhere):

use triblespace::macros::path_expr;

let friends = path_expr!(social::friend (social::friend | ^social::friend)* social::name?);
let index = PathIndex::from_edges(friends.compile(), edges)?;

Both front ends lex into triblespace_paths::syntax::Token, parse with one syntax::parse, and lower the same syntax::Ast, so an expression means the same thing on the command line and in code.

The bottom is an explicit 48-byte summary: automaton-blob handle, state count, and zero vertex and arc counts. Derivation is therefore total even for an empty source or a non-nullable source with no matching labels.

Nullable paths and the vertex universe

A nullable expression uses an accepting initial state. Its zero-hop answers are identity pairs (v, v) for the summary's complete vertex universe. The universe includes both endpoints of every supplied trible, even when its attribute matches no transition.

Non-nullable summaries omit unmatched endpoints. Nullable summaries retain them as the identity universe, but closure runs only over endpoints incident to matching product arcs before mapping the relation back and adding the diagonal. Unrelated attributes therefore do not widen the quadratic closure workspace.

An entirely empty source has no graph terms and therefore no identity pairs. Its native collection derivation is nevertheless the explicit 48-byte bottom, so “covered and empty” remains distinct from “not materialized.”

Cost model: sparse input, potentially dense answer

Summaries retain an endpoint domain and direct product arcs, so they can remain close to sparse input and merge cheaply. Attachment is a different operation: it materializes the complete accepted endpoint relation as CSR plus reverse and domain views. Some regular paths accept every pair of vertices, making that relation Θ(|V|²). No exact materialized representation can avoid paying for that output, and the closure construction also uses bitset scratch space.

The current canonical blob stores product endpoints as full-domain u32 ordinals. Persisted nullable summaries consequently require |universe| × |automaton states| <= u32::MAX, even though attachment closes only the smaller matched support. Crossing that format ceiling is an explicit error rather than ordinal truncation.

Use a native path collection when a stable automaton and repeated queries amortize materialization. For a one-off traversal over a large sparse graph, an application-side search may use less memory; for a fixed small hop count, explicit joined pattern! clauses remain simpler. The path index is a deliberate materialized-view trade, not a hidden lazy traversal.

Query Engine

Queries describe the patterns you want to retrieve. There is no query planner and no compiled plan. Every operator and every data source implements the same Constraint protocol, and the engine consults those constraints while it searches, so binding order is chosen from the values already found instead of being fixed before evaluation.

This chapter describes that protocol, the search that drives it, the result semantics that fall out of both, and — just as importantly — the things the engine deliberately refuses to do. The refusals are load-bearing: each one is what buys a property the engine does provide.

The constraint protocol

A constraint restricts the values that query variables may take. It is not a node in a plan; it is a participant the engine interrogates. The whole interface is six methods, and the engine calls them in a fixed rhythm:

estimate returns None for a variable the constraint does not touch, which is how "irrelevant" is distinguished from "unknown cost". An estimate is a cost quote and nothing else: it steers variable ordering, never correctness. A constraint that lies about its cardinality makes the search slower, not wrong. Whether the answer is Some or None must depend only on which variables are bound, never on their values — a batch shares one bound set, and composites read relevance off the batch.

Only four of the six need an implementation: satisfied defaults to true and influence defaults to "every variable I touch except this one". A new data source therefore joins the engine with variables, estimate, propose, and confirm — iteration and point queries, nothing more. That minimum is what keeps hash maps, PATCHes, succinct archives and device-resident structures all admissible; a seek or leapfrog requirement would disqualify half of them.

Composition is by two constraints rather than by an algebra: IntersectionConstraint (built by and!) and UnionConstraint (built by or!). Both are ordinary constraints implementing the same six methods, so a TribleSet pattern, a HashSet membership test, and an application predicate mix in one query without any of them knowing about the others.

Propose and confirm take a batch

propose and confirm do not take one binding. They take a Frontier: the whole collection of parent bindings sitting at one point of the search. A single binding is a frontier of one, which is exactly the older single-binding protocol.

The reason is measured. With a width-1 frontier only the root level ever proposes widely; every deeper level asks a source for the candidates of one parent, and over real data (a region-size census on dblp) that is a median of 1–7 candidates at every scale, p95 around 200. Any batched tier — a GPU dispatch that pays off at 16 384 candidates, a SIMD probe — therefore engaged once at the root and never again. Batching the parents is what makes a level's region large at every depth.

A frontier is cheap because bindings are indexes, not values (see below): it is an index matrix over the shared level buffers plus an exact row selection. Consecutive selections and their slices are implicit; a genuine permutation borrows a dense list of row ordinals, so restricting a frontier never copies an index row and pays four bytes per row only when the order must be explicit. Correspondingly, the ProposalBuffer is segmented: a proposer calls open(row) before appending a row's candidates, and every entry carries that parent tag. Candidates exposes the tags, so one region can span a whole batch and a confirmer can still tell whose each candidate is.

Sources split cleanly along that seam. A verdict that does not depend on the parent binding — set membership, a byte range, a constant — ignores the tags and filters the whole region in one pass. One that does walks the region with Candidates::for_each_parent, which yields maximal runs of equal tag, so a per-binding setup is paid once per run instead of once per candidate.

Nothing about the batch changes what the query means. Worst-case optimality is untouched — expanding n prefixes together is the same total work as expanding them one at a time, and the AGM bound is a statement about output size, not traversal order.

Batching has two different memory terms. The frontier matrices cost O(width × variables × depth): depth-first's O(depth) frontier traded for a wide one. A proposal region can dominate that term. A source that proposes fanout candidates for each parent materialises Θ(width × fanout) entries before the level is consumed. Each entry currently carries a 32-byte value, a four-byte parent tag, and one packed liveness bit — nominally 36.125 bytes, excluding Vec capacity slack. Thus a width of 16,384 and a fanout of 10,000 is nominally about 5.92 GB (5.51 GiB) at one level, even though no frontier ever contains more than 16,384 rows.

FrontierStats::peak_region reports that proposal high-water mark in entries. Choose a lower width with Query::with_frontier_width for queries that join against high-fanout sources; this bounds the number of parents expanded together, but it cannot bound one parent's own fanout.

Statelessness is the load-bearing choice

Every method receives the current Binding — or a batch of them — as a parameter. A constraint keeps no cursor, no half-finished enumeration, no record of where the search has been.

That single decision pays for most of the engine's structure:

• Backtracking is free. The engine unsets a variable in the binding and moves on. There is nothing to notify, unwind, or roll back, because no constraint holds state that could be stale.
• The constraint tree can be cloned. Parallel execution splits by cloning the engine's state; the constraint tree is shared behind an Arc. If constraints held live iterators, a split would have to duplicate or hand off those iterators, and a borrowed enumeration would tie the constraint's lifetime to the engine's — the self-referential trap that a stateful protocol cannot avoid.
• A level is proposed once. Because no source is ever asked to resume, a level's buffer is written exactly once and never appended to while its variable is bound. That makes "a bound variable's buffer is stable for the lifetime of its binding" an unconditional invariant, which is what lets a binding be an index into that buffer instead of a copy of the value.

Depth-first search with dynamic variable ordering

The ordering is the engine's core performance idea, not an implementation detail.

Query::new asks every variable for an estimate against the empty binding. Then the search repeats one step, over a whole frontier at a time:

1. For each row of the frontier, pick its most specific unbound variable from that row's own estimates.
2. Partition the frontier by that choice and take the next group.
3. Ask the constraint tree to propose candidates for the group's variable, over all of the group's rows in one call. An intersection internally lets each row's tightest child propose and runs the remaining children as confirmers over that output, so the buffer the engine sees has already survived every clause.
4. Turn the next chunk of surviving candidates into the child frontier and descend. Each child row inherits its parent's estimates and refreshes exactly the ones the new binding could have changed — the influence set of the variable just bound.
5. When a level runs out of live candidates, retire it and continue with the next group, then with the next chunk one level up.

The width is a ceiling, and the frontier ramps from one row

DEFAULT_FRONTIER_WIDTH is how wide a chunk may get, not how wide the first one is. A level's first chunk is INITIAL_FRONTIER_WIDTH = 1 binding. Later chunks multiply by FRONTIER_RAMP_BASE = 8 until they reach the ceiling. If the final remainder would be smaller than the chunk before it, the engine merges that tail early; the caller's ceiling remains a hard bound.

That is what keeps time-to-first-result honest. A caller who stops after one row — exists!, .next() — must not pay to build a 16 384-wide frontier it will never look at, and with a first chunk of one it does exactly the work the single-binding engine did. Measured on a first-row-only join, a flat full-width engine is 8.8x slower than the pre-batching engine; one narrow chunk closes the entire gap.

The base is the latency/throughput trade, not an incidental tuning detail. A base-2 ramp was measured and rejected: its last chunk is only about half a drain, so it cut a fixture's useful frontier from 2048 rows to 512 and raised expansions from 3 to 74. The failure was the base, not geometric growth. At base 8, the last term is asymptotically seven eighths of the drain and a 16 384 ceiling takes six rungs rather than doubling's fifteen. Across the 100-query registry on three backings, base 8 retained 99.61% of aggregate widest-frontier width and 93.30% of GPU-routed candidate work (against 97.46% for the flat schedule), at 44.7% more expansions. It therefore keeps almost all useful batch width while avoiding the flat schedule's enormous overshoot for callers that want only a handful of rows. The first row is protected exactly; larger short demands pay at most the next base-8 rung.

A 1:1 descent copies nothing

Step 4 normally builds the child frontier by copying each drawn candidate's parent row and filling in the newly bound slot. When the draw is 1:1 — one surviving child per parent row, in order, covering the whole frontier, with nothing left pending — that copy is pure waste: no row was gained, lost or reordered, so the child block is the parent block with one more slot written, and the child's estimate rows are bit-identical to the parent's.

The two standing invariants are what make it sound. Confirmers may only kill candidates, never revive them, so a surviving row keeps its identity. Buffers are write-once, so the newly bound variable's slot in every row was previously unwritten and filling it destroys nothing. And because such a draw leaves the level spent, the parent frontier is never asked for anything again, so its matrices are handed down rather than shared — which is what lets a whole chain of 1:1 descents run without a single matrix copy instead of only the first one.

Ownership needs no separate flag. The matrices already sit behind Arc so a rayon split copies refcounts, and Arc::get_mut therefore succeeds exactly when no split or steal holds the other half; when it says no, the copying path runs. FrontierStats counts both paths.

The fast path is gated so that it costs nothing when it cannot fire. Recognising a 1:1 draw means deferring the child rows until the draw's shape is known, and that deferral is a second pass — measured at +10% and +20% on two fixtures when charged to every descent. So the engine asks first, from what it already knows: a level holding proposed candidates for rows parents can only yield one child per parent if proposed == rows. Every fan-out level fails that O(1) test and runs the fused single-pass build exactly as before.

This matters most for the shape batching can never help: a chain with fan-out one at every level has no sibling parents to widen the frontier with, so it can only ever be charged for the machinery. Removing the per-level row copy is what brings it back to the single-binding engine's cost.

A row is never moved onto a variable it did not choose, however tempting that is for batch size. propose owns candidate support and first-seen order, and the protocol supplies no cross-variable support-equivalence law, so an estimate-compatible variable is not an interchangeable action. All the leeway lives in the bucketing described next — which is exactly why agreement, and so an unsplit batch, is the common case. FrontierStats counts expansions, rows and groups, so fragmentation is observed rather than assumed.

The plan is represented with the same economy. Until planning runs there is no plan allocation. If every row chooses the same variable, the depth stores only that variable and the rows remain an implicit consecutive selection. If rows disagree but their stable variable groups are already consecutive, only the group boundaries are stored. An explicit row-ordinal permutation exists only when grouping actually changes visitation order. These are different representations of the same exact row selection, not separate execution strategies.

Specificity is deliberately coarse. The sort key is the bit length of the estimate (ilog2(n) + 1), so counts inside the same power-of-two bucket are treated as equally specific; the tie-break then prefers the variable with the largest influence set — the one whose binding will sharpen the most other estimates. Two effects follow. Small differences between two sources' guesses cannot flip the order, which keeps the search stable when estimates are rough. And when the engine genuinely cannot tell two variables apart on cardinality, it picks the one that buys the most information.

Re-sorting on every step is what makes the ordering dynamic. A planner chooses one order from global statistics and lives with it for the whole query; here each level chooses from the estimates under the current partial binding. On skewed data this is the difference between a good average and a good worst case: the popular entity and the rare one take different paths through the same query, because after binding an entity the remaining estimates are no longer the same numbers. Nothing is cached, so nothing has to be invalidated. See the Atreides Join chapter for how estimate fidelity ranks and why this is worst-case optimal.

Proposals: one write-once buffer per level

propose writes into a ProposalBuffer — the engine's candidate store for one variable at one level. Entries are plain 32-byte RawInline values at fixed stride. Alongside them the buffer keeps a liveness bit per entry, packed 32 to a u32, and a u32 parent tag naming the frontier row the entry was proposed for.

Entries are effectively write-once. A proposer may rewrite the region it appended in the current call before it returns — that is how a union applies its sort-dedup — but once the caller can see the region, the indices are frozen, because kills bind to them.

Nothing is ever compacted. A candidate that fails confirmation has its liveness bit cleared and stays exactly where it was; the engine iterates the live entries. That costs a scan over dead entries and buys two things: an entry's index is stable for the lifetime of the level, so a kill can be recorded by index alone, and no confirmer ever has to agree with another about where a candidate now lives.

Packing liveness 32 to a word is what makes that scan cheap: count_live and next_live fold whole words through count_ones and trailing_zeros rather than looking at candidates one at a time. Two things get harder in exchange, and both are paid for inside the buffer's own module. A kill becomes a read-modify-write on a word shared with 31 neighbours. And a region no longer starts on a word boundary — nor does a per-parent run inside one — so its first and last words carry liveness bits owned by neighbouring regions of the same buffer. The invariant is enforced at the boundary of the type: every write path masks to the owned bits and every read path zeroes the bits it does not own, so no caller — the device confirm path included, see below — can reach a neighbour's liveness.

Confirmation is kill-only

Candidates is the region handed to confirm: values are read-only, liveness words are killable, and there is no way to revive an entry or add one.

That restriction is the whole reason confirmation needs no coordination. If a confirmer can only remove, then several confirmers writing into the same region compute their conjunction no matter how they are scheduled:

• Sequentially, each skipping entries that are already dead — the CPU path.
• In parallel, each on its own copy of the liveness words, merged with and_words — the path a batching accelerator takes.

Both schedules are legal, produce identical liveness, and the engine does not have to choose between them ahead of time. A union inverts the merge: it runs each live arm on a scratch copy and combines with or_words, so a value survives if any arm accepts it.

This is also what makes an accelerator safe to bolt on. A device that computes verdicts for a whole region cannot corrupt the search, because the only thing it can do with its answer is clear bits that the CPU would also have cleared.

What the engine refuses: resumable narrowing

An earlier design let a source deliver a level in geometrically growing chunks through a resumable cursor, so a level with a million candidates would not have to materialize a million before the engine could try the first one. It is gone, and the reasons are worth recording because the idea recurs.

It was never adopted by a single leaf source, so the machinery only ever ran its own default. It answers a time-to-first-result question that a pure conjunctive query does not ask — depth-first already yields the instant the stack bottoms out, and the measurement bore that out (0.004 ms against 0.017 ms for the design it was meant to improve). And its one real case, a wide root, is a lottery on iteration order rather than a saving: chunking helps only if the surviving candidates happen to sort early.

What did survive is the geometric part. The deleted cursor carried an INITIAL_CHUNK/WIDEN_FACTOR pair, and the residual engine before it grew its search width geometrically after negative work; both are the same idea, and both were attached to the wrong object. Attached to a level, growth asks a source to resume. Attached to the frontier, it asks nothing of anyone — the engine already owns how many parent rows it expands at a time, so INITIAL_FRONTIER_WIDTH and FRONTIER_RAMP_BASE buy the time-to-first-result property the cursor was reaching for with none of its protocol cost.

Narrowing a wide level is still a real problem; galloping intersection is the standing candidate for it. What the engine will not do is require a seek from sources, because that requirement is what would disqualify half of them.

Parallel execution

With the parallel feature a query is also a rayon producer: find!(...).into_par_iter(). There is no second solver behind it. Splitting walks the same state machine and transfers a whole frontier unit to a sibling:

• Once planning has partitioned a frontier by the rows' preferred next variable, one complete group may be transferred. The left producer advances its group cursor; the sibling is re-rooted at that frontier and fenced to the single group it received. Both share the immutable plan and frontier matrices behind Arc; proposal buffers are not divided.
• A published proposal buffer is immutable, so sibling binding stores share it behind Arc and copy only their consumption cursors. If one sibling later refills that variable, it installs a fresh empty buffer before proposing; the other sibling's indexes keep resolving through the old snapshot. A uniquely owned buffer keeps and reuses its allocation.
• A complete terminal page may likewise be transferred as one unit. This is how a one-variable query can expose parallel work without slicing its result frontier.
• A transfer is admitted only when the left producer has a distinct future: another group at this frontier, or an ancestor level with an unconsumed candidate suffix. A sole group or page with no such continuation stays in place and the producer descends serially; that descent may expose genuinely independent groups deeper in the search.
• The sibling owns no ancestor continuation. Once its fenced group or page is exhausted it ends instead of unwinding into work still owned by the left producer.
• Geometric pages and preferred-variable groups therefore remain intact. A split never turns one accelerator-sized proposal/confirmation batch into two smaller batches merely to feed rayon.
• A leaf just drives the ordinary sequential Iterator::next and folds the results. No engine logic is duplicated for the parallel path.

The ownership rule is itself the split bound: every successful split removes one group or page from the left producer and fences it into the right. Work cannot be handed back or rediscovered, so rayon's pressure splitter needs no engine-specific budget or thread-count tuning knob.

The guarantee is the same bag of rows, not the same order. Constraint trees are shared behind an Arc, so a split is a refcount bump rather than a tree clone; code that wants aggregate observations across shards needs its own synchronization (an Arc<AtomicU64>, say) because clone-local interior state is not a global counter.

Bag semantics at the interface

The engine emits one row per complete binding. When the unbound set empties, that assignment is a result. Nothing deduplicates it.

Hidden variables therefore surface as multiplicity. If an entity has n outgoing edges and a query projects only the entity while a temp! or _? variable ranges over the target, that entity is emitted n times:

#![allow(unused)]
fn main() {
use std::collections::HashSet;

use triblespace::prelude::*;

mod social {
    use triblespace::prelude::*;
    attributes! {
        "C21DE0AA5BA3446AB886C9640BA60244" as friend: inlineencodings::GenId;
    }
}

let mut kb = TribleSet::new();
let alice = ufoid();
let bob = ufoid();
let carol = ufoid();
kb += entity! { &alice @ social::friend: &bob };
kb += entity! { &alice @ social::friend: &carol };

// One row per complete binding: the hidden `_?friend` multiplies `?person`.
let rows: Vec<_> = find!(
    (person: Id),
    pattern!(&kb, [{ ?person @ social::friend: _?friend }])
)
.collect();
assert_eq!(rows.len(), 2);

// Deduplication is the consumer's job.
let distinct: HashSet<_> = find!(
    (person: Id),
    pattern!(&kb, [{ ?person @ social::friend: _?friend }])
)
.collect();
assert_eq!(distinct.len(), 1);
}

This replaced an engine that projected with SET semantics. That engine kept a claims table: an ordered key derived from the head's raw bytes, claimed before conversion, so a second binding with the same public identity was discarded. It was removed because the cost was structural rather than incidental. The table's memory grows with the result set, not with the query; under rayon the claim domain has to be shared across workers, which puts a synchronization point on the hot path of an otherwise share-nothing search; and once user code runs behind a claim, a conversion failure or a panic consumes the key, so another witness cannot retry it — a rule that is difficult to explain and easy to trip over. Worst of all, the multiplicity is genuine information about the data, and the engine was throwing it away on the way out.

Bag semantics is not the absence of a feature so much as the decision about where the feature belongs. Two idioms cover what the claims table used to do:

• Collect into a set. HashSet<_> (or BTreeSet<_>) after find! costs memory proportional to the distinct results — the same memory the claims table cost — but only when the caller actually wants it, and it deduplicates on the converted Rust values rather than on raw bytes.
• Two queries. Enumerate the outer variable, and use exists! for the inner condition. exists! stops at the first witness, so the fan-out is never enumerated at all. This is usually the faster answer, and it is the one that reads like the question being asked: "entities that have a friend", not "entities paired with a friend, deduplicated".

The unit head follows the same rule rather than getting an exception: find!((), constraint) yields one () per satisfying assignment. Use exists! when the question is existence.

Note that or! is a genuine exception at the binding step, not at the interface: a union sorts and deduplicates the candidate values it proposes for a single variable, so two arms that offer the same value for the same variable contribute one candidate, not two. That is a property of the candidate buffer for one level, and it does not extend to complete rows — the same alias witnessed by two different entities is still two rows.

Candidate uniqueness is scoped to one proposal region and keyed by (parent tag, value): equal values belonging to different parent bindings are different members of the relation. Exactness, uniqueness, and physical order are separate properties—knowing that a producer enumerated its complete input does not prove that its output is duplicate-free, and neither property promises an order. When regions over the same parent domain reconverge, as union arms do, uniqueness must therefore be re-established by deduplication unless their disjointness is proved.

Constants live below the variable layer

A pattern position is a Term: either a Var the engine solves for, or a Const pinned at construction. The macro layer folds attribute constants, literal values, and constant entity ids into Const terms.

Constants never enter a Binding, are never proposed, and are excluded from variables(). They behave exactly like a variable that was already bound — RawTerm::position_value returns the pinned value where it would return the binding's value — so every backend's bound/unbound dispatch handles them with no extra match arms.

Keeping them below the variable layer is what makes or! usable. Every arm of a union must declare the same variable set, because the result schema is flat: one row binds one variable set exactly once, so a variable that only exists in some alternatives has nowhere to live. If a literal allocated a hidden variable, then two arms matching different attributes would declare different sets and the union would be rejected at construction — which was exactly the symptom before constants became Term-native. As folded constants they cost no variable, so an arm on profile::nickname and an arm on profile::display_name declare the identical set and compose.

Two more consequences fall out. Literals do not consume the 128-variable budget, so a pattern with 161 constants allocates zero variables. And a pattern whose positions are all constant has an empty variable set, which the search would never visit — so Query::new settles it once, up front, with a single satisfied call against the empty binding. This is why satisfied must be exact once every variable it touches is bound: for a constant subtree there is no variable through which the search could later discover failure.

Where the GPU fits

The optional triblespace-gpu crate accelerates one operation: WgpuSuccinctArchive keeps a succinct archive's value universe, per-axis occupancy boundaries, and six Jerky wavelet matrices resident on the device, and routes a confirm region to a kernel when it has at least DEFAULT_MIN_CONFIRM_BATCH live candidates.

It is not a second engine and not a planner. Estimates, proposals, prefix walks, and satisfaction checks stay on the CPU; a region below the threshold and any device error fall through to the canonical CPU arm, which the crate's parity suite holds to identical liveness words. The substitution is legal precisely because of the kill-only contract: verdicts computed anywhere merge back by word-wise AND, and a device can never revive a dead entry.

The packed liveness layout shapes how a kernel writes its verdicts. The flat index of a verdict kernel is the bit position in the region's liveness words — so candidate i sits at bit bit_offset + i — and one plane_ballot per 32-lane plane produces a whole packed verdict word with every bit already in place, stored by a single lane. No rotation, no read-modify-write, no atomic. The AND and the write-back are ordinary word operations, because the device works on a private copy taken through live_words() and merged through set_live_words(), and those two mask the neighbouring regions' bits out and back in. The kernel needs one device property for this — planes exactly 32 lanes wide — which the host checks before dispatching, demoting to the CPU arm if it does not hold.

The threshold is measured, not guessed. On an Apple M4 Max (Metal via wgpu), against a 262,135-trible archive with fully live regions, the GPU round trip is nearly flat at ~1.4–2.2 ms while CPU cost scales linearly — putting the crossover near 8k live candidates for the range shape (two wavelet ranks per candidate) and near 22k for the lighter membership shape. 16,384 is the single-knob compromise between them; the full crossover table lives in the constant's doc comment. WgpuSuccinctArchive::stats exposes dispatch and fallback counters, so the routing economics stay observable instead of being hidden inside a heuristic.

Where regular paths went

Regular path queries — "everyone reachable through a chain of follows", "all ancestors via repeated parent" — are no longer part of the engine. The path! macro and its query-time evaluator have been removed.

The reason is the stateless protocol. A triple pattern is a relation over a fixed set of variables, and the engine can ask it for candidates under any binding. A regular path is an automaton product traversal: evaluating it inside the search needs per-activation state — where the frontier is, which automaton branch produced which value, what has already been visited — and that state belongs to one live traversal, not to a binding the engine can hand back later. Every attempt to keep it inside the protocol grew the protocol, and the growth did not stay confined to paths: pager hooks, activation receipts, and novelty keys became things every constraint had to reason about, including the ones that only ever wanted to answer "is this triple present".

So paths moved out of query time entirely. The replacement is a materialized closure index: compile the graph edges and an epsilon-free automaton into a product graph and maintain its reflexive transitive closure, then let queries read the closure as an ordinary relation. Reachability becomes a lookup rather than a traversal, which is the right shape for a data model where facts are only ever added — the closure grows monotonically with the edge set.

The stable implementation lives in the standalone triblespace-paths crate. Its PathExpr builder compiles the normal regular-language operations into a fixed epsilon-free automaton; direct Automaton construction remains the low-level escape hatch. The crate persists unionable direct-product summaries over exact source collection elements, closes their selected union, and exposes the resulting endpoint relation through the ordinary two-variable constraint protocol. See Regular Path Indexes for the expression API, lifecycle, freshness boundary, and dense-output trade-off.

What the engine will not do

Four refusals, and what each one buys:

• No cost-based optimizer. There is no plan to compile, no statistics to collect, no cardinality model to keep calibrated, and no plan cache to invalidate. What replaces it is the per-step ordering above, which sees the actual partial binding instead of a summary of the data. The cost is that a genuinely bad estimate cannot be corrected by anything but a better estimate; the benefit is that adding a data source means implementing four methods, not teaching a planner about a new operator.
• No negation. There is no MINUS, no FILTER NOT EXISTS, no OPTIONAL. This is a data-model decision reaching up into the engine: the trible model is monotonic, and a non-monotonic operator would make a query's answer depend on facts not being present — which stops being a stable statement the moment another replica merges in. Monotonicity is what makes pattern_changes! sound, what makes distributed merge coordination-free, and what makes a query result something you can still believe after a pull.
• No query-time recursion. See the section above. Recursion returns as a maintained index, where the fixpoint is computed once against the data rather than repeatedly inside every search.
• No projection dedup. Covered above: the multiplicity is real, the deduplication has a cost, and the consumer is the one who knows whether it wants to pay it.

What the engine does provide in exchange is a short list, but a durable one: predictable latency, skew resistance, no tuning knobs, and one protocol that a TribleSet, a compressed on-disk archive, a HashSet, a search index, and an application predicate all speak equally well.

Queries as Schemas

You might notice that trible.space does not define a global ontology or schema beyond associating attributes with a InlineEncoding or BlobEncoding. This is deliberate. The semantic web taught us that per-value typing, while desirable, was awkward in RDF: literal datatypes are optional, custom types need globally scoped IRIs and there is no enforcement, so most data degenerates into untyped strings. Trying to regain structure through global ontologies and class hierarchies made schemas rigid and reasoning computationally infeasible. Real-world data often arrives with missing, duplicate or additional fields, which clashes with these global, class-based constraints.

Our approach is to be sympathetic to edge cases and have the system deal only with the data it declares capable of handling. These application-specific schema declarations are exactly the shapes and constraints expressed by our queries¹. Data not conforming to these queries is simply ignored by definition, as a query only returns data satisfying its constraints.²

Join Strategy

The query engine uses the Atreides family of worst-case optimal join algorithms. These algorithms leverage the same cardinality estimates surfaced through Constraint::estimate to guide variable choice over partial bindings, providing skew-resistant and predictable performance. Estimates are recomputed from the current binding rather than cached, so there is no invalidation protocol and no separate planning artifact to maintain. For a detailed discussion, see the Atreides Join chapter.

Query Languages

Instead of a single query language, the engine exposes small composable constraints that combine with logical operators such as and and or. These constraints are simple yet flexible, enabling a wide variety of operators while still allowing the engine to explore the search space efficiently.

The query engine and data model are flexible enough to support many query styles, including graph, relational and document-oriented queries. Constraints may originate from the database itself (such as attribute lookups), from custom application logic, or from entirely external sources.

For example, the pattern! and entity! macros—available at the crate root and re-exported via triblespace::prelude (for instance with use triblespace::prelude::*;)—generate constraints for a given trible pattern in a query-by-example style reminiscent of SPARQL or GraphQL but tailored to a document-graph data model. It would also be possible to layer a property-graph language like Cypher or a relational language like Datalog on top of the engine.³

use std::collections::HashSet;

use triblespace::core::examples::literature;
use triblespace::core::query::ContainsConstraint;
use triblespace::prelude::*;
use triblespace::prelude::blobencodings::UTF8String;
use triblespace::prelude::inlineencodings::Handle;

fn main() {
    let mut kb = TribleSet::new();

    let author = ufoid();
    let book = ufoid();

    kb += entity! { &author @
        literature::firstname: "Frank",
        literature::lastname: "Herbert",
    };
    kb += entity! { &book @
        literature::author: &author,
        literature::title: "Dune",
    };

    let mut allowed = HashSet::<Inline<Handle<UTF8String>>>::new();
    allowed.insert("Frank".to_inline());

    let results: Vec<_> = find!((title: Inline<_>, firstname: Inline<_>),
        and!(
            allowed.has(firstname),
            pattern!(&kb, [{
                _?person @
                    literature::firstname: ?firstname,
                    literature::lastname: "Herbert",
            }, {
                literature::author: _?person,
                literature::title: ?title,
            }])
        )
    )
    .collect();

    assert_eq!(results.len(), 1);
    assert_eq!(results[0].0, "Dune".to_inline());
}

The snippet above demonstrates how typed attribute constraints, user-defined predicates (the HashSet::has filter), and reusable namespaces can mix seamlessly within a single query.

Great care has been taken to ensure that query languages with different styles and semantics can coexist and even be mixed with other languages and data models within the same query. For practical examples of the current facilities, see the Query Language chapter.

The Atreides Family of Worst-case Optimal Join Algorithms

The query engine reasons about data by solving a set of constraints over variables. Instead of constructing a traditional left-deep or bushy join plan, it performs a guided depth-first search that binds one variable at a time. The approach draws on the broader theory of worst-case optimal joins and lets us navigate the search space directly rather than materialising intermediate results.

Constraints as the search frontier

Every constraint implements the Constraint trait, whose methods shape the search:

1. variables – returns the set of variables this constraint touches.
2. estimate – predicts how many candidates remain for a variable under the current partial binding, or None when the variable is not this constraint's business.
3. propose – enumerates candidate values for a variable, for every binding in the current frontier.
4. confirm – kills candidates proposed by another constraint, without re-enumerating them.
5. satisfied – returns false when the constraint has no completion from the current binding. Used by UnionConstraint to prune dead variants.

Traditional databases rely on a query planner to combine statistics into a join plan. Atreides instead consults the constraints directly while it searches. Each constraint can base its estimates on whatever structure it maintains—hash maps, precomputed counts, or even constant values for predicates that admit at most one match—so long as it can provide a quick cost quote. Every binding decision asks for fresh estimates under the binding that exists at that moment. Nothing is cached, so there is no invalidation protocol to get wrong.

An estimate affects cost ordering only. It cannot change whether a constraint is relevant, which candidates it proposes, or which rows the query returns. A constraint that misreports its cardinality by a large factor makes the search slower; it does not make it wrong.

Because the heuristics are derived entirely from the constraints themselves, we do not need a separate query planner or multiple join implementations. Any custom constraint can participate in the same search by providing sensible estimates, proposal generation, and confirmation.

A spectrum of Atreides variants

The Atreides "family" refers to the spectrum of heuristics a constraint can use when implementing Constraint::estimate. Each variant exposes the same guided depth-first search, but with progressively tighter cardinality guidance. Every binding decision revisits its estimates; what differs is what quantity they approximate:

• Row-count Join (Jessica) estimates the remaining search volume for the entire constraint. If one variable is bound but two others are not, Jessica multiplies the candidate counts for the unbound pair (|b| × |c|) and reports that larger product. The number can wildly overshoot the next variable's frontier, yet it often tracks the overall work the constraint will perform.
• Distinct-value Join (Paul) narrows the focus to a single variable at a time. It returns the smallest proposal buffer the constraint could produce for any still-unbound variable, ignoring later confirmation filters. This is the behaviour exercised by Query::new today, which keeps the tightest candidate list on hand while the search walks forward.
• Partial-binding Join (Ghanima) goes further by measuring the size of the actual proposal the composite constraint can deliver for the current binding and chosen variable. For an and constraint this corresponds to the intersection of its children after they have applied their own filtering, revealing how many candidates truly survive the local checks.
• Exact-result Join (Leto) is an idealised limit where a constraint predicts how many of those proposed values extend all the way to full results once the remaining variables are also bound. Although no constraint currently achieves this omniscience, the interface supports it conceptually.

All four share the same implementation machinery; the difference lies in how aggressively estimate compresses the constraint's knowledge. Even when only partial information is available the search still functions, but better estimates steer the traversal directly toward the surviving tuples.

Every constraint can decide which rung of this ladder it occupies. Simple wrappers that only track total counts behave like Jessica, those that surface their tightest per-variable proposals behave like Paul, and structures capable of intersecting their children on the fly approach Ghanima's accuracy. The engine does not need to know which variant it is running—estimate supplies whatever fidelity the data structure can provide.

Guided depth-first search

At query start, Query::new asks every variable for an estimate against the empty binding, settles any constraint that is already fully determined by its constants, and orders the unbound variables. The solver then repeats one negotiation per binding:

1. Refresh the estimates that the most recent binding could have disturbed — the influence sets of the variables bound since the last refresh, minus the ones already bound.
2. Re-sort the unbound variables and take the most specific one. The ordering is by candidate-count bit length (smaller first), so counts in the same power-of-two bucket are deliberately treated as equally specific; ties go to the variable that influences the most others.
3. Ask the constraint tree to propose for that variable. An intersection lets its tightest child propose and runs the remaining children as confirmers over that child's output, most selective first, so what reaches the engine has already survived every clause.
4. Bind the first surviving candidate and descend.
5. When a level's candidates are exhausted, unbind the variable, return it to the unbound set, and continue one level up.

Traditional databases rely on sorted indexes to make the above iteration tractable. Atreides still performs random lookups when confirming each candidate, but the cardinality hints let it enumerate the most selective constraint sequentially and probe only a handful of values in the wider ones. Because the search is depth-first, the memory footprint stays small and the engine can stream results as soon as they are found.

Consider a query that relates ?person to ?parent and ?city. The search begins with all three variables unbound. If ?city only has a handful of possibilities, its estimate will be the smallest, so the engine binds ?city first. Each city candidate is checked against the parent and person constraints before the search continues, quickly rejecting infeasible branches before the higher-cardinality relationships are explored.

Per-variable estimates in practice

Suppose we want to answer the following query:

(find [?person ?parent ?city]
  [?person :lives-in ?city]
  [?person :parent ?parent]
  [?parent :lives-in ?city])

There are three variables and three constraints. Every constraint can provide a cardinality hint for each variable it touches, and the combined query records the tightest estimate for each variable:

The estimates are scoped to individual variables even when no single constraint covers the whole tuple. The engine chooses the variable with the tightest bound, ?parent, and asks the constraints that mention it for proposals. Each candidate parent immediately passes through the ?parent :lives-in ?city constraint, which usually narrows the possible cities to a handful. Those cities, in turn, constrain the possible ?person bindings. If a branch fails — for example because no child of the selected parent lives in the same city — the engine backtracks and tries the next parent. The smallest estimated constraints therefore guide the search towards promising combinations and keep the depth-first traversal from thrashing through unrelated values.

Implementation notes

• The search state is a stack of row frontiers over reusable per-variable candidate buffers. Retiring a frontier unsets its variable and pops; because constraints are stateless, nothing has to be notified or unwound.
• Constraints propose a complete candidate region for each parent batch; they do not implement cursors or seek. The engine resumably consumes its own buffer into child frontiers whose width ramps through 1, 8, 64, 512, … up to the query ceiling. This protects the first result from batching overhead while still presenting accelerators with wide regions deeper in the search.
• Highly skewed data still behaves predictably: even if one attribute dominates the dataset, the other constraints continue to bound the search space tightly and prevent runaway exploration. This is the payoff of re-estimating per binding rather than once per query — the popular entity and the rare one take different orders through the same query text.
• A uniquely owned per-variable proposal buffer is reused across sibling levels, so ordinary backtracking does not reallocate. Rayon clones share a published buffer immutably and keep independent cursors; the first clone to refill that slot installs a fresh buffer rather than copying data it will immediately clear.
• Under the parallel feature the same state machine is the rayon producer: splitting transfers one whole preferred-variable group (or terminal page) to a fenced sibling only when the left side retains another continuation. Candidate regions and geometric frontier pages stay intact. Results are the same bag of rows in an unspecified order.

Why worst-case optimal?

"Worst-case optimal" does not mean output size plus a constant factor: a query with an empty result may still need to inspect substantial input. It means matching, up to implementation and logarithmic factors, the worst-case output bound implied by the input relation cardinalities (the AGM/fractional-edge-cover bound), rather than materialising pairwise intermediates that can be asymptotically larger.

The Atreides family follows the generic-join shape behind that result: choose a variable, let the tightest participating constraint enumerate its possible values, and intersect those values through the other constraints before descending. Dynamic cardinality estimates choose among valid variable orders; they improve the realised work on skewed data, while the propose/confirm intersection is the part that avoids oversized binary-join intermediates. The precise guarantee still depends on participating constraints providing complete proposals and sound confirmations—the scheduler cannot manufacture those laws for an arbitrary custom data source.

This combination of simple heuristics, incremental estimates, and a disciplined search strategy keeps the implementation straightforward while delivering the performance characteristics we need for real-world workloads.

The Type Algebra of TribleSpace

Queries as Types, Data as Proofs

TribleSpace grew out of a pragmatic goal: keep data declarative, composable, and statically checkable in Rust. Along the way we discovered that the core operations already form a type algebra. The macros used to define attributes, entities, and queries map directly onto familiar type-theoretic constructions, yielding an isomorphism between relations over triples and types over records.

Attributes Introduce Atomic Types

Each attribute introduced with attributes! defines an atomic type — a unary relation between an entity identifier and the attribute’s value:

"A74AA..." as pub title : Handle<UTF8String>

Formally this is a function title : Id → ValueTitle, or, in relational terms, the set { (id, value) }. In the codebase the macro emits a static LazyLock<Attribute<S>> for each declared attribute, so the generated binding already carries the InlineEncoding that governs the value column. The derived form (omit the hex id) hashes the attribute name together with the schema metadata via the entity-core mechanism — equivalent to Attribute::<S>::named(name) — which is convenient for quick experiments; shared protocols should still pin explicit ids so collaborators and other languages read the same column. These atomic pieces are the building blocks for everything else.

Entities as Intersection Types

An entity! expression forms a record type by intersecting atomic ones. Semantically it is the meet (∧) of its constituent relations:

Entity{A, B}  ≡  { A : ValueA } ∧ { B : ValueB }

At runtime entity! expands to a small Fragment: exported entity ids, a TribleSet containing the asserted facts, descriptive metafacts for the attributes that emitted those facts, and one blob store shared by both sets. The description channel does not participate in intrinsic entity identity or ordinary content queries. Use += to union every channel, and use root()/exports() when you want the produced entity id(s) back alongside the facts. At the type level it represents their conjunction. Records are therefore intersection types: every additional field refines the shape without invalidating existing data.

Merge Drives Dataset Composition

For content facts, the += operator delegates to TribleSet::union, exposed through the AddAssign implementation. Fragment composition applies the same set union independently to facts and metafacts, unions exports, and structurally joins the shared blob stores. TribleSet::union performs set union on the six internal indexes that back a TribleSet. When the entity identifiers are disjoint the effect is classic dataset union; when they coincide we get field conjunction—record extension. This dual role is what keeps TribleSpace’s algebra compact.

One Operator, Two Readings

Everything that looks like “add another field” or “add another record” derives from the single law

union(facts₁, facts₂) = facts₁ ∪ facts₂.

The surrounding context determines how to read the result:

The behaviour follows from the idempotent, commutative nature of TribleSet::union; no special cases are required in the implementation.

Patterns as Row-Type Predicates

A find!/pattern! pair behaves like a row-type predicate:

pattern!([A, B])  ≈  ∀r. {A, B} ⊆ r.

Read this as “find every entity whose row type r is a supertype of this shape.” The macro expands to an IntersectionConstraint built from TriblePattern constraints, so queries literally evaluate the conjunction of the row predicates.

Patterns compose intersectionally as well: combining two patterns is equivalent to requiring both row predicates simultaneously, mirroring intersection types at the query level.

The Lattice Perspective

TribleSets form a join-semilattice under union:

union(a, b) = a ∪ b
a ≤ b  ⇔  a ⊆ b.

Projection, join, and filtering are lattice homomorphisms: they preserve joins and the partial order. Because the same algebra handles both data and metadata, the implementation remains uniform and performant—merging facts is always the same low-level operation.

Practical Payoffs

Seeing the primitives through a type-theoretic lens clarifies several ergonomic choices:

• Queries as proofs. Successful find! rows witness that an entity inhabits the requested type; absence is simply failure to prove the predicate.
• Descriptive schemas. Structural typing drops the need for globally declared records—the type system is implicit in the patterns you write.
• Composable extensions. Adding new attributes is monotonic: existing entities continue to satisfy prior predicates, and refined queries simply intersect more row constraints.

Summary Table

With a single associative, commutative, idempotent union, we obtain both row extension and dataset union, and a unified logical framework that bridges data engineering with type theory. That economy of primitives allows the system to feel simple on the surface yet provide rich type theory underneath.

Architecture Overview

TribleSpace is an embedded knowledge graph whose storage and distribution model is a small join algebra. Facts, blobs, and publication records are immutable; independent stores combine by union. The architecture follows from that choice rather than wrapping an ordinary mutable database in a replication protocol.

The load-bearing principles

Content addressing

Every blob is named by a hash of its bytes. Identical values deduplicate, readers can validate integrity without trusting storage, and a reference has the same meaning in memory, in a pile, or on an object store. Handles fit in a trible's 32-byte value slot, so descriptions, datasets, metadata, and large attachments all use the same reference primitive.

Monotone evidence

A TribleSet is a set of immutable facts. A CollectionStore is a set of immutable algebra records. Their merge is set union, which is associative, commutative, and idempotent. Concatenating independently written piles therefore cannot create a last-writer-wins conflict or change the meaning of an existing record.

This is the practical consequence of the CALM principle: monotone conclusions do not need a coordination protocol. Application-level change is represented explicitly as new facts, version links, or successor DAGs rather than by overwriting an ambient current value.

Authority and computation are separate

A signed COMMIT says that an author places one element in a collection. A signed MERGE or DERIVE endorses a reproducible computation connecting known elements. The former introduces membership; the latter lets a reader reuse materialized work from a producer admitted by the target's WRITE policy. A signature identifies the endorser; it does not prove the equation correct. Readers do not repeat the encoding join or mapping to check it.

Foreign record bytes are signature-checked once at ingress. Typed local publication and trusted store replay do not repeat that check. Admission still uses the proofs and resident capability definitions of each frozen snapshot: a later grant or newly resident definition may activate an old equation. Generic collection authority does not expire as the clock advances.

That separation is why a materialized index does not become ground truth merely because it is convenient, and why collecting an accelerator does not erase the committed facts from which it can be rebuilt.

Who may make that signed assertion can be proven without making storage a policy oracle. A descriptor carries independent READ and WRITE policies. Each is open or a canonical quorum over external trust roots with one semantic threshold. Each resident proof is one self-contained, prefix-signed path from one root: its header binds the exact resource, and every edge carries a capability definition handle, delegate, and signature over the complete prefix. Definitions describe independent invocation and delegation action sets; a child's actions must all be delegated by its parent. The handles may differ. Ordinary collection operations count independently valid rooted prefixes against ACTION_WRITE on the descriptor, with roots selected for that action rather than the grant's exact handle. Sibling paths cannot lend one another delegation authority, and merely finding an unverified or irrelevant proof in storage grants nothing.

Architectural layers

┌──────────────────────────────────────────────────┐
│ Application                                      │
│ entity! · Fragment · find! · pattern!            │
├──────────────────────────────────────────────────┤
│ Typed collections and immutable observations     │
│ publish, select exact covers, materialize views  │
├──────────────────────────────────────────────────┤
│ Collection algebra                              │
│ signed COMMIT · stored MERGE · stored DERIVE     │
├──────────────────────────────────────────────────┤
│ Storage                                          │
│ CollectionStore · BlobStore · WantStore          │
│ CapabilityProofStore                             │
├──────────────────────────────────────────────────┤
│ Data and representations                         │
│ TribleSet/PATCH · SimpleArchive · SuccinctArchive│
└──────────────────────────────────────────────────┘

The boundaries are deliberately narrow. Query constraints do not know how bytes were published. A collection encoding or mapping does not decide replication policy. A blob store does not infer authority from a handle it happens to contain.

Tribles, sets, and fragments

A Trible is a fixed 64-byte entity–attribute–value fact. The attribute determines how the value's 32 bytes are interpreted. A TribleSet stores each fact once while maintaining the six entity/attribute/value permutations needed by the query engine. The underlying persistent adaptive tries make cloned sets cheap and union, intersection, and difference structural operations.

A Fragment is the publication unit applications normally construct. It carries:

• ordinary facts;
• descriptive metafacts;
• exported intrinsic entity IDs; and
• the content-addressed attachments referenced by either fact set.

Fragments compose with +=. The entity! macro derives an ID when no explicit subject is supplied and inserts encoded blob payloads into the fragment's attachment store. This keeps provenance and required bytes together without mixing schema descriptions into ordinary application queries.

Blob storage

BlobStorePut, BlobStoreGet, BlobStoreMeta, and BlobStoreList describe small independent capabilities rather than one all-or-nothing database trait. The main backends are:

• MemoryRepo for process-local work and tests;
• Pile for one append-only, memory-mapped file; and
• ObjectStoreRemote for S3-compatible storage.

Content addressing makes storage placement a physical concern. A missing local blob is not a semantic retraction: another node may still provide the exact bytes later.

Collections are self-describing lattices

A collection descriptor is an ordinary TribleSet, encoded as a canonical SimpleArchive. Its content handle is the CollectionHandle. A root descriptor normally states:

• a human-readable UTF-8 name;
• the canonical member encoding; and
• independent descriptor-local READ and WRITE admission policies.

A derived descriptor replaces the name with its source collection and one concrete mapping entity. Canonical builders derive that entity's id, while readers preserve the substitution rule by validating its algorithm and parameters instead of its minting history. The entity names a mapping algorithm and carries its concrete parameters as ordinary tribles. The target encoding and both policies remain local to the derived descriptor; policy never inherits from the source. Descriptions of the encoding and mapping algorithm travel in the same archive, so a record naming the descriptor remains interpretable without a separate registry entry.

CollectionStore contains three native record kinds:

The exact canonical record value is the semantic object; none of the three has a synthetic entity ID. A repeat insert is a no-op. The network PATCH keys a record by a full-width BLAKE3 fingerprint of the kind and canonical payload; the pile keys it by the collection and member it names and the frame it sits in. Neither key is collection semantics. All three records are signed. A COMMIT or a DERIVE is admitted by WRITE on the collection it names. A MERGE needs no WRITE: a store folds only the MERGEs its own host key signed, each host joining the same foundations into a lattice of its own, and another key's MERGE stays in the store and folds into nothing. Several keys can sign the same DERIVE: these are distinct provenance records but one semantic equation. Inferred equations carry only endpoints, not fabricated signatures or provenance records.

The algebra has no distinguished head. Several commits coexist, and the value of a selected collection view is the join of its admitted members. This makes a commit the atomic publication boundary without inventing a mutable register above it.

Publishing and observing

The collection value is its canonical descriptor handle, and the storage backend owns I/O and durability. store.collection(name, policy) constructs and registers a canonical root descriptor. store.derive::<Target>(source, argument, policy) does the same for a target-owned canonical derivation, and store.attach::<Target>(parent, argument) for an index attached to a root, whose descriptor carries no policy. Mappings between two foreign encoding types use the explicit store.derive_with(source, mapping, policy) and store.attach_with(parent, mapping) coherence seams. store.commit(collection, signer, fragment) publishes attachments, canonical data, canonical metadata, and the signed native record in dependency order. Local publication performs no authorization check and no implicit flush: authorization governs which resident assertions another operation admits, not what a process may append to its own store.

Reads are exact about what they observed, not magical about global time:

• store.snapshot() freezes blob bytes, collection records, capability proofs, and backend state from one coherent known prefix;
• collection.admitted(&snapshot) reads the coverage index in that observation -- every record decided once against the descriptor's WRITE policy and the resident capability evidence -- and returns one semantic Cover<E>, the union of what the collection's frontier stands for, without fetching member data; and
• collection.read(&snapshot) attaches that frontier from the same index and reconstructs the logical value from the nodes whose bytes are resident, descending through the MERGE that produced a node whose bytes are not, through that same immutable observation.

Snapshots carry no clock. Content-change masks compare stored evidence, and collection authorization does not expire with wall time. An application action such as Secrets key delivery supplies its own evaluation time and interprets its deadline facts before counting each root's proof prefix; that does not expire collection membership.

Cover identity is the collection descriptor plus distinct payload handles. Signer, signature, and metadata attestations currently known to the store remain queryable, but no attestation is required for replay, and another one over the same payload does not change the cover or repeat data work.

Each store snapshot observes one known prefix of an append-only store. A concurrent commit may appear now or on the next call, but one observation never combines records from one prefix with payloads from another. The WRITE-policy roots and every signer with sufficient valid resident support participate; unauthorized commits remain inert.

Derived physical representations

The same logical data can be projected into encodings optimized for a particular task. A canonical SuccinctArchive collection or a regular-path summary is connected to its source by a mapping which is a join homomorphism:

f(a ⊔ b) = f(a) ⊔ f(b)

That law makes equations on either side of the mapping reusable evidence. Construction and representation maintenance are separate operations. The foundational support is always Support = Cover<SimpleArchive>: the admitted committed payloads at the root of the descriptor lineage. Mapping and merging never change those denotational coordinates. Multi-hop maintenance invokes each mapping in order over whatever its immediate source already realizes, so every target ends up standing for what its source's frontier stands on. There is no way to ask for a narrower support: a support is what a target stands on, read back from it, never an argument to select by. Two targets derived from one source and read from one snapshot agree on it by construction.

ensure is a live asynchronous store operation. For a root SimpleArchive collection it acquires missing blobs for the admitted foundational support without publishing equations. For a derived target it takes the support already realized by resident immediate-source members, reuses resident target nodes and stored equations, acquires exact missing dependencies for the selected work, and publishes only the missing DERIVE work for its one immediate mapping. It never creates a MERGE, computes an upstream blob, or emits a durable WANT. maintain first performs that same realization work and then repeatedly joins target members in the deterministic dyadic serialized-size tiers. A target join whose exact immutable dependency cannot be acquired leaves a finer target cover in place. Different target covers may denote the same join, and every lattice position uses the same Cover<E> shape while retaining its own typed member handles. Missing derived artifacts are cache misses, not missing facts.

Every store-level ensure or maintain operation returns a fresh StoreSnapshot. The result is the post-operation temporal boundary, including work concurrently published before that snapshot. Read-only CollectionSnapshotExt::collection selects a resident target cover from one such store snapshot and returns a CollectionSnapshot which owns that immutable observation and the realized target cover; the foundational support it stands on is a separate support() query over the same observation. A caller chooses the logical projection later with CollectionSnapshot::view.

Cover carries no route mode. The resolver checks its explicit members first and, only when needed, widens through stored MERGE equations to a resident equal-support route. Rank9 acceleration is another ordinary derived collection. Its members are ordinary Blob<Rank9AcceleratedSuccinctArchiveBlob> values whose first 32 bytes name the exact portable SuccinctArchiveBlob child carrying their source rows. The root and named child together are a complete source-bound accelerated encoding, not a separate sidecar or runtime artifact.

Both raw Succinct and Rank9-accelerated members own canonical joins. The Rank9 join computes the same union, but its result names the exact raw Succinct union as an immutable child. It may consume that raw union when the blob is already resident; it never creates the upstream raw blob or its MERGE record. If the child is absent, target maintenance leaves the exact finer accelerated cover in place. A caller which wants both lattices compact invokes ordinary maintenance on their mapping hops explicitly, in order. Exact requests can instead pin the same foundational support across both hops. A cover-aware view follows each embedded handle through its store snapshot, validates the exact raw/index pair, and only then builds the transient query runtime. A root whose raw child is absent is not a usable query value.

WANT is operational, not semantic

A WantStore records operational interest in obtaining a blob. Blob(H) is the only request shape and names no collection. WANTs do not add collection members, authorize authors, retain all referenced data, or force another node to perform work. They are durable coordination-free questions which a reconciler may satisfy by fetching content into the local store.

The network reconciler discovers exact providers under the opaque, domain-separated locator KDF(H). An H-bound endpoint token rejects false directory entries before dialing. The direct stream then performs a provider-first, requester-second proof of H, bound to both authenticated endpoint identities, without ever transmitting H. Returned bytes must hash to H. This path does not load a collection descriptor or consult READ(C).

Keeping WANT orthogonal prevents evidence convergence from becoming involuntary blob mirroring. A peer can repair a READ-authorized collection's record and authorization-evidence overlay, then decide which blobs and derived representations are useful locally.

Routing is soft state, not semantic evidence

First contacts come from the pile, not from configuration: the keys a selected collection's records and grants name, and for a process beside the pile's sync daemon, the addresses that daemon recorded in the pile's configuration. Peerings, candidate orders, DHT referrals, connection liveness, and provider leases are bounded process-local state. Restarting may forget them without changing a collection, and none of them authorize a peer, promise content residency, or retain a blob. Historical PEER and STORE_SCOPE records remain physically decodable for old piles but do not participate in current synchronization or repository snapshots; semantic rewrites drop them.

Storage and synchronization compose by union

Pile stores blobs, native collection records, capability proofs, and WANT records in one append-only log. ObjectStoreRemote places immutable collection records under content-derived object keys. The network layer uses READ(C)-gated Merkle pushes between peered neighbours, each side pushing its own collection trees on a timer, to union that collection's records and structurally relevant native READ(C)/WRITE(C) proof records. Each proof record is the complete authorization value and has no referenced blob closure. Independently, every resident blob may publish an opaque XOR-DHT lease under KDF(H); knowing H is the bearer capability for its exact bytes regardless of collection policy or collection-repair direction. Collection READ(C) gates collection anti-entropy, but never exact GET. Merge/derive questions are answered from the converged local record index. In every case convergence means unioning evidence; it does not mean electing a winner.

Legacy branch and pin records remain decodable only so old piles can be inspected, conservatively retained, and explicitly migrated. They are not part of the current publication or authorization model.

Patterns & Recipes

This chapter collects idiomatic solutions to common problems. Each recipe is self-contained — jump to the one that matches your situation.

Modeling relationships

One-to-one

An entity links to exactly one other entity via a GenId attribute:

attributes! {
    "..." as author: GenId;
}

let book = fucid();
let writer = fucid();
change += entity! { &book @ literature::author: &writer };

Query both directions: the attribute stores the forward link, and the query engine can traverse it in reverse by binding the value and querying for the entity:

// Forward: who wrote this book?
find!(author: Id, pattern!(&catalog, [{ book_id @ literature::author: ?author }]))

// Reverse: what did this author write?
find!(book: Id, pattern!(&catalog, [{ ?book @ literature::author: author_id }]))

Many-to-many

Use a repeated attribute — the same entity can have multiple values for one attribute. The entity! macro supports this with the * spread syntax:

let paper = fucid();
let tag_ml = fucid();
let tag_neuro = fucid();
change += entity! { &paper @ metadata::tag: &tag_ml };
change += entity! { &paper @ metadata::tag: &tag_neuro };

Or in a single entity expression:

let tags = vec![tag_ml, tag_neuro];
change += entity! { &paper @ metadata::tag*: tags.iter() };

Query all tags for an entity, or all entities with a tag:

// All tags on this paper
find!(tag: Id, pattern!(&catalog, [{ paper_id @ metadata::tag: ?tag }]))

// All papers with this tag
find!(paper: Id, pattern!(&catalog, [{ ?paper @ metadata::tag: tag_id }]))

Hierarchies (parent/child)

Model with a parent attribute. Children point up to their parent:

attributes! {
    "..." as parent: GenId;
}
change += entity! { &child @ tree::parent: &parent_node };

A fixed number of hops is just a chain of clauses, joined on temp! variables:

// Grandparents: two hops up.
find!(grandparent: Id,
      temp!((mid),
            and!(pattern!(&catalog, [{ node_id @ tree::parent: ?mid }]),
                 pattern!(&catalog, [{ ?mid @ tree::parent: ?grandparent }]))))

Unbounded traversal — all ancestors, all descendants — is handled by the standalone triblespace-paths closure index rather than by query-time recursion. Describe one-or-more parent edges with PathExpr, compile the expression, materialize it once, and use its endpoint relation as an ordinary constraint:

let parent = tree::parent.id().into();
let parent_plus = PathExpr::from(Step::Forward(parent)).plus();
let ancestors = PathIndex::from_tribles(parent_plus.compile(), catalog.iter())?;
let node: Inline<GenId> = node_id.to_inline();

for ancestor in find!(
    ancestor: Id,
    ancestors.constraint(node, ancestor)
) {
    println!("ancestor: {ancestor:?}");
}

For collection-backed maintenance and the cost of a potentially dense closure, see Regular Path Indexes. The core query engine stays non-recursive and stateless; the index turns the recursive result into a normal relation before the query begins.

Entity classification with tags

Use metadata::tag with minted GenId tag entities. Give tags human-readable names via metadata::name:

// Mint a tag once
let kind_paper = id_hex!("A1B2C3...");  // or use trible genid for random IDs
change += entity! { &kind_paper @ metadata::name: "paper" };

// Tag an entity
change += entity! { &my_paper @ metadata::tag: &kind_paper };

// Find all papers
find!(paper: Id, pattern!(&catalog, [{ ?paper @ metadata::tag: kind_paper }]))

This is the pattern used by wiki.rs (KIND_VERSION_ID), compass.rs (KIND_GOAL_ID), and files.rs (KIND_FILE, KIND_DIRECTORY, KIND_IMPORT). Tags are entities, not strings — they can carry metadata, participate in queries, and be shared across systems.

Working with blobs

Values larger than 32 bytes live in blobs. entity! carries their bytes in the returned fragment:

// Inline a blob payload — entity!{} stores the bytes in the fragment's
// attachment store and puts the resulting handle in the trible.
change += entity! { &doc @ article::body: "A very long string..." };

// Use the same immutable store snapshot that admitted and materialized the facts.
let body: View<str> = snapshot.get(some_body_handle)?;
println!("{}", body.as_ref());

// Low-level producers may put explicitly when they need the handle first.
let text_handle = store.put("A very long string...".to_owned())?;
change += entity! { &doc @ article::body: text_handle };

When to use blobs vs values:

• If you need to join or filter on the data → value (inline, 32 bytes)
• If it's opaque content you just store and retrieve → blob (Handle)
• Rule of thumb: names and tags are values, content and payloads are blobs

Building entities from optional attributes

When some attributes are always present and others are conditional, build the required part first, then conditionally extend:

let id = ufoid();
let mut change = TribleSet::new();

// Required attributes — always written together
change += entity! { &id @
    metadata::tag: &KIND_REQUEST,
    request::command: command_handle,
    request::created_at: now,
};

// Optional attributes — only if present
if let Some(cwd) = default_cwd {
    change += entity! { &id @ request::cwd: cwd };
}

When querying, use a multi-attribute pattern for the required fields (the query engine proves they exist), and separate queries for optional fields:

// Required: one pattern, no Option<> needed
for (id, command, created_at) in find!(
    (id: Id, cmd: TextHandle, at: Inline<NsTAIInterval>),
    pattern!(&catalog, [{
        ?id @
        metadata::tag: &KIND_REQUEST,
        request::command: ?cmd,
        request::created_at: ?at,
    }])
) {
    // Optional: separate query
    let cwd = find!(
        handle: TextHandle,
        pattern!(&catalog, [{ id @ request::cwd: ?handle }])
    ).next();
}

Schema evolution

Adding a new attribute to existing entities is free — just start writing it. Existing entities that lack the attribute are unaffected; queries that require it simply won't match them. This is the monotonic property at work.

// V1: papers have title and author
change += entity! { &paper @ literature::title: "Dune", literature::author: &herbert };

// V2: add page_count to new papers — old papers still work fine
change += entity! { &new_paper @
    literature::title: "Foundation",
    literature::author: &asimov,
    literature::page_count: 255u64.to_inline(),
};

// Query: papers with page_count (only new ones match)
find!((paper: Id, pages: Inline<U256BE>),
    pattern!(&catalog, [{ ?paper @ literature::page_count: ?pages }]))

// Query: all papers (both old and new match)
find!(paper: Id,
    pattern!(&catalog, [{ ?paper @ literature::title: _?t }]))

Removing an attribute is not directly supported — triblesets are monotonic. Instead, add a new attribute that supersedes the old one, and update your queries to prefer it. The old data remains but stops being used.

Multi-dataset queries

Queries can span multiple TribleSets and even native Rust collections in a single find! call using and!:

let local_facts = TribleSet::new();
let remote_facts = TribleSet::new();

find!((entity: Id, name: String),
    and!(
        pattern!(&local_facts, [{ ?entity @ schema::tag: &KIND_PERSON }]),
        pattern!(&remote_facts, [{ ?entity @ schema::name: ?name }])
    ))

The engine handles the join — it doesn't matter which dataset holds which attributes.

Collection Workflows

TribleSpace publishes data into self-describing grow-only collections. A collection has no mutable head and no privileged linear history: independent signed commits coexist, replicas combine records by set union, and stored merge or derivation equations preserve reusable physical work.

Vocabulary

• Fragment — facts, descriptive metafacts, exported IDs, and referenced blob attachments produced as one composable value.
• BlobStore — immutable content-addressed bytes.
• CollectionStore — a grow-only set of native COMMIT, MERGE, and DERIVE records.
• Collection descriptor — a canonical SimpleArchive which describes a collection's anchor, member encoding, and independent READ and WRITE admission policies. A derived descriptor additionally links one concrete mapping entity carrying its algorithm and concrete parameters. The descriptor's content handle is the CollectionHandle.
• Collection<E> — a cheap descriptor handle whose CollectionEncoding type E owns the canonical member bytes and join. Constructing it validates that the runtime descriptor names E.
• Cover<E> — one typed collection identity plus a PATCH of distinct Handle<E> members selected for one read or derivation. Signatures, authors, and metadata remain queryable provenance, but are not coordinates of the value. Checked union, intersection, difference, and subset operations reject covers from another collection. collection.cover(members) names such a coordinate without store access, which lets a durable manifest preserve an exact cover; it does not by itself admit, evidence, or make those members resident.
• Support — exactly Cover<SimpleArchive>: distinct COMMIT.data handles at the foundational fact collection, admitted directly or through an authorized producer's exact input-record endorsement. It is the denotational coordinate shared by every representation. An equation preserves its witnessed support; it does not endorse every other route to the same blob.
• TryFromCover<E> — the encoding-specific reconstruction hook used by a collection snapshot. A view may join eagerly or retain mmap-backed shards and query their union lazily.
• CollectionSnapshot<R, E> — one immutable store snapshot together with the foundational Support and the resident Cover<E> which realizes exactly that support in this observation. It reconstructs a caller-chosen logical value later with view.
• WANT — an orthogonal local request for content or existing computation; it is neither collection membership nor authority.

MemoryRepo, Pile, and the storage composition wrappers implement both the blob and native collection surfaces. A collection is its descriptor handle; the store remains the sole owner of I/O, durability, and lifetime.

An existing descriptor is opened through the same frozen read boundary used for later observation:

let snapshot = storage.snapshot()?;
let models = Collection::<SimpleArchive>::open(&snapshot, collection_handle)?;

open fetches the canonical archive and queries for a tagged descriptor entity naming SimpleArchive, without requiring either policy. Extra representation facts, retired attributes, or descriptive names do not negate that match. The encoding may require its own interpretation context. This explicit typed-open boundary reports a wrong-type error when no supported encoding interpretation exists. It never registers, rewrites, or otherwise mutates the store.

Raw register_collection::<E> similarly recognizes the encoding context it needs and stores the complete fragment; policy and lineage are queried when a consumer needs them. Descriptor decoding checks the canonical archive bytes, not a closed-world document shape. No descriptor or native-record bytes change when a reader learns these query semantics.

Admission is a positive query

Ordinary admission considers every supported policy interpretation for the requested action. The existing policy binding still names a capability definition handle; a reader queries that definition's invocation-action facts, not equality with a grant's handle. A typed consumer joins the encoding fact and policy link on the same tagged descriptor entity; it cannot borrow a policy from another entity merely because both occur in one archive. Representation-neutral network disclosure queries the tagged descriptor's generic bindings for ACTION_READ. The linked policy must explicitly describe Open or a usable quorum. Unknown kinds, undecodable values, and unsupported thresholds contribute no interpretation.

A subject is admitted when at least one supported alternative authorizes it. Each quorum is evaluated independently: one share from each of two alternatives does not combine into a threshold proof of either. This is union of query results, not another serialized policy kind. Finite READ audiences likewise union; an explicitly recognized open alternative makes the audience Open. No matching READ interpretation means denied disclosure, and no matching WRITE interpretation means no admitted COMMITs. A root with no usable WRITE policy therefore has an empty ordinary collection snapshot. Missing READ policy does not hide otherwise admitted local data: the two actions are independent. Real descriptor/proof-store I/O failures remain errors, as do malformed archive bytes; query invisibility does not excuse failed storage observations.

The explicit collection.policy()/descriptor::policy inspection and root-grant conveniences still require one unambiguous scalar policy. Executable ancestry also remains singular: source, mapping, and argument ambiguity is diagnosed when that route is demanded. General plural lineage is unresolved, not silently selected by hash order or combined into a made-up foundation. Support still belongs to exactly one foundational collection, and native DERIVE(target, input, output, input_witness) still names no separate mapping definition: the target descriptor supplies it. This query-use improvement does not claim to define every possible descriptor interpretation.

Publish a root collection

Register the descriptor once, then pass its returned handle to store operations:

use ed25519_dalek::SigningKey;
use rand::rngs::OsRng;
use triblespace::core::{
    blob::encodings::simplearchive::SimpleArchive,
    collection::{grant_collection_write, AdmissionPolicy, CollectionPolicy},
};
use triblespace::prelude::*;

let team_key = SigningKey::generate(&mut OsRng);
let writer = SigningKey::generate(&mut OsRng);
let team = team_key.verifying_key();
let writer_subject = writer.verifying_key();
let mut storage = MemoryRepo::default();
let models = storage.collection(
    "models",
    CollectionPolicy::new(
        AdmissionPolicy::direct(team),
        AdmissionPolicy::direct(team),
    ),
)?;
let _proof = grant_collection_write(
    &mut storage,
    models.handle(),
    &team_key,
    writer_subject,
)?;

let commit = storage.commit(
    models,
    &writer,
    entity! { metadata::name: "first-model" },
)?;
let snapshot = storage.snapshot()?;
let cover = models.admitted(&snapshot)?;
assert!(cover.contains(Handle::<SimpleArchive>::from_hash(commit.data())));
storage.flush()?;

Local publication deliberately performs no authorization check: the local store is a grow-only record ledger, not an access-control boundary. Observation loads the independent policies from the descriptor. A policy root contributes its own share; every author still needs the policy's distinct-root threshold for ACTION_WRITE on this descriptor. Interpreting the paths requires their resident capability definitions. Invalid or irrelevant candidate evidence grants nothing; inability to enumerate the proof store remains an error. Generic WRITE admission is clock-independent; the passage of time does not expire a grant.

READ and WRITE are explicit because both participate in collection identity. Either may be Open or a canonical quorum over capability roots, with one semantic threshold. Derived collections state their own policies rather than inheriting ambient authority from a source or a network-wide team scope. Whether a subject may delegate a share onward is carried by the independent delegation-action facts in that path's capability definitions, not by a second policy threshold. A child may invoke or delegate only actions its parent permits delegating; invocation alone never confers delegation.

What publication writes

One store.commit(collection, signer, fragment) performs these semantic steps:

1. consume the fragment once into attachments, facts, and metafacts;
2. store the fragment's attachments;
3. encode and store the facts as the canonical SimpleArchive member;
4. encode and store metafacts as the mandatory canonical metadata SimpleArchive;
5. insert a signed COMMIT naming the already typed collection, data, and metadata handles.

Dependencies precede the record which gives them authority. Publication does not flush implicitly. Call flush() at the application's chosen durability boundary or explicitly close the backend. Repeating the same fragment with the same signer produces the same exact signed record and is a set no-op; distinct attestations coexist.

COMMIT is deliberately a source operation over authored Fragment values. Other collection encodings enter the lattice through reproducible DERIVE and MERGE records rather than alternative signed leaf formats.

Importers which must validate additional artifacts before making the source commit visible use the same path with an explicit pause before step 5:

let prepared = PreparedCollectionCommit::from_fragment(candidate);
let mut staged = prepared.stage_for(&mut storage, models, &signer)?;

// Dependencies are resident, but COMMIT is still withheld. Any validation or
// reproducible DERIVE/MERGE publication can use this exact store now.
validate_candidate(staged.store_mut())?;

let commit = staged.finalize()?; // the sole signed COMMIT insertion

Preparation is store-free and dropping either a prepared or staged value never publishes a commit. stage_for accepts Collection<SimpleArchive>, not a raw handle or a reconstructed descriptor fragment.

The native algebra

The collection descriptor is the only collection-control structure represented as a trible archive. The algebra records are fixed-width native records:

COMMIT(collection, data, metadata, author, signature)                  // 192 bytes
MERGE(collection, low, high, result, low_witness, high_witness,
      author, signature)                                             // 288 bytes
DERIVE(target, input, output, input_witness, author, signature)         // 224 bytes

COMMIT is a signed exogenous assertion: no machine can recompute whether an author intended to publish a member. MERGE is an exact join equation within one collection. DERIVE is one observation of the mapping linked by its target descriptor; that descriptor already names its source, mapping algorithm, and concrete mapping parameters.

Each witness is the full fingerprint of an actual native input record, not a blob handle or synthetic entity. MERGE keeps (payload, witness) pairs together in canonical order; DERIVE's witness belongs to its descriptor's exact source collection. The referenced record must produce the signed input payload.

The exact native record is the set element. CollectionStore::insert therefore implements set insertion rather than an update. Concatenating stores unions evidence. Collection records have no synthetic entity identity; their full-width fingerprints identify the exact persisted records for lookup, deduplication, and signed witness references. Support still contains payload handles, never record fingerprints.

Signed equations endorse both materialized computation and validation of the exact input records' support. They do not create new exogenous membership. Publishing a MERGE or DERIVE records work which has already been performed; warm resolution follows that endorsement without executing the join or mapping again. Foreign bytes are signature-checked at the store/synchronization boundary; the producer must satisfy target WRITE through the observation's resident proof and capability-definition evidence. A trusted local store is not reverified on every read. Once a target producer is admitted, attachment selects its resident output without expanding the historical support. Exact record relationships order the physical cover; only selected outputs and their encoding-required dependencies must be resident. A missing result cannot suppress an available finer realization. An explicit support() query follows the exact witness DAG and descriptor lineage without re-admitting ancestors or reading ancestral payloads, metadata, and proofs. A missing or mismatched witness route makes that query fail, not return empty support, and does not prevent reading the endorsed target value.

This deliberately trusts authorized producers to validate inputs and compute correctly. Signatures identify who endorsed that claim; they do not prevent an authorized dishonest producer from lying. Read-time recomputation is not part of the contract.

Low-level signing and store insertion remain unconditional. A publisher which needs to predict whether an authority-aware observation will admit a signer can freeze a store snapshot and call collection.writer_is_admitted(&snapshot, signer): it checks the descriptor WRITE policy and resident exact authorization evidence without scanning collection commits or publishing anything.

Known-prefix snapshots and covers

store.snapshot() freezes one immutable observation containing a resident blob index, collection records, and capability proofs from the same known prefix, together with one interpretation instant for application queries. The snapshot, rather than a source frontier or a later materialization, is the watermark. Ask it what representation is actually readable at that instant:

let snapshot = store.snapshot()?;
let observed = snapshot.collection(collection)?;
let cover = observed.cover();
let value: V = observed.view()?;
// Only a consumer which needs provenance asks for historical support.
let support = observed.support()?;

snapshot.collection(target) admits producers of target records from the snapshot's proof and resident definition evidence and chooses their resident target cover. Neither upstream descriptor residency nor a complete historical witness closure is a prerequisite for this read. The observation retains exact endorsements for its lazy support() query, not a fresh admission query over every source COMMIT. Admitted but not yet derived data is absent: an immutable snapshot never promises work which will happen later. There is no assertion form which takes a requested support: what a target stands on is read back through support(), and a caller who needs two views to agree attaches both from one snapshot and compares their supports. Neither passive observation method reads the clock: identical operations on one frozen store snapshot have identical results even while wall time passes. A passive decision using later-arriving proof or definition evidence requires a new snapshot. Storage snapshots have no timestamp or historical-time selector. changes_since classifies content only. Generic collection authorization has no proof-validity clock; action-specific deadlines belong to the application interpreting that action, with an explicit evaluation time separate from its immutable evidence snapshot.

The lazy coverage memo is shared between consecutive store observations only when changes_since reports NONE. Reading an older operation control can therefore warm a newer, otherwise identical residency observation as well. This shares computed work, not mutable records or storage: each snapshot keeps its own underlying observation and held-blob view. Any reported change forks the memo, including a new blob occurrence or capability proof, so newer membership, authority and residency cannot leak into an older read. Backends whose conservative classification reports ALL keep the separate-memo path.

observed.is_current(&later_snapshot) compares the raw dependencies actually consulted by attachment, views, and explicit support queries. Indexed backends ignore unrelated collection and blob appends. Missing lookups count too: a later matching record or a recoverable physical occurrence of a blob triggers a refresh. This is conservative dependency comparison, not a global sequence number or another materialized catalog. Query deadlines remain caller-owned.

Both forms keep the chosen target cover inseparable from the store snapshot which established its residency. view invokes TryFromCover<E> solely through that frozen observation. For a SimpleArchive, V = TribleSet; for a SuccinctArchiveBlob, V may be an mmap-backed union retaining selected shards. collection.read::<V, _>(&snapshot) remains a concise resident collection read when the intermediate support and physical cover are irrelevant.

Foreground exact-byte acquisition

An operation that needs selected bytes can wrap one immutable peer snapshot in repo::async_store::AcquiringReader. Its synchronous get drives that snapshot's asynchronous exact-handle get; the store owner and runtime must stay alive for the operation. Run synchronous reads outside a Tokio task (an async operation can use spawn_blocking). This does not make resident inventory, records, proofs or change detection live.

Before its first selection, the acquiring reader can retry parked records from the original index using the same frozen proof evidence and acquired descriptor or capability-definition bytes. It cannot discover a later grant or a later collection record. reader_is_admitted_acquiring and writer_is_admitted_acquiring likewise acquire definitions against frozen proofs; unavailable definitions grant no authority, while actual backend or decoding faults are returned rather than hidden as denial.

Use snapshot.collection_acquiring(target) when an operation needs all known target foundations, including those not represented by the resident cover. Its view reads that fixed selection and fails if a selected member cannot be read. Derived foundations are still the known admitted images, not a promise to derive missing images from upstream data.

For an attached collection, first select once with attached_acquiring through the acquiring reader. This exact-gets its target and parent descriptors instead of requiring their residency in the frozen index. Then call read_acquiring, or succinctarchive_union::read_attached_acquiring for fact shards. These read the fixed attached cover and demand its exact residual foundations without a frozen-residency precheck. They do not maintain, publish, or replace a selected view after obtaining bytes. The result carries unread support for unavailable bytes/dependencies or an unrepresentable mapping. A complete operational value requires checking unread; taking only into_value is an explicit partial-read choice. This says nothing about peer records which have not yet arrived or bulk synchronization completion.

AttachedSnapshot<AcquiringReader<R>, E>::into_frozen removes only the adapter, retaining its exact original reader, chosen cover/support/residual and read-set. It allows an async owner to retain selected coordinates and later acquire their bytes without selecting again. It does not replace the underlying snapshot's resident index with the operation's acquired interpretation.

For maintained indexes, attachment checks safe typed framing and retains the stored sections; it does not rerun canonical construction or semantic audits. An interpretation can expose a separate fallible query() preparation step when its wire format omits data needed for querying, such as sparse-summary search positions or BM25 document lengths. Keep that prepared value for the queries which share it. Requesting a different representation can still be real work: TribleSet interpretation constructs PATCH indexes from raw facts, whereas attaching a persisted Succinct index uses its existing index sections.

Physical compaction is support-aware. If distinct source members a and b map to the same payload x, an endorsement of z through b and c does not also endorse a, even when payload order says x <= z. A resident x must remain alongside z to represent {a,b,c}. With only z resident, the snapshot reports {b,c}; it does not invent the missing support. The exact-record route, not an inverse walk over all equations sharing payload handles, decides this.

cover.commits(&snapshot) queries available native provenance over the selected payloads. These attestations are not necessarily authorized membership claims: another signer can attest an already admitted payload without changing its support or becoming an authority root.

This is a coherent known-prefix observation, not a global latest transaction. A concurrent immutable insert may appear in this call or a later one. A mutating ensure or maintain operation returns a new store snapshot; the caller may then ask that snapshot for the collection it actually contains.

Raw record readers still expose dangling native collection records and stored proof records for repair. A member is readable only when its selected payload and representation dependencies are resident in that exact frozen snapshot. An endorsed result does not need its historical input payloads or COMMIT metadata to remain resident. Its exact record witnesses must still close when support is requested; merely holding the result blob does not establish that provenance. A capability proof's signatures can be checked from its bytes alone; action and delegation interpretation also needs the referenced definition blobs. A missing definition does not erase the raw proof, but it cannot support an interpreted grant. These passive observations never acquire, wait, write, or emit WANT.

Exact immutable payload reads are a separate capability of network-backed snapshots:

let snapshot = peer.snapshot()?;
let observed = snapshot.collection(collection)?;
// A queried attachment handle is sufficient; no collection authorization
// argument or mutable writer borrow is needed for this exact byte read.
let bytes: Bytes = snapshot.get(attachment_handle).await?;

PeerSnapshot::get and ObjectStoreSnapshot's asynchronous get can fetch bytes which arrived after the snapshot. That cache operation changes neither the captured records/proofs, the interpretation instant, the frozen residency index, nor observed.cover(). Passive collection selection still uses the captured resident prefix; reading a newly acquired collection member requires another store snapshot. A synchronous consumer can use one Blocking adapter at its outer runtime boundary to decode payloads without threading a mutable writer through its domain queries. It must not nest that boundary inside an already running async task. Closing a peer ends live acquisition, while bytes already captured in its snapshots remain readable.

Foreground clients can own Leech<S> instead of Peer<S>. A leech implements the same local store and exact-acquisition traits and returns the same PeerSnapshot<S>, but neither a snapshot nor a later acquisition refreshes a serving inventory. It cannot activate collection replication or advertise its resident handles. Local puts, commits, proof insertion and close remain available; not serving the pile does not mean a read-only local store. Its lazy host can still participate in discovery and DHT routing, through its pile's sync daemon (PeerConfig::daemon) when its pile names no peer. H remains the read capability, the serving L-to-H map remains private to serving peers, and the provider-first endpoint-bound bearer exchange is unchanged. An ordinary Peer with a zero publication budget or ReadOnly reconciliation still serves its inventory; those settings are not aliases for Leech.

Native applications can call Peer::shutdown_and_join or Leech::shutdown_and_join after cancelling their outstanding acquisition futures. Call this blocking boundary off UI and runtime threads. It withdraws serving, prevents retained readers from starting a new host, revokes their live acquisition capability and joins the host after transport shutdown. A dormant leech remains dormant. Already captured local snapshot bytes stay readable; the application still flushes or closes its backend separately.

The trible::maintenance::Driver shares the CLI's maintenance implementation without opening storage, creating a runtime or spawning workers. The caller selects native Targets, schedules tick, and receives actual operation-boundary Events. An unchanged observation emits no work events. Dropping an in-flight tick cancels the owned future, and the next tick retries incomplete work; synchronous encoding calls can only yield at their existing boundaries. Drop the driver before closing storage to release its retained immutable snapshots.

Record retention is a separate lifetime rule: a retained non-blob record strongly retains every directly referenced blob which is resident, but does not fetch an absent one; proofs reference stable capability definitions, not the opaque resource. A WANT is itself only an explicit durable demand record, never automatic cache-miss bookkeeping.

The live store operations, ensure and maintain and their _with forms, are asynchronous even for local stores. They may fetch exact missing blobs named by frozen records or immutable dependencies needed for the work, publish target collection equations, and return a fresh snapshot; they never emit WANT. Local stores implement the same contract with immediately ready acquisition from their resident snapshot, while a networked store may await exact-H fetch.

Raw Cover value algebra is deliberately a lower-level operation than attaching a CollectionSnapshot. The caller already names exact descriptor and payload identities. Replaying those bytes needs no publishing key or foundational COMMIT/metadata records; algebraic alternatives may reuse authorized signed MERGE math without establishing authenticated foundational support. Such a resolved cover is not evidence that its members were admitted to the collection. Only the collection-snapshot path above claims Support and therefore requires closed exact witness records.

Use cover.commits(&snapshot) when currently resident authorship and metadata provenance matters; zero commits is a valid answer and does not invalidate replay. Several admitted commits over the same payload are distinct provenance fibers but one member of Support.

Reuse merge work without changing meaning

A logical collection value is the join of a cover's members. It does not need one monolithic blob. A resolver may choose members consisting of committed payloads and stored merge results:

    a       b       c           explicit payloads
     \     /        |
      a⊔b           |           reusable MERGE result
        \           /
         (a⊔b)⊔c                logical collection value

Distinct covers can have the same support: {a, b} and {a⊔b} are different PATCH sets, but the stored MERGE equation records that they denote the same join. This is useful for LSM-like maintenance: small commits remain independently attributable, while deterministic merges amortize reads into larger canonical shards. A selected target cover is replaceable computation, never a second history or a new authority root.

Derive another representation

A mapping that not every reader of a node can compute -- a model whose output is canonical on one class of accelerator, an embedding that does not reproduce bit for bit -- is derived: its images are published as signed DERIVE leaves, DERIVE(target, L(F), f(F)) for a source foundation F with locator L(F), and replicate with the collection. Its target encoding implements CollectionDerivation, naming one canonical Source encoding and a runtime Argument carried by the concrete mapping descriptor.

If a downstream crate owns neither the source nor target encoding, Rust's orphan rule prevents that target-owned implementation. It can instead provide an explicit DeriveMapping and select the same engine through derive_with, ensure_with, maintain_with, and rederive_with.

A derived collection is a set of foundations, like a root: its leaves. It carries its own lattice exactly as a root carries its commits -- the host joins held leaf images, eight at a time per support tier, with the mapping's join (DeriveMapping::join_images, by default the encoding's) into MERGEs of its own -- and never reads its source's merges. A mapping maps foundations only, and its target encoding's join is total, so every set of leaves has a join whoever derived them. A deterministic mapping is a join homomorphism,

f(a ⊔ b) = f(a) ⊔ f(b)

so the join of a collection's leaves is the image of everything its source's foundations hold. A non-deterministic one may give one foundation several leaves, and their join keeps what each derivation observed: the DERIVE relation is not a function.

Derivation is scheduled by leaf: any admitted leaf suffices. A source foundation is derived only when no believed leaf for its locator has an output that is here or can be fetched, whoever signed that leaf; any key the target admits may derive it, not only the foundation's owner. A leaf whose output cannot be obtained does not count, and a fetch that fails means the output is unavailable now, not lost: if the original output arrives after the foundation was derived again, both leaves stand, both are joined, and nothing further is derived. A fetch that fails outright -- the store cannot reach other holders, or cannot keep what it got -- is a fault here rather than an answer about any holder, whether the blob was needed to map a foundation or named as a leaf's output: nothing is derived in its place, and the fault is reported once the rest is done. Outputs that are not here are asked for after the rest of the pass -- deriving what needs no fetch, and the carry -- each once. The outputs of one foundation's leaves are asked for until one arrives or all have failed, and not at all once one of them arrived for another foundation whose leaf names the same output. A pass starts no further foundation once eight fetches have failed. Each pass starts the foundations in an order of its own, drawn afresh, so foundations whose fetches always fail cannot keep the same others waiting pass after pass. A leaf counts only when its output reads: damaged bytes a pile still lists are treated as an absent output, and a network peer fetches good bytes for them. The carry joins only nodes whose bytes read, a root's carry too, and says nothing about the rest; a reader's attach validates no byte, so a read that takes a damaged node fails naming it. Only a store that can reach other holders asks at all: a plain pile answers from its own blobs, so a miss there says nothing about elsewhere, and the foundation waits for its output instead of being derived again. Each key derives in an order of its own, and a foundation whose leaf another writer publishes while the pass runs is left to that leaf, so two hosts rebuilding one collection at once do not both derive all of it. A mapping pinned to a class of host (DeriveMapping::computable_here) derives nothing on another host and raises nothing there; that host still carries the collection, whose leaves arrive by replication. A leaf known to be bad is supplemented only on request: rederive_with maps the named foundations again and adds each differing result as a second leaf. Nothing replaces a leaf.

Newly executed joins and mappings publish every successful result and equation, even when a later planning or storage step fails. Publication is operation-ordered rather than phase-batched, so a failure leaves the complete successful prefix addressable instead of stranding its blobs without their equations. Canonical joins, mappings, and logical cover views receive one frozen store snapshot and may resolve immutable dependencies named by their inputs; unrelated resident blobs are never ambient semantic input.

Attached collections

An index that only restates its root in another representation has no lattice of its own to keep. It is attached: its descriptor is the root's handle and the mapping, with no policy, and a MAP record signed by the store's host names, for one node of the root's lattice, the attachment built from that node's own bytes:

SimpleArchive node --MAP--> SuccinctArchiveBlob
                   --MAP--> Rank9AcceleratedSuccinctArchiveBlob

An attached collection has no carry, no COMMIT, DERIVE or MERGE of its own, and no policy: every host builds its own attachments, and a store believes only the MAPs its own host signed, so another host's that reach it stay inert. A read's residual -- foundations no usable attachment reaches -- is not missing data: AttachedSnapshot::read builds their images in memory for the view, and names what it cannot build in AttachedRead::unread. Maintenance carries the root first and then attaches the frontier the carry leaves, so a merged node's attachment is mapped from the merged node's bytes and nothing is built for a node the carry is about to consume. A mapping that reads another attached collection names it in its descriptor: Rank9 is built from the same node's Succinct attachment.

use triblespace::core::collection::{CollectionSnapshotExt, CollectionStoreExt};
use triblespace::core::blob::encodings::succinctarchive::{
    OrderedUniverse, Rank9AcceleratedSuccinctArchiveBlob, SuccinctArchiveBlob,
    UnionArchive,
};

// The store is opened as the host: only its MAPs and MERGEs are believed.
let source = storage.collection("models", source_policy)?;
let raw = storage.attach::<SuccinctArchiveBlob>(source, ())?;
let accelerated = storage.attach::<Rank9AcceleratedSuccinctArchiveBlob>(source, raw)?;

storage.maintain_attached(raw, &host).await?;
let after = storage.maintain_attached(accelerated, &host).await?;

let observed = after.attached(accelerated)?;
let facts: UnionArchive<OrderedUniverse> = observed.view()?;
let unattached = observed.residual();

A read walks the root's frontier widest first and takes a node's attachment when it is usable: a MAP the host signed whose bytes, and the bytes of every attachment it depends on, are here. Otherwise it descends through the host's merges to the nodes that produced it. A foundation no usable attachment reaches is the read's residual: read it from its own bytes (succinctarchive_union::read_attached does this for Succinct and Rank9), build its attachment, or report it. Attachments combine idempotently, so a cover of attachments answers every query as the attachment of the union of its nodes would, for every mapping this crate defines, each built from single facts. A mapping whose image aggregates facts of several entities (a document's term frequency summed over its parts, say) answers so only for covers that never split such an entity; that is a premise about its input, nothing refuses a node that breaks it, and its documentation states it.

For a command-line maintenance owner, trible pile collection maintain-all discovers this source order from explicitly selected descriptor handles and calls those same operations. Shared upstream dependencies run once per pass; a foundational dependency is ensured rather than needlessly compacted. The one-edge maintain command remains available. Either command accepts --watch to retain one open pile and repeat after changes to the previous active pass's raw read-set, including source records, exact acquired blobs, new proofs and resident definitions. repo::ObservedStore transparently tracks those reads across fresh snapshots and acquisitions, including misses and errors, without retaining interpreted values. Per-collection counts remain scoped interests; name-based selection really enumerates the record inventory and is therefore broader than selecting exact handles. Proof changes remain component-wide. Relevant changes during a pass retain one bounded catch-up pass after the post-work baseline; a quiescent no-op pass clears it. Neither WANT-only nor unrelated writes sustain a watch loop. Generic backends without indexed changes_for remain component-conservative. External provider availability is not stored evidence: network callers still own retries and deadlines. There is no ambient AUTH-expiry timer. The CLI registers SIGINT and SIGTERM handlers on Unix before opening its writable pile (Ctrl-C on other platforms). A stop drops the awaited maintenance pass or idle wait and reaches the explicit pile close path. Synchronous store operations and CPU sections remain cooperative; shutdown adds neither a preemption guarantee nor a periodic flush. This adds scheduling, not recursive effects inside a mapping kernel. Exact target selection prevents an old, superseded index from being restarted merely because its descriptor is still present.

The signing author's public key deterministically biases the order of the explicitly selected targets, independently of argument order or duplicate selections. Independent authors therefore tend to attempt different chains first; each chain still runs upstream first, and shared dependencies run once per pass. This is priority rather than exclusive ownership: duplicate work is still possible, no distributed lock is introduced, and each collection's canonical merge/derive plan remains unchanged.

trible pile collection maintain-all self.pile blake3:<RANK9_TARGET> --watch

The executable still needs the encoding and concrete mapping implementation; descriptors are data, not dynamically loaded code. This local maintainer and a network sync process can run independently over the same append-only pile. Readers attach the resident target from their own snapshot and do not wait for a global maintenance frontier.

When a caller needs matching representations across a chain, maintain each edge in order — maintain(raw, &writer), then maintain(accelerated, &writer) — and attach the targets from the snapshot the last step returned. Each target then stands for what its source's frontier stands on, so two targets of one source read from one snapshot agree by construction; there is no support argument to pass along, and none to request narrower.

• ensure(source, &writer) freezes collection records, capability proofs, and resident authorization evidence before acquiring exact missing descriptor, selected data, and representation-dependency bytes needed for that root frontier. Historical metadata is not a materialization dependency. Concurrent records and proofs do not extend its work. The returned snapshot is a fresh observation; attach from it.
• snapshot.collection remains the purely read-only alternative: it performs no acquisition or collection algebra and binds only the resident target cover visible in that immutable snapshot. What that cover stands on is its support().
• For a derived target, ensure publishes a leaf for each foundation of its immediate source the key owns that has no leaf yet, and returns a fresh store snapshot. Missing source members are invisible to selection: an immutable snapshot never promises work which will happen later.
• maintain derives every foundation this host can derive that has no usable leaf -- the key's own, fetching their payloads, and another owner's whose payload is already here -- then carries the target's own leaves the way a root carries its commits. A foundation the mapping refuses holds back neither the rest, nor the carry, nor the fetches the rest is waiting for; it is reported afterwards, after a failure of the carry itself. It also returns a fresh store snapshot.

A derived operation never constructs an upstream member as a side effect, and the carry reads nothing outside the target's own lattice. If a target join cannot run because an optional immutable dependency is absent or the encoding has reached a capacity limit, the finer exact target cover remains the answer.

The signing key is explicit on every ensure/maintain call. Only a new leaf needs WRITE: a key the target does not admit derives nothing and fetches and maps nothing for it. Its maintain carries -- a merge needs only the store's host key -- and reports nothing more, without reading the source; its ensure, the write path, receives UnauthorizedProducer if it owes a leaf for a foundation of its own. Raw local sign and insert remain unconditional; typed publication does not reverify signatures it just produced.

New work authenticates the immediate source and target producers it actually uses. It does not recursively acquire foundational data or re-run ancestral authorization merely because the immediate input was itself derived. A mapping remains one hop; maintain-all is the explicit scheduler for upstream work.

WANT missing content

Sparse evidence discovery deliberately does not fetch commit dependencies. WantStore adds operational interest to one idempotent grow-only set with a single request shape, Blob(handle): obtain those exact bytes.

A reconciler may satisfy it from local workers or discover providers under opaque KDF(H), without activating or even naming a collection. The provider proves H first and the requester second, with both proofs bound to the authenticated endpoints; H itself is never sent, and landed bytes must hash to H. A WANT grants no collection authority and does not change the value of any collection. There is no unwant operation: cache eviction belongs to Yard's physical rewrite policy, which re-records only surviving blob demand.

There is deliberately no request for computation. Asking another node to produce a merge or derivation result is asking it to spend work on the asker's behalf, and a flood of such requests costs the asker nothing; a blob request costs the asker the bytes it receives, so a flood floods itself. A node that wants a computation done runs it and then shares the equation.

Migrate a legacy branch explicitly

Old piles may contain signed commit DAGs and mutable pin records. Current readers retain an immutable PinSnapshot and legacy decoders so operators can inspect and migrate that evidence without restoring the old publication API.

trible pile migrate data.pile branch-to-collection \
  --branch legacy-events \
  --collection-name events \
  --signing-key ./writer.key

The command is deliberately same-pile: source commit blobs must already be resident. It freezes the selected legacy head, validates the complete reachable DAG, and converts each authored node into a native commit using its exact repo::content and metadata::archive handles. A missing metadata archive maps to the canonical empty archive. Contentless merge wrappers are validated but do not become members.

With no further options the target descriptor gives the migration signing key one-root direct READ and WRITE policies. The resulting commits are therefore admitted directly by ordinary collection admission against a store snapshot.

The migration-only --authority option instead uses another trust root for both direct policies:

trible pile migrate data.pile branch-to-collection \
  --branch legacy-events \
  --collection-name events \
  --authority <64-hex-character-ed25519-public-key> \
  --signing-key ./writer.key

Local publication remains unconditional, so this form still writes commits signed by the migration key. A later read admits them only when the store holds enough exact root-to-signer ACTION_WRITE evidence for the resulting descriptor handle. The migration command does not invent, scan for, or store that delegation.

The complete source DAG and every prepared target element are validated before the target descriptor, dependency, or commit is published. Storage failures remain backend errors; authorization is deliberately deferred to reads rather than treated as permission to append locally.

Legacy wrapper parents, messages, timestamps, authors, and signatures are not silently reinterpreted as application metadata. Two source nodes with identical data and semantic metadata map to one intrinsic native commit. Re-running with the same collection identity and key is idempotent.

Migration is the only reason application-facing tooling needs to name a legacy branch. New code publishes directly to collections.

Operational invariants

• Persist dependencies before the record that makes them meaningful.
• Treat a cover's payload identities as semantic ground truth. Availability is expressed as a subset in those coordinates; physical decomposition stays private to same-snapshot materialization. Signed commits and metadata remain lazy provenance queried separately.
• Treat stored signed equations as reusable materialized LSM work. Never replay algebra merely to trust a local equation; apply target WRITE and follow its exact endorsed record route instead. Payload aliases must not enlarge that route's support.
• Persist every successful join or mapping. Yard/GC policy alone decides when its result bytes leave local storage.
• Keep admission, retention, and WANT policy orthogonal.
• Carry exact covers across derivation boundaries instead of asking for an ambient “latest”.
• Flush at explicit application durability boundaries.
• Merge stores by union; never choose meaning from append order.

These rules are sufficient for both low-latency single-process use and sparse distributed collection maintenance without introducing a second execution model.

Garbage Collection and Forgetting

Stores grow as signed commits, derived artifacts, authorization evidence, and user blobs accumulate. Because every blob is content addressed and immutable, nothing is overwritten in place. A node periodically forgets local bytes which no retained record or explicit policy root owns.

Forgetting is deliberately conservative. It removes only a local copy, so later synchronization or an explicit fetch may restore it. An assertion is not semantically retracted merely because one node drops physical bytes.

Structural Ownership

Lifetime is a property of stored structure, not semantic trust. Every retained current native record other than a BLOB strongly owns each blob handle it names directly:

Each direct reference is an independent recursive ownership edge. If the referenced blob is resident, collection retains it and every resident child found by the ordinary conservative blob walk. If it is absent, collection does not fetch it, manufacture a subordinate WANT, fail, or withhold any resident sibling. A blob record does not own itself merely because its bytes occur in a store; it survives only when some retained record, legacy strong pin, or explicit policy root reaches it.

This rule deliberately does not verify a commit or proof signature, perform collection admission, or decide whether an equation is algebraically useful. Those are semantic questions for a reader. Physical preservation must remain stable when later evidence changes what a record means, and it must not let an authorization bug silently destroy bytes. Conversely, retaining bytes grants no authority. A retained proof owns its resident capability-definition closure, but its opaque resource bytes do not become a lifetime edge even when an application interprets them as a collection descriptor handle.

Witness-bound MERGE and DERIVE also name actual native input records through record_references(). Those fingerprints are record identities, not blob handles: do not feed them into the conservative blob walker or the DHT. The retained native ledger preserves the referenced records independently of whether their payloads are resident. Exact witness closure remains necessary to recover Support; eviction must not silently replace a missing record with another record producing the same payload.

The typed reference enumeration lives on the records themselves. MemoryRepo, Pile rewrites, and Yard collection all consume the same enumeration rather than maintaining backend-specific interpretations of record fields.

Explicit Policy Roots

RetentionRoots supplements structural record ownership for callers with local policy. It distinguishes:

• a direct root, which retains exactly the named blob; and
• a recursive root, which retains the blob and every resident descendant found by conservative traversal.

Explicit roots may describe application policy which is not represented by a native record. Unlike an absent structural reference, an explicitly requested blob missing during a retained Pile rewrite remains a transfer error: the caller asked for that exact copy and silently weakening the request would be incorrect.

There is no collection-specific retention planner. In particular, admission results do not select lifetime roots and MERGE or DERIVE records are not treated as weaker cache hints. A retained MERGE owns its resident descriptor, result, and inputs, and a retained DERIVE its resident target descriptor and output, exactly as a retained commit owns its resident descriptor, data, and metadata. A DERIVE's source locator is a one-way image of a handle, not a handle, so it owns nothing. Naming the descriptor matters most for a derived collection: it holds no COMMIT, so its MERGEs and DERIVEs are the only records that keep the descriptor every one of its records needs to be decided.

This ownership rule is deliberately stronger than read-time dependency selection. An authorized witness-bound output can be read without historical input payloads or metadata; nevertheless, if those blobs are already resident, the retained equation or COMMIT still owns them. Cheaper attachment is not an implicit permission to discard historical bytes or records.

Known retired unsigned equations remain inert in semantic readers, but their direct references still retain resident blobs. Reframing preserves their exact bytes. Unknown opaque kinds cannot safely define a smaller live set because their ownership edges are unknown. A retained Pile rewrite instead preserves their bytes and every resident blob; semantic reframe and Yard reclamation still refuse them.

The retired equation kinds -- the unsigned forms and the later witness-bound signed forms -- are opaque to ordinary replay. Retained Pile copying carries their frames and all resident blobs under the same opaque rule; Yard and semantic reframe refuse them. No shipped command restates them any more: the migration that once re-signed them under a current writer's key was applied to the live piles and then removed with the record kinds it produced. Ordinary GC never deletes the historical frames.

Backend Boundaries

MemoryRepo::keep, Pile::rewrite_retained_into, and Yard collection apply the same law:

1. freeze one coherent resident view;
2. enumerate every retained native record;
3. add each independently resident direct reference as a recursive root;
4. add caller-selected direct and recursive policy roots;
5. recursively mark resident candidate handles; and
6. rewrite or evict everything outside that conservative live set while preserving the retained record ledger.

Pile rewriting exposes an explicit choice to preserve or drop WANT records. Preserving a WANT preserves its ownership edges; dropping it omits the record and therefore those edges. Yard's grow-only WANT set has no recency or byte budget: while a WANT record is retained, its resident references are ordinary strong roots. Physical removal belongs at an explicit rewrite/eviction boundary rather than to a hidden in-memory weakening of the record.

For migration safety, a retained Pile rewrite also recreates the exact immutable legacy strong-pin snapshot it observed. A resident pin head remains a recursive root; a dangling pin remains dangling. Legacy collection headers whose old identities cannot express the current algebra are inert evidence and own no current blobs.

Opaque records form a harder boundary. Their span may be known while their ownership semantics are not. Pile::rewrite_retained_into carries each byte-distinct opaque frame unchanged and widens retention to every blob in its observed resident snapshot, even when the caller selected fewer roots. This allows trible pile compact to remove redundant copies without interpreting or discarding unknown ownership. Repeating a copy does not append an opaque frame whose BLAKE3 digest is already present; this is set preservation, not preservation of physical offsets or duplicate occurrences.

Yard collection, compaction, and reclaim still refuse before changing physical state when an opaque record is present. They cannot infer a safe eviction set from unknown ownership. Semantic reframe likewise retains its unknown-kind preflight refusal; it is a different operation from retained Pile copying.

A retained rewrite can also leave collection-algebra frames behind. PileFile::rewrite_retained_into_filtered asks a CollectionFrameFilter about every COMMIT, MERGE and DERIVE frame it would carry, in every encoding the pile still recognises, before drained generations and supersession apply. A frame a concatenated pile holds twice is considered once. Its retention follows what it carries: a frame left behind roots nothing, a carried legacy V3 header roots the blob handles its fields name, and a carried frame of unknown kind or retired capability proof roots the resident blobs any 32 bytes of it name, at any offset (frames have not always kept their fields on 32-byte boundaries), rather than widening to every resident blob. Without explicit roots it therefore keeps exactly what the carried records, headers, proofs, WANTs and pins reach. A blob the walk reaches of which no occurrence matches its hash has no valid bytes to copy. The walk finds one wherever it meets it, and by default the rewrite then refuses before writing anything, as an invalid explicitly selected blob fails the plain rewrite. A caller may set such blobs aside instead: they are listed, not copied, and their damaged bytes are still read for what they name, but a valid blob named only by a damaged word cannot be reached and is left behind. PileFile::plan_retained_rewrite runs the same selection pass and retention walk without writing, so a dry run reports exactly what the rewrite will do. PileFile::prepare_retained_rewrite retains that decided selection and its immutable source snapshot for a subsequent consume-once write_into, without evaluating a stateful filter or walking the blob closure again. Later source appends are outside that prefix. Whole-file tools must still quiesce writers and check source length; this is a frozen observation, not a reusable recipe. The lattice v3 cutover (trible pile migrate SRC lattice-v3) is this rewrite with only the roots its operator names: it drops every MERGE, since each host folds and rebuilds only its own, the DERIVEs of mappings that became attached indexes or were deleted, and every blob only those reached, including the descriptor of a collection no kept record names unless a root keeps it.

trible pile compact SRC --into DST --drop-collection NAME_OR_FULL_HANDLE is another explicit filtered rewrite. Repeated selections leave the named collections' frames behind and close over resident descriptor sources, attached parents and sibling-index dependencies. Missing or ambiguous names refuse before creating the destination. Full handles select exact identities referenced by collection records or present as resident tagged descriptors; absent handles refuse as well. A descriptor needs no name or admitted member. Unreadable descriptors remain conservative: the command does not invent their ancestry. Selected mode opts into reachability GC, so raw blobs with no carried record, proof, WANT, pin or opaque-frame reference are not promised retention. Shared payloads and metadata reached by retained collections remain. The native planner runs before destination creation and reached corrupt blobs refuse. Without this option, compact stays conservative and roots every resident blob. --drop-drained still applies its existing validation when combined. Neither mode retracts distributed assertions or revokes authority; normal synchronization can restore excluded records to a later replica. Retained proof evidence can still name an excluded collection.

Conservative Reachability

Canonical archives contain fixed 64-byte tribles whose value half is one 32-byte inline value. The generic walker treats aligned 32-byte chunks as candidate handles and follows those which name resident blobs. This may keep an accidental extra blob, but it cannot plausibly discard a real reference. Opaque attachments normally behave as leaves because their chunks do not name another resident object.

This asymmetry is the safety rule: uncertainty keeps resident bytes. Missing bytes remain missing without turning collection into acquisition, and no semantic rejection is allowed to erase structural ownership.

Operational Tips

• Schedule forgetting after meaningful imports or merges so one walk amortizes many writes.
• Budget headroom for synchronization, which may restore locally forgotten copies.
• When application policy is uncertain, add an explicit recursive root; conservative extra retention costs space, while an omitted true root costs recoverability.

Distributed Sync

triblespace-net synchronizes collections rather than exposing one ambient store inventory. Its protocol follows the same decomposition as the collection model:

1. one QUIC connection per peer pair carries typed streams: one long-lived recon/1 stream for the grant exchange and for every collection's peering, a walk/1 stream for each push of a collection's tree, and short dht/1 and blob/1 request streams;
2. for each collection both sides select, each peered neighbour pushes the other the collection's trees, its foundation records and scoped AUTH (and, between two Full neighbours, its held blobs), as Merkle walks pruned by what the receiver already holds; the receiver pulls the values it lacks and lands them as they arrive; and
3. exact blob handles fetch only the immutable bytes a resolver actually chooses through a collection-independent, mutually authenticated bearer protocol.

No global team, mutable roster, gossip topic, durable OFFER/GOSSIP bit, or second replicated inventory is needed, and no list of peers. Which collections a host syncs is a register in its own pile, and whom it syncs them with comes from the same pile: the keys its records and grants name. Held sets are local observations, not durable claims merged into a collection. The descriptor already states independent READ and WRITE policy, and iroh authenticates the endpoint key on each direct connection.

Four independent capabilities

The boundaries are deliberately small:

know C           -> find holders of C's descriptor and ask them to peer for C
prove READ(C)    -> be pushed C's records, scoped proofs and held set
know H           -> derive its opaque locator, discover providers, and fetch H
satisfy WRITE(C) -> make a signed COMMIT or DERIVE active in C, and send C to a reader

C is the exact 32-byte collection descriptor handle. H is an exact blob handle. Knowing C permits discovery of its ordinary descriptor-blob providers (H=C). They are peering candidates, not certified collection participants, and a peering request tells them only that the asker selects C and which credentials it holds for C. H is the bearer capability and private discovery secret for one exact immutable value. The provider directory sees only blob-locator KDF images and endpoint-bound tokens, never raw handles or a collection-participant namespace.

READ and WRITE are independent AdmissionPolicy values embedded in the descriptor. Each is either Open or a canonical quorum over Ed25519 roots with one semantic threshold. A derived collection chooses its own policies. Source ancestry, routing knowledge, and possession of a blob do not silently supply collection authority. Downstream delegation is a restriction signed into each proof path, not a second descriptor threshold.

Local stores remain permissive grow-only ledgers. They may contain a COMMIT whose signer does not currently satisfy WRITE(C), or a proof irrelevant to any resident collection. Admission is applied when a snapshot is observed. Later proof evidence may activate an old commit without rewriting or retracting it. WRITE never activates a MERGE: each store believes only the MERGEs its own host key signed, so another host's MERGEs stay in the ledger and fold into nothing.

The collection repair product

For one collection, sync compares three PATCH components:

• every structurally valid foundation record naming exact C: signed COMMIT and DERIVE records independent of current WRITE(C) admission. A collection is a set of foundations; a MERGE is its signing host's own lattice node, is never served, and a MERGE value fails the push that received it;
• every signature-valid native proof scoped to exact resource C and beginning at a root named by C's supported policy bindings, without requiring its grant handles to match those bindings. Subordinate-resource transport also includes proofs for R whose immutable descriptor declares C as its repair audience, under R's own policy roots; and
• C's held-blob set: the resident blobs reachable from C's descriptor, its foundations' direct references and its proofs' capability definitions (see Held blobs below). Only a host that replicates C in full keeps one.

Each set is represented by an immutable BLAKE3-Merkle PATCH. Collection records are keyed physically by the full 32-byte fingerprint of their exact canonical value; authorization evidence uses a repair_collection | proof_hash PATCH whose values share ownership of the original proof bytes. Only C's prefix is exposed; the wire leaf key is the 32-byte proof hash and its value is the complete native proof body. The host keeps one such index per collection overlay, not a shared global inventory. Resident-blob leaves use exact H as their 32-byte key and have an empty value. There is no companion claim blob or authorization closure to transfer.

The record root (record_root) commits to C and to the record and authorization-evidence summaries, including their leaf counts, under a versioned domain (version 4). It discloses no record, proof, count or component root. The held set is not part of it: the held set's Merkle root, all zeros when the set is empty, is the root of its own tree, which only two neighbours that both replicate C in full push each other.

The authorization projection authenticates proof bytes rather than taking a snapshot of who is admitted. Unavailable definitions, delegate-only grants, and quorum-incomplete paths remain evidence. Admission separately interprets the definitions' invocation and delegation action sets under the requested action's policy roots. Generic authority has no clock. Each root path is evaluated independently; no fixed-point over sibling paths can create delegation support.

A grant reaches its subject through the grant exchange (below): any connected holder of a proof naming a key hands it to that key, and the host whose key signed a grant dials its subject. That is the capability delivery boundary, not a Secrets-specific channel. Once one collection participant has the proof, the authorization walk distributes that one record to READ(C) peers. Interpretation needs any referenced capability definitions to be resident; pushes carry proof bytes, never definition bodies, which travel by hash over blob/1. An application may use the same proof kernel for another resource, such as Secrets key delivery, without treating collection READ as decryption authority.

Subordinate-resource transport reads R's resource_collection: C and its policy bindings from the same descriptor entity, without requiring a mutable referencing fact in C. When R's descriptor is not resident, the receiver of an authorization push fetches it by hash from the sender, over blob/1 and at most 1 MiB. If that descriptor routes the proof to C, both land; otherwise the proof stays deferred and the push does not land, so the sender's next push of the same tree carries the proof again. C's records push continues either way, and no WANT is emitted. A push is still gated by the sender's decision to send C to the receiver, not by R's action. Open action policies remain explicitly non-enumerable.

This product matters. Synchronizing only collection records would miss the case where a newly arrived proof activates an old COMMIT. Synchronizing a whole proof store would disclose unrelated capability structure. The complete repair evidence algebra remains Record × AuthorizationEvidence, scoped to C. The held component is different: it observes physical availability and may shrink or be rebuilt. It does not create semantic membership or change WRITE admission. The receiver always derives its admitted view locally; record and proof arrival therefore commute, and a publisher need not possess or present its own WRITE grant merely to replicate an inert signed record.

The host also updates these components independently. When a store snapshot reports unchanged collection records, its existing record PATCH is shared into the next overlay without re-enumeration or Merkle hashing. Changed collections apply selected record additions and removals to that retained PATCH. Native stores obtain the difference from persistent indexes; cold activation still constructs the initial full overlay. Blob arrival still refreshes the authorization observation: a newly resident capability or subordinate-resource descriptor can enable admission or proof routing without changing any record. The held set comes from the store's own index, fixed by the same snapshot; it can change the held set's root, never the record root.

A DERIVE is a foundation of its derived collection and is pushed like a COMMIT. A MERGE never replicates: each host builds its own merge lattice over the foundations it holds, and believes only the MERGEs its own key signed. Another host's MERGE records may sit inert in a pile, but they are neither exported nor accepted, and their result blobs are never held for sync. Signature verification happens when decoding foreign record bytes, not when rebuilding the local PATCH or observing the local store again. Dense record tag 7 carries a 224-byte DERIVE body; each wire value has one additional tag byte. COMMIT's 192-byte body and signature transcript are unchanged.

A records push still carries only records naming exact C. A DERIVE's source witness does not authorize disclosing another collection's records to a READ(C)-only recipient. The witness fingerprint is not a blob handle and is not fetched through the blob DHT. A replica lacking that record closure cannot claim exact support merely because it received the output endorsement; source-record provisioning remains a separately authorized concern. In particular, the current protocol does not automatically transport a cross-collection witness closure.

Choosing what to sync

A host syncs exactly the collections its pile selects. The selection is one register per collection in the pile's own configuration collection: the collection named config under private_policy of the pile's signing key. Its handle is a function of that key (selection::config_handle), so nothing has to remember it, and a pile that was never configured reads as an empty configuration in which nothing is selected.

A register state names its collection (sync_collection) and says whether it is selected (sync_selected). States are ordered by metadata::supersedes: every write mints a fresh id and supersedes every head it saw, so a change needs no clock and the history stays readable. sync_selection reports the heads rather than choosing among them: heads that agree give their value, heads that disagree give Conflicted, and a head the reader cannot decode is skipped. Only Selected selects; an unset, unselected or conflicted register selects nothing. The schema lives in triblespace_core::collection::selection, so the daemon and every program that writes the selection share it. Its anchors were minted with trible genid on 2026-10-05: 99D9A2B7C5636FEFDE482DB3A6D01EE6 for sync_collection and 24EB46425226C0025A3C85D34F8919CE for sync_selected.

An operator writes and reads it with the CLI, under the daemon's key resolution:

trible pile net select DATA.pile [--key KEY] COLLECTION_HANDLE [COLLECTION_HANDLE ...]
trible pile net unselect DATA.pile [--key KEY] COLLECTION_HANDLE [COLLECTION_HANDLE ...]
trible pile net selection DATA.pile [--key KEY]

select and unselect write one state per handle that supersedes every head, so they also settle a conflict between earlier writers. Unselecting is a state, not a removal, and a later select supersedes it. Each line names the handle and the collection's name, or says that no descriptor is resident yet: the selection stands and applies once the descriptor arrives. selection lists every collection the configuration names with its value (selected, unselected, unset, or conflicted between N heads) and its name; it writes nothing and probes nothing.

The host reads the register from each serving snapshot, so peering follows a change at the next store observation. pile net sync also reads it at startup and before each reconcile pass, about once a second, to activate newly selected collections. An unselected collection stays active until the process exits, but its peerings end with UNPEER frames and it is no longer pushed. A selected collection whose descriptor is not resident is fetched by its handle through the DHT (the descriptor warmup under Live request scheduling), and peering for it begins once the descriptor validates. A node that does not select C never asks to peer for C and refuses every request for it.

Where the sync daemon listens

The same configuration holds one more register per endpoint key: the addresses the sync daemon running as that key is bound to. A state names the endpoint (sync_endpoint) and lists every address (sync_address, a SocketAddress); a state that lists none says the endpoint is bound to nothing. States supersede each other like selection states, with a fresh id per write. pile net sync writes the register (selection::write_sync_addresses) on its first pass and whenever the sockets its endpoint is bound to change, unless the register already holds exactly them. A socket bound on every interface is recorded as loopback, because the reader is a process on the same machine that opens the same pile. A reader (selection::sync_addresses) takes every head's addresses: they are hints for a dial, so heads that disagree are tried together rather than reported as a conflict. The anchors were minted with trible genid on 2026-10-06: 0F1FE7641D8246229EDAB3C2FF3E152C for sync_endpoint and 55C20BF071C57CAA74D0481ED4C3077A for sync_address.

This register is where the daemon's endpoint ticket went. Nothing copies a ticket from one daemon into another's command line any more: a process reaches its own pile's daemon through the pile, and every other peer through the pile's content (Where peers come from).

One connection per peer

A ConnectionTable holds the connections of both directions, keyed by the remote's TLS-authenticated endpoint key. Every connection, dialled or accepted, runs one accept loop, and the first byte of every stream is its type tag, read before any permit is taken:

An unknown tag, or an unknown dht/1 operation, resets only its own stream (RESET_UNKNOWN), and a stream that sends no tag within ten seconds is dropped. recon/1 and walk/1 take no permit. A request stream (dht/1 or blob/1) does: an opener keeps at most 16 open per connection, the accepting side holds at most 32 per connection and resets the rest (RESET_BUSY), and at most 16 are served at once across the host, each within 300 seconds. A stream first takes one of 8 permits its connection has. On a connection that carries no peering it then takes one of 8 permits that every such connection shares, and then one of the host's; a stream on a neighbour connection then takes only the host's. Keys cost nothing, so for strangers a share per key or per connection would bound nothing: strangers that hold their requests open until their deadline, on however many connections, occupy at most half the host, and the other half stays for the peers it syncs a collection with. Those take nothing from the strangers' share, so the connection's own share is what keeps one neighbour that holds its requests open from occupying the whole host.

The dialler opens recon/1 as it connects, and its first frame, OPEN, carries the dialler's sequence number. The counter starts at the wall clock in nanoseconds, so a restarted node's dials outrank the ones it made before. When a pair holds two connections, both sides keep the same one: of crossed dials, the one dialled by the lower key; of two dials by one side, the one with the higher sequence number. The loser drains: nothing new is opened on it, and it closes once no request stream is in flight on it and no frame has crossed it for two seconds. What this side had asked to peer for on it, it asks for again on the connection kept.

A connection closes after 120 seconds without a frame on any of its streams, sent or received. Push timers cap at 60 seconds, so a connection that carries a peering does not go idle. Above 64 connections in one direction the least recently used one is evicted, draining and unused connections first. A neighbour connection, one that carries a peering, is never evicted.

A connection carries exactly one recon/1 stream, the dialler's. One opened by the accepting side, or a second one, is a protocol violation and closes the connection. Frames go both ways at once, and both directions are read concurrently. When the stream ends, for any reason, the connection closes with it: the peer finished or reset it, it failed, or a writer waited 60 seconds for stream credit because the peer stopped reading. Every push on the connection, sent or received, and every peering ends, frames still queued for it are dropped, and nothing is reopened on it. What replaces it is a later dial, which the candidate order governs (see Neighbours): a peer whose stream ended comes back only through a fresh dial when a collection's order next reaches it. A peer that ends every stream as it arrives therefore costs a collection one dial per redraw of its order, at most one a minute, rather than a loop of streams.

A connection's frame queue refuses no frame, so what it holds is bounded by what is put on it. A side asked to peer leaves the request unanswered while the connection already queues 1,024 frames, at most 64 KiB each, and the grant exchange answers one proof request per digest it sent; a peer that asks faster than it reads therefore stops being answered rather than growing the queue, and an unanswered peering request waits until the stalled stream's end closes the connection. A push is not on this queue: it has a stream of its own, and QUIC's flow control holds a sender whose receiver stops reading (Walk streams).

A recon/1 frame is its kind (one byte), a big-endian u32 payload length, and a payload of at most 64 KiB. Every frame about a collection starts with the collection's 32-byte handle, so the frames of every collection share the stream:

Peering flags are one byte: bit 0 is the send flag, bit 1 the full flag, and in a PEER_REQUEST bit 2 marks an invitation. A proof digest is a presence byte, a Merkle root (zeros for the empty set) and a big-endian u64 count. Credentials, like the proofs of a PROOFS frame, are a big-endian u16 count followed by that many proofs, each a big-endian u16 length and its bytes. A reader skips a frame kind it does not know and a proof whose bytes do not decode; a known frame whose payload does not parse is a protocol violation and closes the connection.

Peering

Peering for C is two keys' wish to sync C over the one connection between them, and C is the unit of everything that follows: each collection has its own neighbours, pushes and timer, multiplexed on the connection. A side asks a key to peer for C only if it selects C. The PEER_REQUEST names C and carries the asker's flags and credentials:

• the send flag says the asker admits the peer, which passes READ for C under the asker's evidence;
• the full flag says the asker replicates C in full; and
• the credentials are the asker's own proofs for C naming itself, truncated at its prefix, READ and WRITE alike, at most 1,024 of them. Those that do not fit the request travel first in CREDENTIAL frames.

Apart from the grant exchange, which hands a key only the proofs naming that key, these credentials are the only collection data a side sends before it is admitted. Its other frames before admission carry only handles, hashes, flags and digests.

The key asked accepts only if it has C active with its descriptor resident, selects C, and has room for the peering (at most 16 per collection that others asked for or that it invited, unless this connection is already peered for C), and only if the asker passes READ for C under its evidence and the presented credentials, or the asker set its send flag and passes WRITE. PEER_ACCEPT carries the acceptor's own flags. C flows only in a direction whose sender set its send flag. A writer that cannot read therefore asks a reader with its send flag set, the reader accepts with its own flag unset, and records flow from the writer to the reader only. A refusal is a PEER_REFUSE frame; recon/1 stays open.

Admission is evaluated again whenever its inputs change: a store observation that changes C's evidence or brings a missing definition, a CREDENTIAL frame, a change of replication mode. A peering's send flag follows READ admission and its full flag the replication mode; a changed flag goes out as PEER_FLAGS, and a peering whose two send flags are both unset ends. When a node's own proofs for C change, it sends CREDENTIAL frames to C's neighbours and to keys that refused it. Credentials a peer presents are kept only if they name the peer and are evidence for C. Unselecting C sends UNPEER to each of C's neighbours, and a closed connection ends its peerings.

Two Full neighbours are the two sides of a peering in which both set the full flag and each counts the other as a reader or writer of C: this side sends C to the peer, or the peer passes WRITE under this side's evidence. Only they push each other their held sets. A DHT provider that may neither read nor write C is never a Full neighbour, whatever its flag says, so it cannot make this side fetch the blobs it names and pass them on; changed evidence re-evaluates which peerings count.

A side remembers, per connection, the requests it refused and the capability definitions whose arrival could admit them: admission names missing definitions (QuorumOutcome::Undefined) rather than collapsing to a refusal. Those definitions are fetched over blob/1 from the asker, then from its credentials' root and delegated keys among connected peers. When a credential, a fetched definition, a new proof or the refuser's own selection admits a refused request, the refuser sends an invitation: a PEER_REQUEST with the invitation bit, which counts against the receiver's room for inbound peerings. The asker never asks again on its own.

Neighbours

Each selected collection has up to five neighbours this side asked for, plus the peerings others asked for. The candidates come in three tiers:

1. grant-chain keys (C's policy roots and the roots and delegates of its retained proofs) and owners (signers of C's records who pass WRITE) that pass READ under local evidence;
2. the other grant-chain keys and owners; and
3. DHT providers of blob_locator(C), the holders of C's descriptor blob.

Within a tier keys are ranked by blake3(salt || key) under a salt drawn fresh for each order. Readers come first because a signer that cannot read does not relay; free DHT keys come last. The position in that order is the only retry state: a refusal, a failed dial or an ended peering (its connection closed, as it does when its recon/1 ends) moves on to the next candidate. A key that refused is not asked again on that connection, and the others come round again only when the order is redrawn. Reaching the end redraws it with a new salt, no sooner than 60 seconds after the last draw unless the candidates changed. A redraw that is not merely of changed candidates also looks up C's DHT providers again, at most three such lookups at once per host, each with a three-second lookup window. Neighbour sets are refilled every second.

Every ten minutes, when a collection's five asked-for places are full and another candidate is available, one healthy asked-for neighbour chosen at random is unpeered and replaced by the next candidate, so neighbour groups that formed separately during a partition meet again after it heals.

HealthSnapshot::peerings lists every peering on an open connection: its collection and peer, whether this side asked for it, whether both sides accepted, which way C flows, and which side refused the other.

Pushes

Each neighbour exchanges one kind of one selected collection on a bidirectional walk/1 stream: records, authorization evidence, or (between Full neighbours) held references. Either side may open it. Each sends its own immutable delta against the last agreement on this connection; both directions share the stream. Nothing is broadcast or forwarded merely because it arrived.

Each selected collection retains its existing Trickle-style timer: two to sixty seconds, doubling while quiet, with one transmit opportunity in the second half. A local records/evidence change restarts the minimum interval unless an incoming exchange is landing it. A successful changed receive resets it once, not for each intermediate snapshot. Held-set changes wait for the ordinary timer. A successful exchange followed by genuinely new local keys starts another immediately; a lagging published subset of the agreement does not. Failed exchanges wait for the timer. Timer inputs are physical serving observations, not delta roots.

The walks task uses only current connections and never dials. At most one exchange per peer, collection and kind runs at a time. Simultaneous opens preserve the stream opened by the smaller peer id and reset the other as a duplicate; a newer incoming stream can replace an existing incoming one. Both Open and Incoming require local selection and an active descriptor. Active peering flags determine send/receive directions; credentials provide fallback admission only when there is no active peering. A false send flag cannot be widened by another grant.

An agreement G is immutable PATCH state scoped to peer, collection, kind and connection. It contains only keys known held on both sides, retained after both LANDEDs and successful actual writes and stream finish. Failure clears local G without retracting landed values; close/deselection clears it and retires running drivers, so a stale completion cannot restore it. G is not durable. Fresh connections may compare full sets again, with equal-prefix pruning.

Successful symmetric exchanges compute the same union. An asymmetric final finish failure may clear G on only one side: these sound common-held lower bounds can temporarily differ, particularly while publication lags. Delta commitments remain independent and valid; ordinary publication and later exchanges can converge the bookkeeping. There is no base-negotiation or distributed durable receipt added here.

Walk streams

The opener writes the stream tag and OPEN naming collection, kind (records 0, authorization 1, references 2) and profile (0 pull Prefix, 1 cached RIBLT, 2 offer Prefix with opener sending, 3 offer Prefix with opener receiving). OPEN remains 34 bytes. Profiles 2/3 require Records or AUTH and the opposite locally authorized one-way role before driver registration. Profile 0 always uses pull semantics, even under a one-way local authorization view. Unknown profiles and inconsistent offer roles are refused, not guessed from data frames. The accept loop hands both halves to the walks task. Missing OPEN within ten seconds, or no task, resets the stream. Frames have a one-byte kind, big-endian u32 payload length, then at most 64 KiB. Keys are 32 bytes relative to the kind's base; authorization hashes bind the full collection32 || proof-id32 key. Integers are big-endian.

Every SUBTREE payload is 73 bytes, 78 including the frame header, regardless of fanout. Depth is 0..32; all prefix bytes outside depth must be zero. Depth0 is the ordinary empty-prefix delta-root announcement; an empty delta has zero hash and count. There is no ROOT, LEAF, NODE child array, HELD or reply bitmap. Unknown kinds are skipped; known malformed payloads fail only the exchange. The retired 0x02 and 0x04 kinds are not compatibility paths.

For the prefix algorithm, freeze native PATCH difference Δ = pinned minus G, or empty if not sending. Announce [] and then visit its actual compressed trie nodes continuously FIFO, putting children at the back in ascending edge order. Children are separate later announcements, never embedded metadata. A nonempty compressed root prefix is announced after [], including a singleton's full-depth key. FIFO concerns logical trie levels; compressed byte-prefix lengths need not be monotonic. There is no level barrier in pull mode. Offer mode additionally obeys its bounded receipt window below. DONE follows this traversal.

A peer announcement matching our own delta at that prefix prunes its queued proper descendants. If we have not announced the prefix yet, our own matching announcement still goes out exactly once; if already sent, it is not duplicated. That statement concerns work remaining when the match arrives: network delay and unequal frontiers permit earlier in-flight descendants. FIFO alone supplies no universal one-level overshoot bound. Previously agreed siblings cost no frames because difference construction omits their keys. Independently acquired overlap uses the same prefix-matching rule.

The receiver reconstructs ONLY the peer's delta in a separate typed PATCH, initially empty. An exact locally held subtree commitment can populate that portion; otherwise full-depth key announcements authenticate their own leaf digest, and values are pulled by key or reused from the latest live observation. Unknown intermediate hashes are provisional. At DONE, after all requested values, fetches and inserts finish, the reconstructed peer root/count AND every received prefix/hash/count must match, with canonical compressed prefixes. Count sums, frame counts and the local union are not membership proofs. Omitted leaves, foreign replacements at equal count, duplicate prefixes, wrong hashes and wrong prefixes cannot close. Exact non-root subtree inclusion currently enumerates its keys; only empty-prefix whole-tree inclusion uses a shared clone.

Records must decode with valid signatures to COMMIT/DERIVE for the collection under the requested fingerprint. Proofs must match their id, signatures and collection evidence; a subordinate descriptor is fetched and checked, or the proof remains deferred and fails completion. References have no VALUE payload: resident blobs are noted held, absent ones fetched by hash and landed with their held note. All actual landing/fetch/deferred outcomes precede LANDED. A value request is served only from the immutable advertised delta. A receive-only side announces empty and cannot advertise its private pinned tree or serve ordinary values. Offer-mode receipts only repeat commitments the sender already offered.

One-way typed offers

Locally opened one-way Records and AUTH exchanges select offer profile 2/3. References retain their existing exact-fetch/landing path; two-way Prefix and CachedRiblt are unchanged. Each generated SUBTREE commitment is registered under its exact prefix/hash/count on this stream, before writer completion. A nonempty HAVE must match such a registered commitment; not-yet-generated or mismatched receipts fail the exchange. Registration is neither a completed-write receipt nor cryptographic proof that the peer observed the offer. Duplicate receipts are idempotent. HAVE prunes queued descendants and suppresses unqueued eager bodies, but cannot recall already queued/written frames.

The receiver acknowledges an explicit offered leaf if its immutable typed value is already held, or after its validated VALUE has actually landed. A routed AUTH proof returns no credit while its descriptor is missing, fetched or unlanded; the descriptor and proof must both land successfully. Retained sound G may certify an exact offered subtree, including a leaf whose physical publication lags. An already queued descendant after such a coarse HAVE creates no new body debt. Private-prefix equality grafts and unlanded VALUEs remain proof-only: they never establish HAVE eligibility. Adding unrelated private siblings with the same offered-key possession and G leaves logical receipt decisions unchanged.

An opaque cold root hash does not prove offered-set containment in a larger private set. Thus this first implementation gives leaf possession feedback and G-certified coarse pruning, not general early cold-subtree pruning. Positive leaf feedback deliberately reveals possession of already-offered identities; it does not claim timing or traffic-analysis secrecy.

At most 64 offered leaves await possession receipts. Their bodies are sent speculatively without VALUE_REQUEST, before waiting for those receipts; an all-missing transfer can progress beyond the window through ordinary landing ACKs. Authenticated offered keys alone admit eager VALUEs, with the existing signature, collection, fingerprint and proof checks. Racing duplicate bodies are validated but land at most once. A missing receipt eventually fails under the existing progress deadline; there is no silent pull fallback. Offer/receipt indexes and admitted-body identities share the explicit 1,048,576 entry ceiling. This window bounds instantaneous speculation, not cumulative duplicate bytes over an entire cold high-overlap stream, and makes no universal speedup claim.

HAVE releases speculative credit only. Exact peer delta reconstruction, all fetch/landing obligations, final LANDED and successful actual writes/finish remain necessary for G. The driver drains queued receipts before finishing; failed/retired streams cannot be revived by late landing callbacks.

After both LANDEDs, propose G' = G union Δlocal union Δpeer. The driver retains it only after actual queued writes and successful finish; an enqueue is not a receipt. The peer's LANDED separately acknowledges our send, even if our receive fails. Successful LANDED write separately acknowledges what landed here. Failures reset/stop the exchange and clear reusable G, while already landed data remains. Publication lag cannot shrink G: pinned subsets produce empty differences.

Budgets grow on demand: FIFO positions, received prefix commitments and matched announcement indexes each allow 1,048,576 entries; the peer delta proof permits at most 1,048,576 leaves. Its advertised count is checked before exact-root reuse, so even a fresh identical larger set is refused. This is a real honest-large-delta limit, not just a hostile frontier bound or a claim of unbounded synchronization. Withheld-value demand has a separate 32,768-entry ceiling. Requests and blob fetches share 64 slots; at most 1,024 unacknowledged values go to landing before reading pauses and stream flow control applies.

Incoming frame/value waits and blob fetches have 60-second progress deadlines, not total-transfer timeouts. After DONE, peer landing may take as long as the connection remains. Actual outgoing byte writes and final finish are separately progress-bounded/cancellable, including when a driver is retired. A walk takes no request permit but uses transport stream credit and counts as connection use. Small and scratch/sim gates do not establish many-collection live stream-credit liveness or throughput.

Cached full-component discovery

This source-only ALPN32 candidate defaults to cached RIBLT for mutually authorized locally opened streams, through both the CLI and Algorithm::default(). Explicit trible pile net sync --walk-algorithm prefix or PeerConfig::walk_algorithm = Algorithm::Prefix retains the Prefix option; --walk-algorithm cached-riblt remains accepted too. This changes the source default only, not a live daemon or its configuration. Incoming streams explicitly choose either supported algorithm regardless of the local opening preference. With simultaneous opens, the existing lower-peer-id survivor determines which explicit algorithm runs. Cached symmetric RIBLT is refused unless both send and receive are authorized for this collection. Local opens with one-way authority choose prefix, so receive-only holdings never enter symmetric equations. Collection/component admission and reference eligibility are otherwise unchanged.

Each published collection observation owns three separate latest cache Arcs: Records, scoped AUTH, and positive resident References. A generation binds the immutable native source, collection, component and profile to one flat grow-only Vec<CodedSymbol<KeySymbol>>. Running exchanges retain that generation; they have residual decoders, not full per-peer encoders or cell caches. Normal local publication and incoming landing use the same owner update. An unchanged key set reuses its cache Arc, while the exchange still pins the current typed observation for body serving and authorization. Actual native PATCH differences new \\ old and old \\ new update the already-prepared prefix by checksum/symbol XOR and signed membership count. Publication copies the entire prepared vector first: update cost includes O(prepared cells) copy, PATCH differences and sparse scatter, not merely the small edit. Old leased buffers remain separately resident. Completed cached exchanges use the agreed native union only transiently for health, receipts and moved-source checks, then release it instead of retaining per-peer G. They also discard any older Prefix G for that component. Prefix exchanges continue retaining their reusable agreement as before.

Preparation is lazy, initially 1,024 cells, doubling up to 1,048,576 cells. Growth and an update's prepared-prefix snapshot are serialized; the new generation is then updated privately. Growth scatters the entire frozen source only into a private new tail; existing cells never change. This avoids adding symbols to an already-advanced upstream encoder. Old leases can grow their own generation without becoming the latest owner. There is no retained reverse-membership index; the public mapping reconstructs each changed symbol's memberships. Cell storage is 48 logical bytes (and measured Rust layout on the tested target), at most 48 MiB per prepared generation, plus Vec capacity, native PATCH sharing and decoder state. Exhaustion, poisoning or count overflow fails the exchange or publication; none is a success or silent prefix fallback. Each decoded local-only or remote-only set and the eager-body deduplication set have a 1,048,576-key ceiling. Unlike Prefix's materialized delta proof, the cached COW proof does not cap absolute source cardinality at that value: source counts must fit i64, while newly retained difference state is bounded independently. A transient typed frontier may exceed either final source's cardinality while eager bodies arrive. Cursors into those difference vectors drain at most 32,768 queued body keys and 64 active owed bodies/fetches; the queue bound is not a lifetime delta bound. The decoded-support ceiling and prepared-cell ceiling are separate limits: fitting the former does not guarantee decoding within the latter. This candidate does not support unbounded cold synchronization. Local exhaustion is reported as a resource limit, separately from malformed peer frames; cache poisoning/count-invariant failures are separately unavailable, never success.

Profile 1 uses exact 32-byte symbols, including trailing zeros. SipHash-2-4 hashes precisely those raw bytes, with no Rust Hash length prefix. Its public 16-byte key is the first 16 bytes of BLAKE3(domain || collection32 || component-byte), where domain 2D1743DA8080C9DBDAEB886FEE34089C was minted with trible genid. The checksum u64 is also the mapping seed. Mapping is pinned upstream ngerakines/riblt-rs commit d0737aeeac0f705ea0f26ebf7421a8b9f307b7d0: every symbol contributes to cell 0; advance the u64 PRNG by wrapping multiply by 0xda942042e4dd58b5, then compute gap = max(1, ceil((f64(index) + 1.5) * (4294967296 / sqrt(f64(prng) + 1) - 1))) and saturating-add that integer gap to the index. Operations use IEEE754 binary64 in the stated order, matching RandomMapping; cross-platform profile parity is a requirement, not a Rust-layout assumption. This is a public deterministic map, not authentication or adversarial-workload protection. Roots reject a different profile key.

Cells describe the whole component, never source \\ G. The sender emits its root immediately but deliberately waits for the peer's root before preparing or sending any cells. This initial feedback barrier validates the profile and avoids both cold preparation and coded oversend for equal roots; it does add startup latency and is not a feedback-free root-plus-cells stream. After that exchange, cells proceed without cell-by-cell request rounds. A bounded residual-only decoder receives peer cells minus cached local cells: XOR symbol and checksum, subtract signed count. Indices are contiguous and bounded. Before admission, the residual count must fit the signed decoded-support bound, independently of source size. Count arithmetic is checked throughout peeling. Upstream's unbounded try_decode is not used on network input: the local boundary adapter reuses its public mapping and keeps its Encoder as an oracle. The adapter charges mapping, cell updates and queue work inside every cascade loop, with a total budget of 64 times the maximum cell count. This is an explicit work budget, not a latency guarantee. Queue membership is deduplicated and bounded by the received prefix; purity is rechecked at dequeue, the recovered key must map to the triggering cell, and repeated or cross-sign recovery is rejected. Exact zero includes all symbol bytes, not merely count/checksum. Incremental mapping heaps retain only recovered difference keys, never a full-source inventory. Peeling reveals local-only bodies, which are pushed eagerly from the pinned typed source; remote-only bodies are owed pushes, not an additional VALUE_REQUEST round trip. References continue through the ordinary exact-blob fetch/held-note path. Bodies never enter decoder windows. Duplicate bodies are admitted at most once.

After complete peeling, local-only keys must exist in the original source, remote-only keys must not, and both recovered sets must be unique. A sparse native Merkle union summary combines the COW source minus local-only keys with the remote-only key PATCH, sharing all unchanged native subtree summaries. It checks the exact peer root/count without waiting for unknown typed bodies and without a copied full-source key inventory. AUTH hashes still bind physical collection32 || proof32 keys. Typed peer reconstruction and canonical signature admission independently guard final landing.

Once this exact proof holds, stop coded output only when the complete locally written cell prefix reaches the decoded incoming prefix. Opposite residual counts and identical XOR equations then give the peer the same solvable prefix. Equal root/count with the same profile/domain can stop at zero cells; the upstream decoder's initially true decoded() alone cannot. DONE before this proof fails. Eager bodies take priority over unqueued cells; one active write, one queued frame and a single deferred cell bound coded lookahead independently of prepared megabytes. Each cell is locked/copied/encoded individually: 52 payload bytes, 57 framed bytes, not zero-copy wire. Root, OPEN, control frames and bodies are additional traffic. Body/fetch/landing obligations, both LANDEDs, actual LANDED write, successful finish and current Task generation all remain required. Premature EOF, reset, malformed data, failed landing or retired drivers cannot turn decoding into success.

Landing

One landing task per host lands the values of every push in arrival order. It takes up to 4,096 waiting values at a time, inserts them in one pass, publishes one serving snapshot, and then tells each push how many of its values landed and how many inserts failed. Record publication builds operation-local typed PATCH deltas and applies one native difference and union per batch, rather than rehashing the retained large tree after each record. Structural/collection and exact attached-value checks still precede the key-only set operations; old snapshots retain their unchanged native subtrees. A group whose snapshot cannot be published counts as failed and withdraws serving. Later leaves of every push are therefore checked against what already landed. The task also reobserves the store every two seconds, so appends by other processes reach the serving snapshot, and from there pushes. Landing does not flush: explicit close remains the persistence boundary.

Serving snapshots are a latest-value handoff. The peering, scheduling and walks tasks each read the newest observation, never a queue of them. Evidence that arrives outside pushes (proofs from the grant exchange, fetched definitions, descriptor warmups) crosses the bounded admission bridge and lands at the next Peer::refresh, which also reobserves the store at once.

Grant exchange

A grant notifies its subject. On every connection, and whenever it changes, each side sends on recon/1 a PROOF_DIGEST of the proofs it holds naming the other side: the subject-truncated prefixes its collections' evidence indexes under that key. A side whose own proofs naming itself differ sends one PROOF_REQUEST per digest and connection; the answer is PROOFS frames closed by PROOFS_END and holds only the proofs naming the asker's TLS-authenticated key, at most MAX_PROOFS_PER_EXCHANGE (1,024). The sender answers once per digest it sent on that connection, and a request before any digest goes unanswered. The answer reflects what is held when it is sent, so an asker whose request crossed a newer digest already has the newer state. A received proof, like a peer's credential, is kept only if it names the receiver or the sender. It lands when its held descriptor validates it, waits in memory while the descriptor is missing (HealthSnapshot::available lists its collection), and is dropped otherwise. A waiting proof waits for every sender that delivered it and is dropped when the last connection of the last of them closes, so keys that came and went do not keep the bound full. Waiting proofs are bounded: one sender keeps at most 256 waiting and every sender together at most 1,024, and the rest are dropped before their signatures are checked and fetch nothing. A proof counts against every sender it waits for, not only its first, so a sender that repeats others' proofs spends its own share on them and keeps no more waiting than its share once the others leave. A proof that is evidence under a held descriptor never waits, so these bounds keep none of those out. A proof for a collection that is not active is validated against the resident blob at its resource, which is taken for a descriptor only up to 1 MiB and decoded at most once per store observation, not once per proof.

The definitions a proof names are fetched over blob/1 from the sender, then from its root and delegated keys among connected peers, as are those a refused peering request waits for. Definitions are policy metadata and are fetched under the same 1 MiB bound as descriptors, so a key cannot make a node store a blob of its choosing by signing a proof whose capability handle names it. A host also dials the subject of every grant its own key signed, once per grant and process, so a grant written while the subject is unknown still reaches it.

Blob transfer is lazy and bearer-addressed

A records push does not carry descriptor dependencies, payloads, metadata, attachments, or derived artifacts. It transfers lattice evidence, not those bytes; only a references push between two Full neighbours fetches blobs, and only those the sender holds in that collection. The separate bounded host metadata warmup described below is not part of a push. A resolver can then select the cheapest resident support-equivalent cover and request only the missing immutable handles that matter to that computation.

Knowledge of a full content hash H is the read capability for those exact bytes and the secret needed to discover them. Publication and lookup derive a full-width opaque locator under a random 32-byte context key:

L = BLAKE3(LOCATOR_CONTEXT || H)

The input is exactly one 64-byte BLAKE3 block, so a locator costs one compression. Until generation 27 it was a derive_key over a context string, which cost two; the locator is computed for every resident blob on every index build, so the difference is paid many times.

This is not encryption or protection against guessing the exact blob content. Someone who can guess those bytes can compute both H and L and confirm a matching provider token. Confidential low-entropy content needs encryption; the additional locator hash prevents disclosure of H from a copied L, not reconstruction of H from already-known or guessed bytes.

Every served resident blob may renew a soft lease at L on nearby XOR-DHT nodes. The lease carries the token keyed(provider; H || TOKEN_CONTEXT): keyed by the provider's authenticated endpoint ID, over H and a random 32-byte context, one block. A requester who knows H rejects forged candidate entries before dialing; the directory learns neither H nor collection membership.

An exact GET sends only L. The provider first proves knowledge of H with keyed(provider; H || requester); only then does the requester answer with keyed(requester; provider || H), and only then do bytes flow, checked against H on arrival. Each proof is one block and binds both authenticated endpoints. H sits on opposite sides of the two messages, so while H is secret a requester that knows only L cannot echo the provider's proof back, and the proof a provider hands out is not the requester proof of the reversed connection. The separation rests on H being secret, not on algebra: an H equal to an endpoint identity would make the swapped pair coincide, but such an H is public anyway. An exchange whose two endpoints are the same identity authenticates nothing and is refused.

When a DHT node answers FIND_VALUE(L), it also consults its already-installed, snapshot-coherent L→H index. If L is resident there, it returns its own endpoint-bound token without waiting for a PROVIDER_PUT. This deliberately extends the old lease-only answer: one reply slot is reserved for the resident self hint and any stored self entry is deduplicated. At most 63 other live leases follow in peer-ID order; when 64 are available, one chosen uniformly at random makes room. Without a resident self hint, the usual limit remains 64 leases.

Each directory node keeps a bottom-k sample of every key's publishers rather than its first arrivals. It ranks each (key, provider) pair by BLAKE3 keyed with a salt it draws once at start-up and never sends. A key may store an even share of the remaining membership budget, never less than one 64-entry reply and never more than 1024, which bounds what one request walks. A newcomer to a key at that cap is kept only if it ranks below the key's largest-ranked member, which it replaces. A shrinking cap trims a key's largest-ranked members when the key is next touched. A renewal keeps its place. Arrival order therefore does not decide membership, renewals cause no churn, and the K replicas of a key keep independent samples. Because the salt is secret, a publisher cannot choose identities that rank well. A key storing more than 64 members answers each request with a fresh uniform sample of 64, so repeated requests reach every member.

This is a query-time answer, not a new lease or publication attempt. It reads no payload, creates no WANT or collection authority, and still works with a zero announcement budget. A missing or withdrawn serving snapshot supplies no self hint; independently stored leases remain ordinary soft hints. The blob locator namespace does not synthesize collection-participant hints from a resident descriptor. Client replica selection, token verification, and the mutual bearer GET are unchanged: a known holder outside the selected DHT replicas is not probed as a fallback. Remote publication is still needed when the holder is not itself one of the selected replicas.

A returned self token therefore does not demonstrate that any publication occurred. Foreign entries in this host's replies still come from retained advertisements. The blob_directory diagnostic reports valid self and foreign token counts separately, alongside its existing totals, without logging handles or tokens. These classify hint sources, not cryptographic publication receipts; neither category proves that a subsequent GET will succeed.

A lookup is one FIND_VALUE(L) per hop. Each authenticated reply carries the responder's K closest verified routes and its hints for L, so no directory round follows routing: every replica the lookup reaches has already answered with its hints. Named referrals remain unverified candidates until they answer in turn. The requester checks every token and drops a hint whose token does not prove knowledge of H before any dial. The local directory answers like a replica's when the first authenticated reply, or the closed lookup, places this endpoint among the key's closest K; no cold lookup starts from a retained local provider lease before routing progress. Publication runs the whole lookup and puts to the closest K responders, ignoring their hints. Collection discovery, which supplies the last tier of a collection's peering candidates under the descriptor's own locator blob_locator(C), runs the whole lookup and takes the union of every verified hint.

An exact-H fetch starts a verified provider as soon as a reply names it, rather than waiting for the lookup to finish. Lookup requests and exact provider GETs share three concurrent request slots. While routing is open, provider GETs occupy at most two, leaving capacity for fresh routing; a usable provider takes the next such slot. Two progressing acquisitions leave only one routing slot in the same unchanged routing window; this is not a proof of equal discovery coverage under every timing pattern. Empty early hints do not end a lookup whose routing remains open. At most 64 distinct providers can be attempted per fetch. Duplicate hints never spend another attempt. Pending providers are ranked among the hints received so far, bounded by the remaining attempt allowance, by keyed(salt; L || provider) under a salt the requester draws once and never sends; collection discovery ranks its union the same way. A public rank such as the provider's XOR distance from L would let a provider grind an identity that every requester tries first. A later better-ranked hint can replace pending work, not an already-started request. The transient attempt subset is therefore deliberately arrival-sensitive; it need not equal the closest 64 in the final reply union. Collection discovery still waits for its canonical, reply-order-independent union.

This removes routing's final barrier when early evidence is useful, not every possible head-of-line delay. Three stalled requests can still occupy the shared slots; in particular, two stalled provider GETs and a stalled lookup request keep a later replica's hint from arriving within a shorter caller deadline. An early large set of valid but unavailable hints can also exhaust the 64-attempt cap before a later useful hint arrives. The caller's existing end-to-end timeout, per-operation deadlines, pooled-connection cancellation rules, provider-token checks, mutual bearer proof, body-receive memory bound, and final hash check are unchanged. Routing expiry drops only issued lookup requests; their hints are then unknown, so a fetch that found no verified hint reports the lookup incomplete rather than the blob absent. A progressing body continues within the original caller deadline. Early verified success or caller cancellation drops all owned futures without fabricating protocol failures or closing healthy pooled connections. No provider hint or result is persisted by this scheduling step.

The direct stream also keeps H off the wire. The requester sends only L. The provider resolves L in its resident locator index and proves knowledge of H first, binding the proof to both authenticated endpoint IDs. Only after verifying that proof does the requester send its role-separated proof of H. The provider then returns bytes, which the requester hashes and compares with H. A party which merely copied L therefore cannot learn H from a requester or successfully serve bytes for it.

Exact body reception admits at most sixteen active body futures. They share one 1 MiB scratch buffer, held only for a bounded ready-read batch and its local write, never while awaiting the network. Each batch also has a read-poll limit so a stream yielding tiny fragments cannot monopolize one executor turn. Received bytes grow a temporary file; one immutable mapping backs the completed Bytes. An idle or partial stream therefore does not block a healthy body from using the scratch buffer.

The existing per-body bound is 64 GiB. A separate process-wide 64 GiB temporary backing budget accounts in rounded 1 MiB units, including partial files and completed Bytes until their last owner drops. Retaining many tiny completed bodies can exhaust this budget too. Body-slot and backing-budget exhaustion are local, immediate errors, not evidence against the provider: they preserve its pooled connection for a later attempt. Cancellation, truncation and other failures return their resources. Network waits remain inside the caller's deadline; synchronous local file writes cannot themselves be preempted by that async deadline. These limits neither guarantee throughput nor replace the final comparison of the received bytes' hash with H.

Fetching neither asserts collection membership nor activates a commit. It does not consult C or READ(C), and creates a durable WANT only when the caller asks the WantStore to record Blob(H). Provider leases are bounded soft state and may disappear without changing semantic data or local retention.

Directory proof-node reuse experiment

provider/directory_model.rs is a wire-free comparison of signed entry leases and signed immutable inventory roots. Its shared_proofs.rs extension retains the same membership/lease PATCH repair and the same final inclusion verifier. Only the proof transfer changes: each root-bound membership carries an ordered list of node digests; a receiver fetches missing native PatchNode bodies from that selected support's proof and retains at most 4,096 validated nodes in a PATCH<32>. It neither asks for a full inventory nor consults a global member registry. The parent's complete-topology placement oracle remains an explicitly synthetic comparison, not an implementation of distributed responsibility.

Node presence supplies bytes, never authority. An all-hit proof must still match the provider, member, signed range, exact inventory root and live lease, and must pass the entire canonical node walk. The cache is populated only after successful membership admission. Original signed absolute expiry survives forwarding and cache reuse; changing the inventory requires new root-to-member associations. The model assumes one trusted zero-skew absolute clock and already-selected first-byte responsibility ranges; it does not solve either clock recovery or dynamic ownership discovery.

The first Stars run (2026-09-07 23:24 UTC, base d0db6da2, rustc 1.98.0, unoptimized tests with debug information disabled) used 256 deterministic opaque members and then added one member. The partial receiver selected 132 of the original members, then 133. Both representations produced identical content and lease roots, active sets, signature counts, repair-request counts and complete proof-validation work:

These are concrete model field bytes, not measured network traffic. Each visited node contributes its actual key/representative width and fanout using the existing collection node response field layout; a test cross-checks actual codec output for its supported 32-byte keys. The model's inventory keys are 64 bytes. Inline paths send one body per verified node. Shared paths send one 32-byte reference per node, one explicit 32-byte request per missing body, and that body; both include one path-length byte per support. Common signed lease, root/member, membership/lease repair and transport framing bytes are excluded, as are latency and any cache-miss request scheduling effects.

Every phase admitted one signed lease. Even unchanged renewal verified that new signature, though it sent no membership support. Growth still checked 41,806 child summaries across the complete 677-node proofs and stored 257 new root-bound entries; the three new bodies are a byte-reuse result, not reduced validation or membership churn. Reconstructing the original independent Vec paths also makes no resident-memory or allocation saving claim. A partial receiver retains only selected leaf bodies, but branch representatives and sibling summaries still disclose information outside its responsibility.

Run cargo test --locked --offline -p triblespace-net --lib provider::directory_model -- --nocapture for the eleven focused tests. Lease expiry despite cache hits, root grafting, malformed miss bodies, eviction, and the original budget invariants are covered. There is no production wire, host, routing, persistence, or deployment change in this experiment.

Routing is process state

Peer::new(store, key, config) starts a production host immediately, as a long-running sync daemon needs. Peer::lazy(store, key, config) keeps the same store API but defers its host until the first absent-handle acquisition, explicit network fetch, or collection activation. Resident reads, snapshots, local writes, flush, and close start no thread or endpoint and build no serving/bearer index. Its snapshots are still exactly the backing store's resident snapshot type. Acquisition returns local observation errors rather than treating them as misses, and a host startup failure leaves resident operations usable. Startup is attempted once per peer; opening a new peer is an explicit retry. Activation logs startup failures without activating the collection, while acquisition returns the error.

A foreground H-only reader need not activate any collection. It can use a distinct ephemeral transport key, its pile daemon's key as its first contact (PeerConfig::daemon), and a zero provider-publication budget without borrowing its authorship signer's or the running daemon's endpoint identity. Network acquisition requires an enabled Tokio runtime at the calling async boundary; local-only operations need no runtime. Puts and landed walk values become visible in a fresh local observation without an automatic disk flush. Explicit close withdraws host snapshots and closes the backend at the final persistence boundary; manual flush remains available at an application's chosen durability boundary. Neither operation starts networking or authors WANT. Failure to obtain a fresh store snapshot is returned by try_refresh and withdraws the previous serving observation; existing frozen readers stay frozen.

Where peers come from

A host's peers come from its pile, never from configuration; there is no list of peers to give it. The keys a selected collection's content names, the signers of its records and its grant-chain keys, are its first peering candidates (tiers 1 and 2 under Neighbours), dialled by key through iroh's address discovery. TLS authenticates every connection, so each connection a host dials makes the key it dialled a verified DHT route, and the provider lookup at the collection's next draw has a contact: a host whose only route is the signer of a record it holds finds the collection's other providers, tier 3, through that signer. A connection a host accepts makes no route, by opening or by the requests it carries: anybody can dial with a fresh key, as a process that asks its pile's daemon with a key per call does.

A process whose pile names no peer, such as a foreground reader, has one first contact: the pile's sync daemon. PeerConfig::daemon names it by its key, the key the pile's configuration is written under; the daemon itself leaves it unset. The host notes that key as a candidate route and teaches its endpoint the addresses the daemon recorded (Where the sync daemon listens), which Peer::new reads at once and a lazy peer reads when its host starts; with no addresses recorded, iroh's discovery finds the daemon by key. From there the daemon's DHT replies lead on.

Endpoint addresses and relay URLs only provide iroh transport paths; they are not peering candidates, collection participants or collection rendezvous identities. DHT referrals and the peers a host reaches itself may become live routing candidates, but there is no synchronized PEER roster and no durable peer record in the current protocol. Liveness, connections, peerings, candidate orders, DHT buckets, and provider leases are operational soft state; restarting may forget them without losing semantic data, and each selected collection draws its candidate order afresh. The pile already holds what a restart needs to find its peers again: the keys its content names and, for a process beside the daemon, the daemon's addresses.

Restart contact experiment

routing/warm_start.rs is a test-only experiment, not enabled host persistence. It runs the real RoutingTable and IterativeLookup with synthetic lookup responses. Its one durable relation is a bounded set of previously authenticated endpoint identities, canonicalized with PATCH<32> and packed into ordinary RawBytes. A temporary-file write, sync, reopen and restore exercise the byte boundary. The test makes no new pile record, replicated roster, schema identity, JSON catalog, or provider advertisement. The host currently has no learned-peer persistence to reuse: a fresh route table holds at most the pile daemon's key and grows from the connections the pile's content opens; retired PEER records remain inert. The experiment's fixtures still seed the table with a fixed bootstrap identity, which stands for whichever first contact a restart has.

The maximum payload is 256 * K * 32 = 163,840 bytes, before any filesystem overhead. Parsing reads at most that bound plus one byte and validates the whole canonical endpoint set before admission. Restore uses ordinary candidate admission, so self, bucket bounds and existing local failure cooldowns still apply. Neither an old authenticated contact nor a content-addressed cache blob is current authentication, honest routing behavior, or collection/blob authority. Unqueried referrals and failed routes are not exported. Restored candidates do not re-export themselves as fresh evidence. No H, locator, token, lease deadline, or process-local monotonic timestamp appears in the packed relation.

Here, "authenticated" follows the table's existing Verified state, not a new serving-longevity claim. The production host promotes only peers it reached itself, a key it dialled or one that answered its request, and never an inbound caller such as an ephemeral foreground client. The experiment does not change that provenance.

The adverse fixtures distinguish three things: a dead configured bootstrap can make cold discovery impossible while surviving hints still find a route; stale closest-K hints can exclude a healthy configured seed; authenticated peers can answer successfully while retaining a closed, unhelpful topology. Existing failure cooldowns help a second lookup only in the dead-peer case and do not survive another restart. A test-only first wave reserves one configured DHT seed alongside at most ALPHA - 1 warm candidates, then admits the deferred closest-K set into the same lookup. This is DHT discovery, never direct-peer blob fallback. It adds at most K deferred identities and one admission phase, not durable state.

The first-wave fixture has one configured seed and instantaneous synthetic responses/failures. It does not settle multi-seed scheduling, the real three- and ten-second budgets, transport address rediscovery, snapshot replacement cadence, or handling a crash during replacement. Node churn and warm knowledge must be measured before selecting those policies. Endpoint IDs alone cannot help when no address discovery/direct route remains. Provider-directory deadlines and publication cursors are deliberately excluded: restarting put would renew a lease, and a saved publication cursor does not prove remote leases survived. An immutable cached answer needs its original expiry and provenance separately; reusing a serialized Mono or snapshot generation as freshness is invalid.

Run the focused tests with cargo test -p triblespace-net --lib warm_start -- --nocapture; add --ignored after -- for the 1,024-node, 64-target measurement. Reported requests and logical waves describe routing work, not elapsed network latency, successful provider acquisition, or live-network availability.

The first Stars run (2026-09-07 23:04 UTC, base d0db6da2, rustc 1.97.1, unoptimized net/lib tests) trained one client on 32 targets, saved 100 endpoints in 3,200 bytes, then independently restarted it for each of 64 other targets. The 1,024-node fixture gives remote peers established XOR-bucket views; only the measured client restarts. The totals were:

This is a 5.5% request reduction for the healthy closest-K fixture, not a universal cache speedup. A deliberately sparse 64-node line improved from 64 requests/64 waves to 20/7 with 1,568 saved bytes. Conversely, 20 dead cached contacts suppressed a healthy seed: cold reached the target in two requests, whereas warm closest-K failed after 20. The first-wave policy reached it in wave two but still spent 22 requests/eight waves completing the lookup. Twenty authenticated closed-view peers also suppressed discovery without any failed requests; the first-wave policy recovered that fixture in 21 requests/seven waves. These counterexamples make independent bootstrap opportunity a condition of future integration, not an optional cache optimization.

Live request scheduling

Inbound request streams wait for the process-wide permit while their connection holds them, within the per-connection bound of One connection per peer. Saturation resets only the excess stream and never closes the connection, whose recon/1 stream and other requests carry on.

The one connection per peer carries collection sync and bearer/DHT operations alike. Iroh's transport authentication binds each connection to its endpoint ID. There is no generic AUTH or SYNC_TEAM exchange: collection evidence goes only to a peer that passes READ(C) under the sender's evidence, and WRITE(C) only lets a writer that cannot read have its peering accepted so it can deliver. Exact bytes are gated only by the endpoint-bound mutual proof of H.

A refused peering is a frame, and a refused push is a reset of its walk/1 stream. Either ends only that request or push, not the connection, and cancels nothing unrelated. A malformed recon/1 frame closes the connection; a malformed walk/1 frame ends its push. A failed request stream invalidates the connection unless the failure stayed on that stream: the peer reset or stopped it, or the connection was already lost.

Concurrent callers for one peer share a single dial. Pending dial ownership is cancellation-safe: the final departing waiter removes a dial nobody completed, while the others keep it and can take over its cancelled initializer. This cleans up abandoned dials, not a new limit on simultaneous requests or detached transport handshakes.

For an active collection with a missing descriptor, the host owns one independent bearer fetch until its result reaches the bounded admission bridge. Later warmup rounds cannot spawn additional copies while that handoff is blocked. Removing the interest or dropping the host cancels the pending fetch; a completed miss or failed attempt permits a later retry.

That same owner performs best-effort metadata warmup for active collections, even before their descriptor is admitted and when they have no records. From a descriptor of at most 1 MiB it queries only UTF-8 name handles on tagged descriptor entities and typed definition handles linked by their resource-policy bindings. The warmup does not interpret the binding's authority or require it to be supported by this consumer. Unknown annotations, source descriptors, mapping parameters, proof paths, and record payloads are not traversed. This is acquisition, not READ or WRITE authority, and introduces no durable WANT.

At most four collection attempts run at once, in round-robin order over the active collections. Each warmup round examines at most 256 of them; ready resident-only attempts release their slot immediately instead of waiting for the next 30-second round, and a newly activated collection starts a round at once. Each owns a 30-second end-to-end deadline including admission-channel handoff; the descriptor read gets at most ten seconds. Extraction examines at most 256 typed rows and admits at most 64 distinct direct dependencies, with names first. Up to four children per owner run concurrently (sixteen child requests across these owners), each with a one-second end-to-end deadline. Every remote response is limited to 1 MiB before receiving its body; resident descriptor bytes are also size-checked before decoding. Existing exact receive-slot and global staging quotas still apply.

These are warmup limits, not schema validity rules or assertions of absence. An oversized descriptor or dependency, excess fanout, slow provider, or blocked handoff can leave metadata unprepared. No completion catalogue is retained; later rounds reobserve the store and retry. Ordinary foreground exact-H acquisition retains its existing larger byte allowance and deadline, and passive snapshots and resident-only stores do not acquire anything.

Foreground exact-H acquisition has one end-to-end deadline, normally ten seconds, including capability readiness, the cold dial of a first contact such as the pile's daemon, DHT lookup, and bearer GET. Its three-second routing window begins with the first authenticated replica response, not before that cold dial. This lets a healthy delayed first contact connect while a stalled secondary route cannot consume the whole remaining budget before GET. Background publication, collection provider lookups and descriptor warmups start their three-second lookup window immediately. A caller's shorter deadline still wins; progress does not reset either deadline, and no failed lookup starts an unbounded retry loop. When the routing window expires, issued requests still awaiting a reply count as failed learned routes. They cannot occupy every slot again on the next background attempt merely because cancellation preceded the dial deadline. Unissued candidates and partial authenticated responders are preserved; the daemon's route fails like any other. Each routing bucket also retains at most K local learned-route failures for sixty seconds, separately from its positive routes. Repeated third-party referrals cannot erase or extend that cooldown: it gates both seeds and reply candidates even when positive bucket retention differs from the lookup-local shortlist. Direct authenticated success clears a cooldown immediately; expiry permits a new probe. This is bounded, disposable liveness state, not authority or a permanent blacklist. This does not make a cold dial to a sole first contact that takes longer than three seconds fit the background window: if each attempt starts equally cold, background retries can still miss that endpoint. Foreground acquisition retains its separate end-to-end bound. Diagnostics distinguish a successful lookup with no provider hint from failure to reach a replica, a provider transport/protocol failure, or exhaustion of the end-to-end budget. A directory miss is an observation, not proof that H does not exist. Diagnostics do not log the bearer handle.

Lattice-aware sparse replication

The network does not force every replica to mirror every blob. Pushes of records carry foundations only; each host joins them into its own lattice:

COMMIT(C, a)       COMMIT(C, b)       replicated
       \             /
        MERGE(C, a, b, c)             this host's own node, never replicated

DERIVE(D, x, d)                       replicated: a foundation of D

A node receives the small semantic overlay and plans covers over its own resident merge results. Missing derived results are computed by the ordinary live ensure path, which may acquire exact missing dependencies and publishes missing DERIVE work only. maintain additionally writes this host's size-tiered MERGE work, which stays local. These operations take an explicit signing key; newly required derivation work needs target WRITE. Evidence and computation still converge by union; no central scheduler or query planner is required.

Durable WANT remains orthogonal operational policy. WantRequest::Blob(H) is the sole exact-content request. The reconciler performs KDF(H) discovery and the mutual bearer proof directly from its coherent store snapshot; no collection descriptor, repair overlay, or proof is involved. A successful landing satisfies the request, while a DHT miss or failed proof leaves it pending. Merge(C,a,b) and Derive(D,input) let one process state demand while a network or worker process fulfills it. WANT grants no READ, WRITE, or collection membership. Retained WANT records do keep their resident direct blob references alive through the normal record-root GC rules. GC never fetches missing references merely to retain them; dropping WANT records is an explicit rewrite/retention-policy choice.

Held blobs and replication policy

A collection replicated in full has a held set: a positive PATCH of resident handles, kept by the store itself (triblespace_core::collection::held) for the collections Peer::set_replication puts in Full mode. A host that replicates on demand or shallowly keeps none. Seeds are the blobs C's replicated records name -- the descriptor, each COMMIT's data and metadata archive, each DERIVE's output -- and the capability definitions named by the proofs C's authorization evidence keeps. That is the predicate AUTH applies (descriptor::validate_proof_evidence): a proof over C from one of C's policy roots, or over a resource whose immutable descriptor entity both routes to C and declares the proof's root, whether or not that resource is tracked as well. A valid proof irrelevant to C seeds nothing. A descriptor that arrives after its proof lets the proof be judged on arrival; one that is resident but cannot be read judges nothing. A MERGE result is never a seed, and neither is a blob no record names. Reachability is conservative: an aligned 32-byte word is a child when that exact H is resident. It issues no network request, creates no WANT and remembers no absent word.

Each blob is scanned once, when first reached, and its edges to the children resident at that moment enter one edge cache shared by every collection. A new seed, edge or membership then extends a held set without reading bytes again; an already scanned blob shared with another collection joins it for free. A seed whose bytes arrive after its record is scanned on arrival. The late child -- resident only after its parent was scanned -- is caught two ways:

• it is itself a seed of some collection; or
• a Full neighbour holds it in C: its references push notes it held in C here too, however it arrived, and the receiver fetches it first when it is not resident. A note is routing evidence kept in memory: a reopened store forgets it unless one of C's records reaches H, and a note of a blob that is not resident when the next snapshot is taken is dropped.

Otherwise it stays out of held(C) until the index resets or the store is reopened, though it stays readable by its hash. That is a stated limitation: there is no periodic rescan. Between two Full neighbours, a late child that one side holds through a seed or a note reaches the other side's held set by its next references push.

Every snapshot carries the held sets as they stood when it was taken, and they never change afterwards. Taking a snapshot first extends the sets by the closures of what arrived since the previous one -- every seed of a new record (one another record already named included), seeds whose bytes arrived, notes from references pushes -- computed against that snapshot. That is the publication barrier: an observation is never handed out ahead of its new records' closures. A held set is closed under the edge cache for readable children: a blob joins only once every child the cache names for it has joined. A blob that cannot be read, absent or resident and failing to read, is unknown on every path, and nothing above it joins. When a collection is first tracked, its existing closure is computed by the next snapshot, which reads every blob it reaches once; nothing here starts a thread. A collection whose records the store cannot select stays unsettled -- absent from the held sets, never published short -- and is selected again at the next observation. A snapshot that lost a blob its predecessor had resets the index, and every collection is recomputed as if newly tracked. Short-lived readers of the same pile track nothing.

Two Full neighbours push each other their held sets like any other tree (Walk streams): matching prefix commitments prune independently held overlap, and unchanged agreed sets send an empty-prefix empty delta, DONE and LANDED.

This set is partial and conservative. A readable aligned word is not a typed semantic reference; an absent word says nothing about global residency. For encrypted blobs, H identifies the stored ciphertext; neither READ(C), this set, nor the exact-H protocol supplies an application's decryption keys.

Peer::set_replication(mode, collections) selects acquisition independently of activation and authority. pile net sync --replication applies one mode to every selected collection:

These reconciler modes are separate from the bounded metadata warmup of active descriptors described above. Demand still does not hydrate record payloads without an exact WANT.

Naming a collection for replication does not grant READ or WRITE and does not activate it. Structurally valid but WRITE-inert records can name direct roots; acquiring their bytes does not admit those records. Full mode never sends aligned payload words to the DHT to choose speculative requests. Local aligned-word scanning builds the positive held set only, and a references push brings only handles a Full neighbour holds, fetched directly from that neighbour. Unavailable neighbours and failed fetches prevent a promise that Full has reached complete closure, and an equal held root says only that two neighbours hold the same set.

Exact WANTs and direct roots use the existing KDF(H) discovery, mutual bearer proof and final hash verification. A shared eight-wide fetch window serves one eligible class's finite round under its original deadline. Ready verified bodies land individually, without a per-body flush or serving rebuild; Peer::reconcile() publishes one snapshot after the completed batch. Cancellation drops owned network futures, leaving already landed bytes for the next refresh. Unstarted work is not falsely fulfilled, and failed exact requests retain bounded retry backoff.

Missing work is derived from known required handles and the current readable snapshot, not a retained mirror of durable answers. A raw physical occurrence alone is insufficient: ordinary reads retain corruption detection and valid duplicate fallback. Explicit close remains the persistence boundary. None of these scheduling bounds is a measured throughput or end-to-end completion guarantee.

Wire surface

The ALPN is /triblespace/pile-sync/32. Generation 32 adds explicit one-way Prefix offer roles and HAVE/eager typed bodies; generation 31 and earlier are refused at the handshake. Generation 31 added the explicit OPEN algorithm discriminator and cached whole-component RIBLT profile. Generation 27 replaced the bearer locator, directory token and exact-GET proofs with one-block constructions, so a generation-26 peer cannot connect at all; generation 28 exports foundations only, so collection sync never carries a MERGE; generation 29 replaced the gossip wake plane and the repair/0 session with one connection per peer and its recon/1 stream; generation 30 moves collection sync off recon/1 onto walk/1 pushes, so a generation-29 peer, which announces and pulls on recon/1, is refused at the handshake like every older one. Within the ALPN, each stream's first byte selects its layout (One connection per peer): peerings and the grant exchange run on recon/1, and every collection tree travels on a walk/1 stream. Bytes that opened earlier layouts are not accepted: 0x0E was the collection repair session that pushes on walk/1 replaced, 0x0D its record/AUTH-only predecessor, and 0x06, 0x07 and 0x0C the DHT operations now under dht/1. A stream that opens with one of them is reset as an unknown tag. Mixed-version collection sync is not supported: deploy a cohort together.

FIND_VALUE answers in one reply what FIND_NODE and PROVIDER_GET (0x0C and 0x07) answered in two rounds. Every recon/1 frame is listed in One connection per peer, and every walk/1 frame in Walk streams.

There is deliberately no store manifest, global inventory authorization, push-broadcast record, receipt RPC, remote mutable head, or unpublish operation.

The CLI names local policy only. The collections come from the pile's selection (Choosing what to sync) and the peers from the pile's content (Where peers come from):

trible pile net sync DATA.pile \
    [--key EXISTING_SELF_KEY] \
    [--replication demand|shallow|full] [--bind IP:PORT] \
    [--provider-publication-budget ATTEMPTS] \
    [--health] [--health-collection HANDLE] \
    [--telemetry-collection HANDLE] [--telemetry-worker NAME] \
    [--duration SECS] [--quiescent-for SECS]

There is no --peers or --collection flag, and no direction flag: which way a collection flows between two keys follows the send flags of their peering, which follow admission. At startup the daemon prints its node id, its bound sockets and the number of selected collections, and prints that number again whenever the selection changes.

--bind binds the endpoint to exactly that local socket instead of iroh's default sockets. At startup the daemon prints bound: <ip:port>[ <ip:port>...] on stderr, the sockets it actually bound, and prints the line again whenever they change. It records the same sockets in the pile's configuration (Where the sync daemon listens) on its first pass and after every change, so a process that opens the pile and names the daemon's key reaches it with nothing copied by hand. trible pile net identity --key KEY prints node: <endpoint id>; there is no ticket to print, since nothing takes one. After each pass the daemon prints one plain line on stderr per peer with which at least one collection's last push was confirmed or last receive landed whole since the previous pass: reconciled with peer <endpoint id hex>: <n> collections.

The long-running daemon uses one existing durable key for its authenticated endpoint, the configuration that holds its sync selection and its addresses, health reports and telemetry. Key resolution is --key, then TRIBLESPACE_KEY, then self.key beside the pile's lexical path; select, unselect and selection resolve the key the same way. There is no independently configured network or reporting key. Maintenance uses that same local key for its work and telemetry; worker labels distinguish processes, not identities. This CLI convention does not change the separate ephemeral H-only foreground-reader use described above or grant any authority.

Whatever a node selects, it may publish provider leases for and serve resident exact blobs under bearer handle H, and may service durable Blob(H) WANTs through KDF(H). The replication mode does not participate in collection identity or change which evidence is semantically valid.

Before opening the writable pile or starting its peer, pile net sync registers Unix SIGINT and SIGTERM handlers; other platforms retain Ctrl-C handling. Stop drops in-flight awaited reconciliation or the idle wait, then withdraws the serving observation and explicitly closes the backend. It does not change request deadlines or the application's clocks. Synchronous work and close are not preemptible, so this cooperative boundary is not a shutdown-time bound.

Convergence and failure model

• Concatenation, local insertion, and landing from pushes perform set union; held sets remain replaceable local observations.
• Duplicate records collapse by their complete canonical value; fixed-width indexes and the wire use a full-width BLAKE3 fingerprint of that value. Native capability proofs continue to collapse by their cryptographic identity.
• A lost push does not consume the only opportunity: in every interval of at most a minute, each collection's timer pushes its trees again to each neighbour it sends to, and a push that did not land forgets its confirmed agreement, so the next delta starts without that local base and matching prefix commitments prune what already landed. A neighbour lost to a failure is replaced by the next candidate in the collection's order.
• An invalid record, proof, subtree commitment, or blob fails its push or fetch and cannot retract previously accepted evidence; a malformed recon/1 frame closes its connection, and a malformed walk/1 frame ends its push.
• Missing selected output/dependency blobs leave that realization unavailable; complete finer members remain usable. Missing witness records leave support unknown, not empty, and do not authorize cross-collection disclosure.
• A DHT miss says only that no live provider was found; it says nothing about whether H or its collection exists.
• Concurrent writers and offline replicas reconverge without preserving pile byte order.

The result is two orthogonal elemental loops: per-collection timers plus admission-gated pushes say what changed in a collection and, between two Full neighbours, which blobs a peer holds in it, while KDF(H) discovery plus mutual bearer proof retrieves only the immutable bytes the local lattice resolver decides to use.

Observing replica health

Peer::health() and NetSender::health() expose a bounded immutable sample of work the host already performs. Sampling does not run a probe or advance the host's observation clock. For each collection and peer the sample keeps the ends of both directions, the pushes of this side's trees and the pushes of the peer's trees received here (when the last one ended, whether it ended well, and how many did either way), with the roots of the records and authorization trees the peer confirmed and the roots that landed whole here. Semantic health compares record and AUTH evidence, not blob cache equality: Demand and Full peers can legitimately cache different bytes, held sets are not part of the record root, and the references tree is in neither set of roots. Held-set churn neither creates semantic stalls nor extends their progress grace. HealthSnapshot::pair reads a pair as Current when the peer confirmed the records and evidence this side serves now, where this side sends, and the last push from the peer landed whole, where it receives; Behind names the direction that is not; and a stale sample, a changed local frontier, a withdrawn serving snapshot, an ended peering or no end within the age limit leaves it Unknown. Receiving records is progress evidence, not proof they have already become admitted. HealthSnapshot::peerings lists every peering on an open connection, and HealthSnapshot::available the collections named by proofs naming this node that wait in memory for their descriptor.

The long-running CLI can publish these observations as native facts using pile net sync --health. Reports use the same key as the endpoint. An explicit --health-collection <HANDLE> also enables reporting into an existing admitted source collection; otherwise the node's private swarm-health collection is used. The health collection is deliberately not activated for sync, so a broken network cannot prevent the local reader from seeing its own warning. A new report is published every minute with created_at only: freshness is reader policy, not a producer-asserted lifetime. pile net health --max-age <SECONDS> accepts a maximum sample age (default 180 seconds, also configurable through TRIBLESPACE_HEALTH_MAX_AGE_SECS). Readers use the creation interval's upper bound plus that duration and ignore any legacy expires_at annotations; future-created reports remain unknown, and zero accepts no arrived sample as fresh. Conditions use stable episode identities until their state or attached quantitative evidence changes; a reader can acknowledge an unchanged alert once without acknowledging each heartbeat. When the reader's age limit is reached, the report itself supplies an attention event even when the daemon stops appending entirely.

These reports distinguish unknown, progressing, matching, and stalled work, with a ten-minute repair/publication grace period. A stale host loop is detected separately from a fresh reporter. An idle DHT awaiting its ordinary eight-hour renewal is not a failed availability probe. Publication acknowledgement says only that a provider advertisement was accepted, not that a remote blob fetch works. Neither the observed participant set nor a pairwise matching frontier proves whole-swarm convergence. pile net health reads the local observations; Orient can consume the same maintained facts and presentation ledger.

Colony work telemetry

pile net dashboard reads explicit resident telemetry collections and the private local health fallback. Terminal and embedded GORBIE views share that observation; neither scans the blob/native-record inventory, probes links, acquires missing bodies, maintains a target nor appends a descriptor. GUI refresh retains one sampler with explicit stop/join/close ownership. A read failure remains visible; it is not replaced with an empty healthy colony. Reader failures cross into the display as fixed categories, never backend error text or source chains: those may contain a full bearer blob handle. Control stripping and truncation alone do not redact a read capability.

The sampler retains its final display projection while every contributing collection observation remains dependency-current in a newly refreshed pile snapshot. It then ages a copy instead of querying historical report headers again. This is not a report catalogue or an alternative index: changed inputs run the same queries again. Wall time remains a separate dependency. Moving before the original projection time or reaching a previously future worker sample triggers a query, because either may enable a rate omitted earlier. Rate expiry also clears its prior-sample metadata; the original result stays unchanged so an in-range clock correction can restore a valid interval.

Partial or failed inputs never qualify for reuse and keep being retried. This first cut is deliberately all-source: an unreadable selected health collection also prevents telemetry reuse. A missing key disables the optional health source instead and does not have that effect. Scoped dependency checks, snapshot refresh, frame aging and rendering still cost work on unchanged ticks; no idle-CPU or whole-pile performance claim follows from skipping the queries.

Aggregate telemetry is ordinary relational data, not an external metrics database. triblespace_net::telemetry describes a subject (endpoint, operator worker label, role, optional stage/target/peer) and timestamped samples linked to it. Use an opaque subject ID; readers never recompute or validate it. For example, selection, certification, mapping and joining are separate stage subjects, so a slow unchanged-target pass cannot be mislabelled a slow mapping. Actual runtime-selected backend is a sample annotation, not a claim inferred from compiled features. Configured parallelism, active operations, process CPU and operation wall time are different measurements. parallel_compiled explicitly records whether the relevant parallel path is in this binary; compiled_backend records available build capabilities. A lazy pool with one observed thread cannot tell disabled code from unused code. Keep successful passes that publish no equations: their nonzero elapsed time can expose the cost of proving that there is no new work. They are not the same as skipped polls, failed observations, or zero computation.

Each sample carries a process session, explicit wall observation time and monotonic nanoseconds since that session began. Producers use entity! at the existing work boundary, then the normal signed collection commit:

use triblespace_net::{health_record as h, telemetry as t};

let subject = entity! {
    h::attrs::endpoint: node,
    t::attrs::worker: "rollups",
    t::attrs::role: "maintenance",
    t::attrs::stage: "mapping",
    h::attrs::collection: target,
};
let sample = entity! {
    metadata::tag: &t::KIND_SAMPLE,
    t::attrs::subject*: subject,
    h::attrs::session: &session,
    metadata::created_at: (now, now).try_to_inline()?,
    t::attrs::elapsed_ns: session_started.elapsed().as_nanos(),
    t::attrs::active: 1_u64,
    t::attrs::completed: completed_in_this_session,
    t::attrs::backend: "cuda",
};
store.commit(telemetry_collection, reporting_key, sample)?;

This example is a producer seam, not automatic instrumentation. Do not publish zero-filled counters after a failed observation. Counters describe completed work in that session, not distinct blob coverage; queued work describes the observed selection, not all unknown descendants. Received bytes count verified payload successfully landed, while sent bytes count payload successfully sent; neither includes transport overhead. A peer-scoped link sample reports an actually observed path and RTT, not a DHT candidate or an invented link speed. Process CPU nanoseconds are reported once in a process scope. work_ns sums completed operation wall durations; concurrent durations can overlap, so its rate is not CPU cores. Never duplicate process CPU across target rows and add them together.

The reader keeps two recent header witnesses per subject and queries metrics where needed. Unknown facts coexist; multiple observed values remain visible without invalidating the whole collection. A rate needs two fresh, distinct samples in the same session, increasing monotonic time and nondecreasing counters. Restart, reset, clock skew, missing or ambiguous data yields an unknown rate, not zero. The wall clock establishes sample age; it is not the rate denominator. Reported throughput is an interval average, not physical bandwidth capacity or proof that background hydration is keeping up.

Publication uses an explicit operator-selected collection and authority. The dashboard's --telemetry-collection may name one shared authorized source or separate node sources. Merely naming an endpoint in a sample is not a proof that endpoint authored it; trust comes from the selected collection's admitted writers. This feature issues no grant, activates no shared sync selection and changes no service. The private local health stream stays independent so broken telemetry replication cannot hide its own warning. Coverage is only the observed participants, never an implicit roster of the whole colony.

Sampling is aggregate and deliberately coarser than packet/operation tracing. The existing facade telemetry feature remains the explicit per-span profiling sink. Neither facility invents a retention policy for append-only evidence. No raw H bearer capability, payload, locator-to-H inventory or bearer proof belongs in these colony observations.

Sync producer

pile net sync opts in with --telemetry-collection <H>, optionally --telemetry-worker <NAME>. The node's existing key signs the samples and its public key identifies their endpoint. The destination must already be a resident SimpleArchive source collection admitting that writer. A derived SimpleArchive is not a source: its reader ignores root COMMITs. Reporting never creates a descriptor, key, grant, derived index or replication selection. To replicate these samples the operator separately selects their collection with trible pile net select.

The producer attempts an ordinary signed append every sixty seconds at the existing sync-loop boundary. Five subjects distinguish the measurements:

• hydration: successful verified reconciler puts and their payload bytes; one landing shared by a WANT and a selected root counts once. Blobs a references push lands are not reconciler puts and are not counted. Local hits and failed puts add no received bytes. queued counts distinct unreadable known handles from exact WANTs and selected direct roots at the end of that tick's acquisition window. It excludes unknown descendants and is not a global blob inventory. A failed exact-set observation leaves it absent.
• serve: actual accepted inbound GET exchanges in flight, successfully sent payloads/bytes and interrupted or failed exchanges. A payload counts only after its write and stream shutdown succeed, not merely after lookup. This is not a remote landing or persistence acknowledgement.
• repair: no active count yet. Host health records the ends of pushes and receives, not the ones in flight, so the measurement stays absent rather than estimated.
• publication: active and completed provider-advertisement operations, separate from body transfer; acknowledgement is not blob availability.
• process: one cumulative user-plus-system CPU observation across all process threads, absent if unsupported or unavailable. Core's compiled parallel-path bit is separate from runtime pool size or actual CPU effort.

These are boundary observations, not a new timer task inside an awaited hydration tick. A long tick delays sampling; only returned ticks contribute their landing counters and completed wall duration. Direct foreground reader acquisitions are outside the hydration subject. Unmeasured hydration in-flight counts, transport path/RTT and internal stage timings stay absent. Stale host health contributes no invented activity zeros. work_ns sums completed or interrupted GET durations in the serving scope and returned-tick durations in hydration, never CPU time. Reporting errors discard backend cause chains and leave older samples to age. No reporting append flushes the pile; the owner's existing close remains its persistence boundary. Same-pile writes remain real changes: scoped disjoint reads ignore them, while broad name lookup can still observe the new records.

Maintenance producer

pile collection maintain and maintain-all can opt in with --telemetry-collection HANDLE --telemetry-worker LABEL. The existing maintenance key signs samples and its public key identifies the local endpoint, matching the daemon's durable identity. Neither the endpoint nor the telemetry signer has a separate override. The collection must already exist as a SimpleArchive source and admit that writer; a derived view ignores root COMMITs and is rejected. This option does not create keys, descriptors, grants, maintenance targets or sync selections. After configuration is validated, publication errors use finite, payload-free categories and do not turn successful maintenance into a failure. Choose distinct worker labels for simultaneously running processes on one endpoint; restarting the same worker keeps its scope identity but starts a new session, so a rate never bridges that restart.

Four scopes keep distinct quantities separate: process CPU, maintenance hops, passes, and successful passes with no merge/derive publication calls. Hop configuration reports outer-loop concurrency, not the size of a lazy Rayon pool; Core's compiled parallel-path capability is reported separately. The merge/derive counters increment after a local equation insertion succeeds. They include idempotent successful calls and multiple witness equations for one output. They are not counts of new physical records, unique outputs, kernel invocations, or changes in the collection census. Successful publications remain counted even if a later part of the pass fails.

Sampling uses existing loop/hop boundaries, with a shared attempt limit of 60 seconds by default (--telemetry-interval-secs) and one final close sample. There is no detached heartbeat or extra observer thread. A long synchronous hop can therefore leave a stale report; freshness must not be manufactured. Known merge/derive backlog and separate selection/certification/mapping/join timings are left absent until those operations provide measured seams.

Appending telemetry never advances or filters the maintenance baseline. Disjoint exact-collection interests ignore the append. Broad name/all-record interests may wake once after an emission; all boundary attempts, including failed publications, share the rate limit. Do not hide such a wake by moving the baseline across the write: concurrent source changes must remain visible. An unchanged-file poll which never enters a pass must not increment pass or no-publication counters, and must still reach the telemetry cadence check.

Schema anchors were minted with trible genid on 2026-09-16; their exact values and field meanings are pinned in triblespace-net/src/telemetry.rs.

Directory representation experiment

The live receiver-local ProviderDirectory uses four BTree indexes: membership values, member counts by exact locator, expiry order, and XOR responsibility then rank order. provider/patch_directory.rs is a test-only alternative with two PATCHes:

• !(locator XOR local_endpoint) | !rank, segmented 32 | 32, owns the provider/deadline/token value. Segment counts answer membership and exact-locator cardinalities; the first ordered key identifies the farthest responsibility's largest-ranked member, and a locator's first suffix its own.
• deadline_ns | locator | provider orders expiry. Keeping the original locator/provider order here preserves tie-breaking under the bounded prune budget, not merely the eventual set of live results.

No wire, lease, trust, publication, persistence or live-directory policy changes. Differential tests compare every retained value, deadline, capacity decision, exact-key result and farthest member against the existing implementation across 24,000 deterministic mixed operations and targeted expiry/fanout boundaries.

Stars measurements on 2026-09-07 used exact prototype 512513bf, rustc 1.98.0, and fresh processes for each representation. Each table cell is PATCH / BTree; times are seconds. GET traverses every locator once; insert and renewal visit every locator/provider pair. RSS is the post-insertion minus pre-insertion process reading, excluding the already-built deterministic input corpus.

These are single samples, not confidence intervals or full release/network measurements. Only triblespace-core and triblespace-net received test-profile overrides opt-level=3, debug-assertions=false, overflow-checks=false; dependencies otherwise retained the same test profile. Both compared modes used those identical settings, eight build jobs, debug information disabled, incremental off, and no RUSTFLAGS. Debug-only timings were much worse for PATCH because its invariant audits are intentionally expensive; they are not a sound basis for the production comparison. Both profiles passed the differential tests.

The memory saving is substantial, but immutable membership replacement and deadline reindexing make renewal slower, and multi-provider insertion can also lose. This prototype is retained for evaluation, not promoted to the live directory. A proposed Cell lease-value shortcut was rejected before edits: shared PATCH leaves require Send + Sync for cross-thread ownership. Adding an unsafe ownership promise or per-leaf synchronization just to optimize these already-fast local operations would add a new tradeoff, not finish this one.

Reproduce with cargo test --locked --offline -p triblespace-net --lib --features sim provider::patch_directory; the ignored patch_directory_scale_probe accepts TRIBLESPACE_DIRECTORY_REPRESENTATION (patch or btree), TRIBLESPACE_DIRECTORY_KEYS, and TRIBLESPACE_DIRECTORY_PROVIDERS. Run each mode in its own process and state the profile settings with any result. Corpus shape, allocator state, CPU profile, and fanout matter; this is not a measurement of DHT announcement throughput.

Resource Capability Proofs

TribleSpace authorization uses prefix-signed paths. A proof names one opaque resource and carries one trust root's authority through one or more delegates:

magic | resource | root | (capability handle | delegate | signature)+

The 32-byte magic is the handle of this grammar's SimpleArchive description. It is also the native Pile record kind: there is one format identity, not an outer kind wrapping a separately tagged proof. An incompatible grammar gets a new description and magic rather than a version byte inside the decoder. resource is an uninterpreted 32-byte identity. Collections use their descriptor handle, but the capability kernel does not require the resource to be a collection or even a blob. The root and delegates are canonical, non-weak Ed25519 public keys; alternate encodings of one principal cannot count as different quorum roots.

Every signature covers the complete preceding proof prefix through its own delegate, including the magic, resource, root, capability handles, and earlier signatures. Edges cannot be reordered, grafted onto another path, or moved to another resource without invalidating a signature. The signature authenticates the definition handle; interpreting its actions requires the definition blob.

Every exact prefix ending after a signature is itself a complete proof. An intermediate subject's authority depends only on that prefix: a later invalid signature, unavailable definition, or invalid attenuation cannot revoke an earlier valid grant. This does not make malformed framing acceptable, and a store's signature-checking ingress can reject an invalid whole proof before retaining any of it. Publishing the exact shorter prefix remains possible.

Canonical wire value

The proof header is 96 bytes:

Every edge is 128 bytes:

There is no count, mode, flag, validity interval, parent pointer, or alternate field order in the proof body. Its exact length determines the nonzero edge count, bounded at 255. Decoding checks the magic, length, and canonical keys; signature verification is a separate strict check. The proof ID is BLAKE3 over the exact unpadded body.

The current grammar is pile-auth-proof-v5, with descriptor handle 9E00F6EB8D63E5EB1A3ECFA284118F157B2F08CBDC95ECCEB8C2392AD794069D. Its anchor C7116B04EE6DA4BADFDE77D79692AEFF was minted with trible genid on 2026-09-14. The grammar change does not change existing collection descriptors or their canonical READ/WRITE capability definitions. Old proof signatures cannot be converted to the new wire value without signing the new bytes.

Invocation and delegation are independent action sets

A capability definition is an immutable SimpleArchive. Its ordinary facts describe two finite sets of opaque action IDs:

I(P) = actions named by capability_action
D(P) = actions named by capability_delegate_action

I permits invocation. D permits granting invocation or onward delegation. There is no implicit rule that being allowed to invoke an action permits delegating it. A child definition Q may follow P exactly when:

I(Q) ∪ D(Q) ⊆ D(P)

The handles need not be equal. For example, I(P) = {READ, WRITE} and D(P) = {READ} lets the subject read and write, but delegate only READ. A delegate-only grant, I(P) = {} and D(P) = {READ}, can confer READ without allowing its own subject to read. A child with I(Q) = {READ} and D(Q) = {} can read but cannot pass that grant onward. This is not an intersection of invocation bits along the path.

The existing capability_action attribute and standard one-action READ/WRITE blobs keep their exact identities. The new capability_delegate_action anchor is 628AB41154C8BAE6F985591B58487374, minted with trible genid on 2026-09-14. Unknown annotations are ignored by the generic action interpreter. They may be meaningful to the consumer of a particular action.

The construction and interpretation boundaries are deliberately separate:

let proof = CapabilityProof::new(resource, &root, grant_handle, intermediary);
let proof = proof.delegate(&intermediary_key, narrower_handle, recipient)?;
proof.verify_signatures()?; // bytes only; no definition acquisition
proof.verify(
    &snapshot,
    root.verifying_key(),
    recipient,
    CapabilityRequest::new(resource, ACTION_READ),
)?;

new signs a root grant. delegate checks the signing key is the current leaf and signs an extension; it does not fetch definitions or promise that the child is authorized. verify interprets the prefix to the requested subject through the supplied reader. validate_delegation checks the complete path when that is the operation required. Definition reads are shared by handle within one quorum operation, rather than loading a catalog of the store.

Policies select roots per action

A request is (resource, action), not (resource, grant handle). Resource descriptors still use their existing resource_policy links and exact capability_handle bindings. A consumer reads the bound definition's invocation actions to select the applicable policy. A compound bound definition can apply the same policy to several actions; different grants implementing an action do not need entries of their own in the resource descriptor.

Consequently, changing a grant's descriptor does not change the resource's identity. A root configured only for READ cannot grant effective WRITE by signing a definition that mentions both actions: a WRITE request uses the WRITE policy's roots, not the grant's choice of root.

For roots {r1, r2, ...} and threshold t, a subject needs valid prefixes from at least t distinct configured roots for the requested action. Two paths from one root count once. A configured root supplies its own share, not an exemption from a multi-root threshold. Supported alternative policies are evaluated independently; shares from different alternatives cannot be pooled.

Sibling paths cannot lend delegation support to each other. To delegate a quorum's authority onward, a subject extends the independently rooted paths whose definitions allow that delegation. No second delegation threshold or fixed-point authority forest is needed. Arrival order and wall-clock time do not affect generic authority over the same proof and definition set.

Storage and interpretation

CapabilityProofStore is a grow-only set of canonical proof values, with insertion, deterministic enumeration, and exact lookup by proof ID. Presence is evidence, not authority. The caller supplies policy roots, subject, request, and a reader for the referenced capability definitions.

Native Pile and MemoryRepo insertion verify proof signatures. Trusted native replay and CapabilityProof::from_bytes check structure without repeating cryptography. The generic authorization kernel still verifies the signatures of the prefixes it consumes; structural decoding alone is not its trust boundary. Pile indexes retain shared owning anybytes::View values.

The generic Pile framing magic and block span precede the canonical proof. Framing and zero padding are outside its signatures and hash. A zero padding edge cannot be a valid edge because its delegate key would be weak. Minimal framing is required; a one-edge proof plus framing occupies one 256-byte block. The network sends the exact unpadded proof.

Conservative collection retains resident capability-definition blobs and the record-kind description. It does not interpret the opaque resource as a blob reference or fetch missing definitions. A proof can therefore be retained and transported before its definitions are resident. It confers no interpreted action through an unavailable definition.

Collections consume authority

Collections interpret the standard action IDs as:

READ(C)  = request(resource C, ACTION_READ)
WRITE(C) = request(resource C, ACTION_WRITE)

WRITE admission decides which signed COMMITs and DERIVEs contribute membership. A DERIVE needs WRITE on its target; source WRITE alone is insufficient. A MERGE needs no WRITE: a store believes only the MERGEs its own host key signed, and any other key's MERGE folds into nothing, whatever that key may write. A MERGE signs the payload handles of its inputs and result, a DERIVE the locator of its source foundation and its output; neither names an input record. Readers admit foundation producers, then follow the host's own merges without repeating ancestor authorization or loading ancestral data, metadata, or proof definitions. Descriptor lineage and the producing records are still required to establish Support; selected output blobs and their encoding dependencies are required to read the value.

The signature identifies a producer, not a mathematical proof. An authorized incorrect producer can still endorse a false computation, and a bug in the host's own merge is believed as signed. Readers do not replay that computation. Because a store folds only its host's merges, another key cannot inflate what a node stands for by signing an alternative decomposition of it. Current collection-record signatures are checked at foreign ingress or explicit audit, not again by ordinary trusted-local attachment, so that holds for records whose signatures were checked when they entered the store.

Local publication can retain an inactive record before a suitable proof arrives. Later proof or definition residency can activate it. Generic WRITE membership does not expire as the clock advances. READ separately controls which authenticated peers may receive a collection's records and proofs; it is not a precondition for interpreting local data. Exact blob retrieval remains orthogonal: knowledge of a blob handle is the read capability for those bytes.

grant_collection_read and grant_collection_write issue the unchanged invoke-only standard definitions. grant_collection_capability accepts an already stored definition, which may combine invocation and delegation. The helpers inspect the resource policy and insert one root's proof; open policies need no proof, and each root contributes independently to a quorum. collection_read_audience enumerates the finite authorized audience, including valid intermediate subjects, or returns Open when no finite list suffices.

Application restrictions and resource discovery

Generic proofs and collection admission have no clock. An action such as Secrets key delivery may interpret additional facts in a capability definition, including a deadline. The quorum APIs ending in _if expose each validated subject prefix to a predicate before counting that root's share. The consumer must inspect restrictions in every ancestor definition on that prefix; changing the child handle cannot erase an ancestor's restriction. A rejected later prefix does not cancel a previously acceptable earlier prefix.

Key-delivery expiry can stop future delivery, not revoke possession of a key already delivered. It does not expire COMMIT membership or collection READ. Secrets-specific delivery grants need not be listed in collection descriptors: a separate immutable resource may have its own delivery policy.

Ordinary collection facts can reference such a resource with resource_handle; the immutable resource can declare its repair audience with resource_collection. This names the collection whose READ-authorized peers may exchange the resource's proofs; it does not inherit that collection's authority. Their anchors, minted with trible genid on 2026-09-14, are respectively 5059416BAF824B364FF52F56A0D1CF9F and 62E951EC3ABB7F9B7F123BB7DE2F9F99. The resource's own policy roots still decide its actions.

Repair and discovery

For proofs naming exact resource C, collection AUTH repair uses C's declared policy roots. It no longer filters by grant-handle equality: different definitions can delegate the same action. This byte-local relevance check does not interpret the definitions or confer READ. Peering admission and push admission separately interpret the requested READ action using resident definitions, the pinned proof set and the credentials a peer presented, before any record or proof is sent.

Pushes transfer records and proofs, not capability-definition blobs, payload closure, or WANT bookkeeping. Definition bytes travel by hash over the bearer blob protocol: for a proof received through the grant exchange or as a peer's credential, a node fetches the definitions it names from its sender, then from the proof's root and delegated keys.

Subordinate-resource transport also routes proofs naming R when R's immutable descriptor contains resource_collection: C and resource_policy on the same entity. It checks R's policy roots and signatures, while the push remains gated by READ(C). No mutable fact in C can create or revoke the route. A missing R descriptor is fetched by hash from the peer that pushed the proof; if it does not route the proof to C, the proof stays deferred without blocking C's records push, and the authorization push does not land, so the peer's next push of that tree carries the proof again. Its repair PATCH prefix names the routing audience C, not necessarily the proof's signed resource R.

Explicit protocol cutover

AUTH v5 and witness-bound equations use /triblespace/pile-sync/26; older protocol peers are not a compatibility bridge. Before switching a replica, provision the selected descriptor and capability-definition blobs together with the required fresh proofs through an explicitly authorized handoff. Otherwise both endpoints may need READ evidence which record repair cannot yet disclose.

Reissuing a grant is a new signature, not translating an old proof. Preserve the intended resources, roots, recipients, actions, delegation rights, and quorums. Generic grant-read/grant-write issue invoke-only, non-expiring collection grants; they are not a migration for bounded Secrets delivery or other application restrictions. Audit old prefixes, including expired ones, before deciding what the appropriate owners may reissue. Neither a descriptor being selected for sync nor a blob being resident authorizes a broader grant.

The boundaries remain independent: proof presence is not authority; routing, pushes, and DHT presence are not authority; WANT records durable local demand; and blob residency is not semantic validity.

Importing Other Data Formats

Import pipelines let you bring external datasets into a TribleSpace collection without hand-writing encodings or entity identifiers every time. This chapter introduces the import namespace, explains how the JSON importers map foreign fields onto attributes, and outlines how you can extend the same patterns to new formats.

Import Namespace Overview

The triblespace_core::import module collects conversion helpers that translate structured documents into raw tribles. Today the namespace ships with two deterministic JSON importers and an N-Triples (RDF) importer:

• JsonObjectImporter hashes sorted and deduplicated NIL || attribute || value rows using the same protocol as entity!, so identical fact sets reproduce the same entities. It accepts a top-level JSON object (or a top-level array of objects). Construct it with a blob sink (e.g., a collection's underlying store or a MemoryBlobStore) and an optional 32-byte namespace when independently imported corpora must not share identities. Each import_* call returns a Fragment that exports the root entity id(s) and contains the emitted facts.
• JsonTreeImporter preserves the full JSON structure and ordering by emitting explicit node and entry entities (a JSON AST). It derives content-addressed identifiers from the JSON values themselves so identical subtrees deduplicate across overlapping imports. Unlike the object importer it can represent arbitrary JSON roots, including primitives. Each import_* call returns a rooted Fragment for the imported JSON value.
• ntriples::ingest_ntriples (and the file-backed ingest_ntriples_file wrapper) reads the line-oriented N-Triples serialization of an RDF graph and emits one trible per statement. URIs become stable entity ids via the import::rdf_uri attribute; predicate URIs become attribute ids via Attribute::<S>::from(entity!{ metadata::iri:, metadata::value_encoding: }) — the IRI is the canonical identifier; literal values map into the appropriate native InlineEncoding based on their XSD datatype.

JsonObjectImporter uses a fixed mapping for JSON primitives:

• strings → Handle<UTF8String>
• numbers → F64
• booleans → Boolean

Arrays are treated as multi-valued fields; every element becomes its own trible under the same attribute identifier. Nested objects recurse automatically, linking parent to child entities through GenId attributes derived from the containing field name. After one or more imports, call metadata() to retrieve metadata as a Fragment exporting the derived attribute ids. The fragment contains attribute descriptors plus multi-value hints (a metadata::tag edge pointing to metadata::KIND_MULTI). Use clear() when you need a completely fresh run (drop the per-field attribute caches and multi-value tracking).

Mapping JSON Fields to Attributes

Attributes are derived through the entity-core mechanism — Attribute::<S>::from(entity!{ metadata::name: <field handle>, metadata::value_encoding: <S as MetaDescribe>::id() }) — which hashes the sorted and deduplicated canonical NIL || attribute || value rows to produce a stable attribute id from the JSON field name and its fixed InlineEncoding. The importer caches the resulting Attribute<S> per field so the hash only has to be computed once per run. Arrays are treated as multi-valued fields: every item is encoded and stored under the same attribute identifier, producing one trible per element.

After an import completes the importer regenerates metadata from its cached attribute map. The import_* call returns a Fragment exporting the root entity id(s) for the imported document and containing the emitted facts; call metadata() to retrieve a separate Fragment exporting the derived attribute ids and containing attribute descriptors plus multi-value hints (via metadata::tag pointing to metadata::KIND_MULTI). Merge those descriptors publish those descriptors as metadata alongside the imported facts when you want queries to discover the original JSON field names or project datasets by encoding without repeating the derivation logic. Field names are stored as metadata::name handles to UTF8String blobs so arbitrarily long keys survive roundtrips; metadata::name is a general-purpose entity naming attribute, but importers use it for field names here. Importers intentionally avoid emitting attribute usage annotations; those are reserved for code-defined attributes so each codebase can attach its own contextual names and descriptions.

You can import multiple documents by merging fragments:

let mut all = Fragment::empty();
all += importer.import_str(doc1)?;
all += importer.import_str(doc2)?;
// all.exports() yields the root ids; all.facts() yields the merged tribles.

When exporting back to JSON, pass an immutable blob-store snapshot (for example from MemoryBlobStore::snapshot) to export_to_json so UTF8String blobs can be inlined. If a blob is missing or unreadable the exporter returns an error with the handle hash instead of silently emitting a placeholder, keeping roundtrips lossless when blobs are present. The exporter uses the same fixed mapping in reverse: Handle<UTF8String> → JSON string (via blob lookup), Boolean → JSON bool, F64 → JSON number, GenId → inlined object (unless already visited). Attributes that use other encodings are ignored so JSON roundtrips stay predictable even when the dataset mixes in format-specific extensions.

Nested objects recurse automatically. The parent receives a GenId attribute that points at the child entity, allowing the importer to represent the entire object graph as a connected set of tribles. Because those GenId attributes are also derived from the parent field names they remain stable even when you import related documents in separate batches.

Lossless JSON Import

JsonTreeImporter trades the compact attribute/value encoding for a lossless JSON AST representation. Each JSON value becomes a node tagged with a kind (json_tree::kind_*). Objects and arrays emit explicit entry entities that store field names and indices (json_tree::field_* and json_tree::array_*), preserving ordering and allowing repeated keys. Numbers are stored as raw decimal strings via Handle<UTF8String> so precision is not lost. Array and field indices are stored as U256BE to keep ordering exact even for large collections.

Because node identifiers are derived from the content of each value, identical subtrees converge automatically when you import overlapping backups. This makes lossless imports a good archival layer: you can keep full-fidelity raw JSON and still layer semantic projections on top.

Each import_* call returns a rooted, self-contained Fragment containing the JSON AST facts, every referenced string/number/field-name blob, and the fixed json_tree::* schema in its metafacts. The importer also mirrors content blobs into the backing store supplied at construction. Merge returned fragments when you ingest multiple documents; their descriptions and attachments converge under the same set union. metadata() remains available when a caller wants to inspect the schema directly, but persisting an import does not require a separate metadata step.

Importing N-Triples (RDF)

The import::ntriples module reads the N-Triples serialization of an RDF graph and emits one trible per statement. The importer is a pure function of its input: it returns self-contained fragments whose shared blob stores carry every byte referenced by their facts or metafacts.

use std::io::Cursor;
use triblespace::core::import::ntriples::ingest_ntriples;

let data = br#"
<http://example.org/frank> <http://example.org/firstname> "Frank" .
<http://example.org/frank> <http://example.org/birthyear> "1920"^^<http://www.w3.org/2001/XMLSchema#integer> .
"#;
let import = ingest_ntriples(Cursor::new(&data[..]))?;
assert_eq!(import.triples, 2);

// `facts` is the RDF graph and already carries its predicate vocabulary.
let mut facts = import.facts;
// URI↔id annotations are optional queryable content.
facts += import.uri_map;
Ok::<(), triblespace::core::import::ntriples::IngestError>(())

URI → entity id. Every subject and URI-valued object gets a stable triblespace Id derived from its URI via the import::rdf_uri attribute: the URI is stored as a UTF8String blob, wrapped in an entity! fragment exporting a single rdf_uri edge, and the fragment's content-derived root id becomes the entity id. The same URI always produces the same id across processes, so repeated imports over the same data reach the same TribleSet — even across machines. The rdf_uri inverse-mapping edges are returned separately in NtImport::uri_map: merge that fragment into the graph when pattern!([{ ?e @ rdf_uri: ?uri }]) should recover source URIs, attach it as metadata when recovery should not affect content queries, or drop it when URI recovery is unnecessary.

Orphan blank nodes. A blank node that is referenced as an object but has no outgoing facts cannot use the normal intrinsic-entity derivation: every such factless node would otherwise collapse to the same empty entity. Instead, the importer retains a separate, domain-separated skolem protocol over the immutable source-document content hash and the literal blank-node label. Importing the same byte-identical document repeatedly therefore reproduces the same ids and converges under set union. Different labels remain distinct, as do labels in documents with different serialized content. This is deliberately content addressing of the N-Triples document, not RDF graph canonicalization: changing comments or whitespace creates a new document scope.

Predicate → attribute id. Predicate URIs become attribute ids through the entity-core derivation exposed as Attribute::<S>::iri(predicate). Because attribute ids are hashed together with the chosen InlineEncoding, the same predicate used for two different literal types produces two different attribute ids — which is what you want: :birthyear "1920"^^xsd:integer and :birthyear "1920" (untyped string) shouldn't collide. (JSON field names use the same shape but with metadata::name instead of metadata::iri, so the resulting ids are also distinct from same-spelled IRIs.)

Literal → native value. XSD datatypes map into the appropriate triblespace inline encodings:

Unrecognized datatypes fall back to Handle<UTF8String> so no data is lost — the lexical form ships through verbatim. Numeric parse failures fall back to the string path too.

Roundtrips and querying. Because both ids and attribute ids are derived, you can query the imported graph without inventing a separate schema:

use triblespace::core::attribute::Attribute;
use triblespace::prelude::inlineencodings::I256BE;

let birthyear = Attribute::<I256BE>::iri("http://example.org/birthyear");
for (entity, year) in find!(
    (entity: Id, year: i128),
    pattern!(&facts, [{ ?entity @ birthyear: ?year }])
) {
    println!("{entity} born in {year}");
}

N-Triples only. The current importer handles the line-oriented N-Triples format: one statement per line, URIs in angle brackets, literals in double quotes with optional ^^<datatype>, and acyclic blank-node graphs. Turtle-style prefixes and quad/N-Quads are not supported; cyclic blank-node graphs are rejected rather than assigned arbitrary identities.

Managing Entity Identifiers

The importer buffers the encoded facts for each object as complete 64-byte NIL || attribute || value rows, sorts and deduplicates them, hashes their contiguous bytes with BLAKE3, and takes the final 16 digest bytes as the entity identifier. It passes those same defining rows through the shared intrinsic entity builder, ensuring identical fact sets produce identical IDs even across separate runs. You can supply an optional 32-byte namespace via the constructor; it prefixes the canonical row stream and deliberately prevents identities from unifying across namespaces. The returned Fragment exports the resulting plain Id root and contains exactly the object's defining rows plus any recursively imported child-object facts.

This hashing step also changes how repeated structures behave. When a JSON document contains identical nested objects—common in fixtures such as citm_catalog or Twitter exports—the deterministic importer emits the same identifier for each recurrence. Only the first copy reaches the underlying TribleSet; later occurrences are recognised as duplicates and skipped during the merge. Even if the hash itself is fast, that deduplication step reduces workload on datasets with significant repetition.

Extending the Importers

To support a new external format, implement a module in the import namespace that follows the same pattern: decode the source data, derive attributes with Attribute::<S>::iri(...) for URI-identified predicates or Attribute::<S>::named(...) for display-name origins such as JSON fields, encode values using the appropriate InlineEncoding, and carry the dynamic attribute fragments in the result's metafacts. If the format supplies stable identifiers, mix them into the hashing step or salt so downstream systems can keep imports idempotent.

PATCH

The Persistent Adaptive Trie with Cuckoo-compression and Hash-maintenance (PATCH) is TribleSpace’s workhorse for set operations. It combines three core ideas:

1. Persistence. Updates clone only the modified path, so existing readers keep a consistent view while writers continue mutating. The structure behaves like an immutable value with copy-on-write updates.
2. Adaptive width. Every node is conceptually 256-ary, yet the physical footprint scales with the number of occupied children.
3. Hash maintenance. Each subtree carries a policy-selected summary that allows set operations to skip identical branches early.

Together these properties let PATCH evaluate unions, intersections, and differences quickly while staying cache friendly and safe to clone.

Node layout

Traditional Adaptive Radix Trees (ART) use specialised node types (Node4, Node16, Node48, …) to balance space usage against branching factor. PATCH instead stores every branch in the same representation:

• The Branch header tracks the first depth where the node diverges (end_depth) and caches a pointer to a representative child leaf (childleaf). These fields give PATCH its path compression — a branch can cover several key bytes, and we only expand into child tables once the children disagree below end_depth.
• Children live in a byte-oriented cuckoo hash table backed by a single slice of Option<Head>. Each bucket holds two slots and the table grows in powers of two up to 256 entries.
• A heap Leaf owns its key and value. An archive-backed LocalLeaf is instead a tagged pointer directly into immutable archive bytes. It has no allocation or reference count of its own; the enclosing PATCH keeps the archive alive.

Insertions reuse the generic modify_child helper, which drives the cuckoo loop and performs copy-on-write if a branch is shared. When the existing allocation is too small we allocate a larger table with the same layout, migrate the children, and update the owning pointer in place. Because every branch uses the same structure we avoid the tag soup and pointer chasing that ARTs rely on while still adapting to sparse and dense fan-out.

Resizing strategy

PATCH relies on two hash functions: an identity map and a pseudo-random permutation sampled once at startup. Both hashes feed a simple compressor that masks off the unused high bits for the current table size. Doubling the table therefore only exposes one more significant bit, so each child either stays in its bucket or moves to the partner bucket index + old_bucket_count.

The byte_table_resize_benchmark demonstrates how densely the table can fill before resizing. The benchmark inserts all byte values repeatedly and records the occupancy that forced each power-of-two table size to grow:

ByteTable resize fill - random: 0.863, sequential: 0.972
Per-size fill (random)
  size   2: 1.000  # path compression keeps two-entry nodes fully occupied
  size   4: 0.973
  size   8: 0.899
  size  16: 0.830
  size  32: 0.749
  size  64: 0.735
  size 128: 0.719
  size 256: 1.000  # identity hash maps all 256 children without resizing
Per-size fill (sequential)
  size   2: 1.000  # path compression keeps two-entry nodes fully occupied
  size   4: 1.000
  size   8: 0.993
  size  16: 1.000
  size  32: 0.928
  size  64: 0.925
  size 128: 0.927
  size 256: 1.000  # identity hash maps all 256 children without resizing

Random inserts average roughly 86 % table fill while sequential inserts stay near 97 % before the next doubling. Small nodes stay compact because the path-compressed header only materialises a table when needed, while the largest table reaches full occupancy without growing past 256 entries. These predictable fill factors keep memory usage steady without ART’s specialised node types.

Archive-backed leaf lifetimes

A LocalLeaf is safe only while the allocation containing its bytes remains alive. Each PATCH therefore carries an exact persistent owner cover: a binary Patricia trie keyed by the data address of each retained Arc<dyn ArchiveOwner>. Retaining the owner also prevents its address from being reused. The cover is deduplicated by address and structurally shared across snapshots. Its governing invariant is

owners(LocalLeaves(root)) ⊆ cover

Structural operations preserve that invariant as follows:

• archive insertion retains the owner before publishing its LocalLeaf;
• cloning clones the root and cover together;
• union joins both covers before either root is moved or detached;
• intersection retains both input covers, because it may reuse a leaf from either side;
• difference retains the left cover, because it can only reuse left-hand leaves; and
• consuming iterators carry the cover beside their detached traversal queue until every queued key has been copied out or dropped.

The cover is a lifetime receipt rather than a reachability index. Operations may conservatively retain an owner whose leaves disappeared from the result; direct clearing and removal paths that empty a PATCH clear that provenance. Aggregate reconciliation may nevertheless install or retain a conservative cover on an already-empty PATCH, so emptiness alone does not promise immediate release. A TribleSet shares one such cover across all six PATCH indexes, joining any divergent covers once at aggregate set-operation boundaries.

Hash maintenance

PATCH is generic over a sealed hash-maintenance policy. Sealing is part of the correctness boundary rather than API conservatism: set operations treat equal subtree summaries as equal key sets, so an invalid third-party implementation could silently lose data. The built-in policies share the same canonical trie and differ only in the summary cached by leaves and branches.

The default XorSip128 policy preserves PATCH's original hot-path layout and cost. On first use in a process it samples a private random key. Each leaf fingerprint is the 128-bit output of SipHash-2-4 under that key, and each branch stores the XOR of its children’s fingerprints. On insert or delete, the old contribution is XORed out and the new one XORed in, so aggregate maintenance is constant-time. Set operations compare aggregates first: equal fingerprints short-circuit under the practical assumption that they denote equal key sets, while unequal ones force a structural walk. For any fixed pair of unequal sets, the false-positive probability is approximately 2^-128 under the keyed-hash assumption.

The raw subtree fingerprints are process-local implementation values, not serialized identities. They must remain opaque to untrusted chosen-input callers. Although XOR is linear, the usual linear-dependency construction requires observing the fingerprints of chosen keys; the private key makes that attack inapplicable without such an exposure oracle. PATCH's raw root aggregate therefore stays crate-private.

TribleSet::fingerprint preserves the useful O(1) public cache-key API without opening that oracle. It applies a domain-separated SipHash-2-4 PRF to the root aggregate under a second process-random key initialized beside the leaf key. TribleSetFingerprint::as_u128, Debug, and Hash expose only this nonlinear blinding. Equal sets retain equal tokens within one process, while the XOR of public singleton tokens reveals nothing useful about the aggregate of their union. The token remains a 128-bit cache hint, not a durable content identifier or proof of equality.

The Blake3Merkle policy provides a stable 256-bit root for durable indexes and anti-entropy. Leaves and branches have separate domains. A branch commits to the key width, canonical compressed-prefix bytes, fanout, subtree leaf count, and each (edge, child leaf count, child digest) tuple in ascending edge order. The count and routing metadata used by a reconciliation proof is therefore authenticated at the branch that advertises it; insertion order and the cuckoo table's random physical placement cannot affect the root. BLAKE3 has no inverse update law: an edited branch is marked dirty and summarized once, in canonical order, when its branch editor closes. Only branches on the copy-on-write edit path pay that cost.

For a complete owned key inventory, use PATCH::<N, IdentitySchema, (), Blake3Merkle>::from_keys. It canonicalizes order and duplicates, then constructs the compressed trie bottom-up so every retained leaf and every final branch is hashed once in the construction pass. Debug builds additionally recompute branch hashes to audit the invariants. Sorted unique input takes a linear fast path; other input first pays for sorting and deduplication. Repeated insert remains the right operation for a small edit to an existing snapshot. The patch_bulk benchmark compares those two construction paths on deterministically shuffled distinct 16-, 32-, and 64-byte keys at 10,000 and 100,000 keys; source-buffer cloning is excluded from both timings.

BLAKE3's native chunk tree is not PATCH's tree. It represents fixed-size chunks of one byte stream, while PATCH is a sparse radix trie with path compression and changing fanout. Reusing BLAKE3 chaining values would require PATCH to cache the chunk tree's geometry as a second tree and would not make a child edit algebraic. Explicit branch framing through BLAKE3's streaming API is both simpler and canonical. Team-specific anti-entropy salts belong outside PATCH: key or domain-separate the stable Merkle root at the protocol boundary rather than making the collection's identity depend on who is comparing it.

Archive-backed leaves do not cache their fingerprint, so PATCH avoids hashing them when an exact, cheaper decision is available. Pairs of leaf nodes involving a LocalLeaf compare key bytes directly. A LocalLeaf paired with a subtree of cardinality other than one rejects fingerprint equality from the cached count; a unary branch remains eligible for the ordinary fingerprint path. These are performance shortcuts only, not collision remediation. Pairs without a LocalLeaf retain the normal cached-fingerprint path.

Consumers can reorder or segment keys through the KeySchema and KeySegmentation traits. Prefix queries reuse the schema’s tree ordering to walk just the matching segments. Because every update is implemented with copy-on-write semantics, cloning a tree is cheap and retains structural sharing: multiple owners can clone, mutate independently, and merge results without duplicating entire datasets.

Blobs

Blobs are immutable byte sequences used whenever data does not fit in a trible's fixed 256-bit value slot. BlobEncoding gives those bytes a portable meaning, just as InlineEncoding interprets the value slot.

Handles, encodings, and stores

A blob's handle combines its content hash with a compile-time encoding. The same bytes have the same content identity, while the encoding tells a caller how to validate and decode them. Handles fit inline in tribles, so large values remain ordinary typed graph edges.

Storage is split into small traits:

• BlobStorePut inserts encoded values and returns their handles;
• BlobStoreGet resolves typed handles;
• BlobStoreMeta reports residency metadata without fetching bytes; and
• BlobStoreList enumerates stored objects where a backend supports it.

MemoryRepo, Pile, and remote object stores implement the relevant capabilities. Content addressing makes repeat insertion idempotent and allows caches to copy bytes without coordinating their names.

Fragments carry their attachments

The usual application path does not put each blob manually. entity! accepts a Rust value for a handle-valued attribute, encodes it, and puts the resulting bytes into the returned fragment's shared attachment store:

use triblespace::prelude::*;
use triblespace::prelude::blobencodings::UTF8String;
use triblespace::prelude::inlineencodings::Handle;

attributes! {
    // Local prototype: derive the attribute from its name and encoding.
    pub body: Handle<UTF8String>;
}

let article = entity! {
    body: "A long string which lives in a content-addressed blob.",
};

assert_eq!(article.facts().len(), 1);

Composing fragments with += unions their facts, metafacts, exported IDs, and attachments. store.commit(collection, &signing_key, fragment) copies those attachments before it publishes the signed commit which refers to them. There is no separate staging manifest to keep in sync.

Put and get explicitly when needed

Low-level code can still work directly with a store:

use triblespace::prelude::*;
use triblespace::prelude::blobencodings::UTF8String;
use triblespace::prelude::inlineencodings::Handle;

let mut store = MemoryBlobStore::new();
let handle: Inline<Handle<UTF8String>> = store.put("Fear is the mind-killer.")?;
let snapshot = store.snapshot()?;
let value: View<str> = snapshot.get(handle)?;
assert_eq!(value.as_ref(), "Fear is the mind-killer.");

Explicit insertion is useful for importers, representation builders, and code which must know a handle before constructing the referring entity.

Archives are blobs too

Canonical SimpleArchive encodes a TribleSet as sorted, duplicate-free 64-byte rows. Collection descriptors, commit data, and commit metadata use this representation. The handle of a descriptor archive is therefore the collection's identity; the handle of a data archive is one collection element.

SuccinctArchive and other query-oriented formats are also typed blobs. Their collection encodings define canonical joins, while mappings define canonical cross-encoding transformations. A materialized derived artifact can therefore be stored, reused, or forgotten without changing the authority of the signed source commits.

Conservative references

Every retained current native record other than a BLOB strongly retains each resident blob it references directly, together with that blob's resident child closure. This includes signed MERGE and DERIVE equations: their descriptors, inputs, and results are roots just as a COMMIT's descriptor, data, and metadata are. Preserved WANT records likewise own their referenced blobs; self-contained authorization proofs have no payload blob references. Retention does not depend on signature validity, admission, or whether a materialized equation is useful. An absent reference neither triggers a fetch nor prevents resident sibling references from being retained. A BLOB record does not root itself merely by existing.

The default child walker examines complete 32-byte chunks at offsets 0, 32, 64, ... from the start of each blob and follows candidates which can be read from the resident store snapshot. Canonical SimpleArchive value slots lie on these boundaries. Accidental matches can retain extra objects, but the scanner does not decode schemas or discover handles at arbitrary byte offsets. Formats which rely on this walk must expose their child handles on its chunk boundaries.

Pile and Yard also retain the resident record-kind descriptions of retained frames; the BLOB kind description becomes a root when an independent root selects a blob for retention. See Garbage Collection and Forgetting for explicit policy roots and backend rewrite boundaries.

This division keeps tribles compact, blobs verifiable, and publication self-contained while letting physical storage and cache policy evolve independently.

Pile Format

The on-disk pile keeps blobs, native collection records, native capability proofs, wants, and decodable legacy pin evidence in one append-only file. The write-ahead log is the database: all indices are reconstructed from the bytes already stored on disk. This design avoids background compaction, manifest management, or auxiliary metadata while still providing a durable content-addressed store for local collections. The pile file is memory mapped for fast, zero-copy reads and can be safely shared between threads because existing bytes are never mutated—once data is validated it remains stable.

While large databases often avoid mmap due to pitfalls with partial writes and page cache thrashing [1], the pile's narrow usage pattern keeps these failure modes manageable. Appends happen sequentially and validation walks new bytes before readers observe them, so the memory map never exposes half-written records.

Record model: one frame, 256-byte-aligned records

Every record the pile writes begins with the same 64-byte common prefix and occupies a 256-byte multiple. COMMIT and WANT fit in one block; signed MERGE and DERIVE use two. Blobs and variable capability proofs may continue after the first block and are zero-padded to their declared span:

The magic was minted on 2026-08-20 as two trible genid calls, 0371B249F0626B2ABDDB80E23EA96905 and 9D9656A5EA5A497320351F3BE712CF82, concatenated and truncated to 28 bytes. Those widths are chosen so the body starts at byte 64, and two things follow from that.

Fixed-record fields and blob payloads align. A 32-byte digest, handle, or signature component in a fixed record begins at a multiple of 32 — and because records themselves begin on 256-byte boundaries, the alignment holds at absolute file offsets, not merely within the record. The predecessor framing put a 36-byte prefix in front of the body, which left every such field four bytes short of a boundary and made each one straddle two. Capability proofs also preserve this alignment: their length-delimited grammar has a 96-byte header and 128-byte edges, with 32-byte handles and keys before each signature.

The record kind resolves. 32 bytes is a blob handle, and the handle names a SimpleArchive describing the record kind: its name, the exact byte layout of its body, and the KIND_PILE_RECORD tag. A reader meeting an unfamiliar record can therefore resolve what it is rather than merely failing to recognise it — the same move the collection layer made when descriptors replaced bare definition ids. Each description is rooted at the 16-byte id the kind was already minted under, so widening the field renamed nothing. The handles are pinned in triblespace-core/src/repo/pile/record_kind.rs and a test recomputes every one of them from its description, so editing a description is a format change that fails loudly with the new value rather than silently reframing the pile.

Pile::publish_record_kind_descriptions (exposed as trible pile migrate <PILE> run record-kind-descriptions) stores those description archives for every kind the current binary writes, which makes those kinds resolvable there. Content addressing makes the call idempotent, and the migration's census distinguishes "already resident" from "left to store" so a re-run reports honestly instead of repeating its worklist. Retired PEER, STORE_SCOPE and pile-artifact-offer kinds remain recognized by their pinned constants and decoder, but fresh piles neither write nor advertise those dead formats.

The arithmetic works out exactly. A signed commit contains six 32-byte fields, so 64 + 6 × 32 = 256: one block, nothing wasted. A one-edge capability proof is 224 bytes starting at the record-kind field at offset 32, so its 256 unpadded bytes also fill one block. Longer proofs use the minimal additional whole blocks without changing their canonical body.

A collection descriptor remains an ordinary blob. Capability proofs are self-contained native records and have no companion claim blobs. One typed WANT kind stores a grow-only request set; retired assertion, retraction, and pin records remain structurally readable as explicit migration input but have no current replay effect.

The span includes the header. Zero is invalid; decoders perform checked span * 256 arithmetic and require that the complete record fit in the observed pile prefix. A header-only record therefore has span 1. A blob has span 1 + ceil(payload_length / 256), and the decoder requires the generic span and blob-specific byte length to agree exactly. A u32 block count keeps the prefix compact while permitting a single record of almost 1 TiB.

The common framing is load-bearing:

• Position independence. Blob data starts at the constant record_start + 256; there is no offset-derived padding. A record means the same thing at any offset, so records survive relocation and cat a.pile >> b.pile is a valid merge of two piles.
• Alignment for free. Because every newly written record is a 256-byte multiple, a pile composed entirely of current 256-byte-framed records stays aligned under the atomic lock-free append. Every blob payload in such a pile lands on a 256-byte boundary, satisfying GPU storage-buffer binding requirements (CUDA / Metal min_storage_buffer_offset_alignment) for zero-copy aliasing.
• Cache-friendly headers. Each header begins on a cache-line boundary and admits safe typed views with the zerocopy crate.

Reserved kind-body bytes are zeroed and are not part of the content hash; per-record metadata belongs in tribles, not in the header, so identical bytes never fork into distinct blobs.

Unknown kinds inside a valid frame decode as opaque records. Normal pile replay semantically skips them and continues with subsequent known records; PileRecords still exposes their exact offset, length, kind, and raw bytes. This is a forgetful projection: any future kind introduced under this frame must remain independent of the meaning of known records. In particular, it may not change the validity or effect of a known record, constrain an old writer's otherwise-valid append, or make an existing record depend on a companion record of the new kind. Such an extension—or any other extension whose absence cannot conservatively mean “no effect”—requires a new frame magic instead.

Concatenation is associative ordered physical composition. Current WANT, collection, and capability-proof records collapse to order-independent set union. Decoded legacy pins remain a right-biased log; retired WANT frames and bounded unknown kinds are inert, so concatenating stale history cannot change current demand.

Compatibility surface: v0.46.4, and a reframe for everything else

The last released version is v0.46.4 (tagged 2026-06-10). Its entire record vocabulary is three markers — the 64-byte-aligned V1 blob, branch, and tombstone records — and those are the only records external deployments may hold. They are read forever.

Everything introduced between that release and the current framing never shipped: the V3 record family, the three generations of collection records, the retired WANT logs, retired team state, retired local cells, and the 36-byte legacy envelope. None of it is a writable compatibility commitment. The current reader still recognizes each exact historical boundary so old piles can be inspected or migrated without guessing. trible pile migrate <pile> reframe --into <dest> re-encodes the whole pile into the current framing and drops only explicitly disposable inert records. Retired unsigned equations are preserved exactly; genuinely unknown kinds fail a full preflight before any destination write.

WANT has a cheaper in-place cutover when a whole-file reframe is unnecessary: before first starting a current binary on a pre-cutover pile, explicitly run trible pile migrate <pile> run monotone-wants. It scans the retired log, resolves its final active set once, and appends only missing current positives. The operation is additive and idempotent. It is deliberately excluded from any implicit migration run because it promotes retired history into live demand; after cutover, old frames stay inert even if a stale pile is concatenated.

The re-encode is semantic and in source order, which is what makes it faithful:

• Blob payloads are content-addressed, so copying changes no identity, and their insertion timestamps are stamped afresh — the old timestamp was a local fact about one physical file, not part of the blob identity.
• Legacy pins are replayed in order. Retired WANT state is resolved in order once and emitted only as current positive markers; already-current markers union with that projection.
• Collection records and capability proofs are grow-only sets, so order is irrelevant and re-insertion is idempotent.
• Retired unsigned equations stay inert but retain their exact source frames and each independently resident direct blob reference recursively. Reframe never turns old unsigned bytes into a receiver-signed endorsement.
• Earlier V3 definition-ID collection frames also survive byte-for-byte. This does not add a new interpretation of their historical dependency fields.
• Records that no longer carry retained evidence are dropped and counted: retired PEER, STORE_SCOPE and pile-artifact-offer state. This includes older retired derivation record generations whose old wire shape cannot express the current collection algebra. Current native MERGE and DERIVE records are grow-only materialized work and are preserved exactly; like every retained current native record, they strongly own each independently resident direct blob reference recursively.

Every current collection record's signature covers a domain-separated transcript over its fields, not its frame. Structural replay and semantic reframe preserve those exact signed fields without repeating crypto. Explicit audit verifies the raw records separately, including bad-signature evidence that a trusted local pile may already contain. Opaque ownership, including retired local cells, makes semantic reframe fail closed.

An unknown frame is corruption, not a record from the future

Unknown kind and unknown frame are different questions and get different answers. Conflating them would cost one of the two.

An unknown kind inside a valid frame is forward compatibility: the frame states the span, so the record has an exact boundary, and replay crosses it as an opaque record. Because the kind is a handle, a reader can go and resolve what it was.

An unknown frame is corruption. Nothing about the bytes is trustworthy — not even where the next record starts — so the decoder fails at exactly that offset rather than guessing. This is not a limitation to design around; it is the detection the wide magic buys. 28 bytes is a sentinel, not just an identifier: a mismatch is 224 bits of evidence that these bytes are not a record, so a torn write, a truncated file, or a mis-seek is caught where it happens instead of being read as plausible garbage. The error path must stay sharp; it is never softened into a skip or a warning.

The two failures are still distinguished. A torn or truncated tail — including one that is a proper prefix of the magic — reports ReadError::CorruptPile { valid_length }, which is what amputate repairs. A legacy marker this reader no longer decodes reports ReadError::UnsupportedRecord { offset, marker }, which amputate deliberately refuses to truncate: the remedy there is reframe, not deletion.

Design Rationale

This format emphasizes simplicity over sophisticated on-disk structures. Appending new records rather than rewriting existing data keeps corruption windows small and avoids complicated page management. Storing everything in a single file makes a pile easy to back up, replicate over simple transports, or merge by concatenation, while still allowing it to be memory mapped for fast reads. Internally the pile tracks an applied_length watermark; offsets below this boundary are known-good and only the tail beyond it is rescanned when refreshing state.

Operational workflow

1. Open the file. Pile::open builds the struct around a File handle and memmap2 mapping. It does not read any records yet (and it does not create missing files — create the file explicitly for a fresh pile).
2. Load and validate. refresh acquires a shared lock, walks bytes beyond applied_length, and rebuilds the blob, collection-record, capability-proof, WANT, and legacy pin-snapshot indices in memory. It fails loud on a corrupt or torn record (ReadError::CorruptPile { valid_length }). It skips bounded unknown envelope kinds as opaque records and distinguishes an unknown legacy marker as ReadError::UnsupportedRecord { offset, marker }. It never mutates the file. Callers rarely need to invoke it directly: Pile::snapshot and collection/WANT/pin-snapshot observations refresh internally before observing records, so external writers are visible without a standalone scan.
3. Amputate only when asked to. amputate is the explicit, opt-in repair path: it re-runs validation under an exclusive lock and truncates the file back to the last valid record, discarding a torn record left by a crash. It crosses complete opaque envelopes and may truncate a torn opaque tail at its known start. It refuses UnsupportedRecord without modifying the file because an unknown unenveloped record's boundary is unknowable. It is deliberately not part of the normal open sequence. The operator CLI additionally requires the exact boundary reported by the current reader: trible pile amputate <path> --truncate-to <byte-offset>. A stale generic repair suggestion or a guessed offset therefore cannot trigger truncation. The reader compares that offset and calls set_len while holding the same exclusive file lock, so an intervening repair cannot move the boundary between validation and mutation.
4. Append new records. put (through the BlobStorePut trait), CollectionStore::insert, and WantStore operations extend the file. Each append immediately feeds the bytes back through the record scanner so in-memory indices stay synchronised without waiting for a manual refresh. Blob records use a single write_vectored call; fixed-width collection and WANT records use one append of their complete 256- or 512-byte frame, and native proof records append their complete bounded frame once. Records larger than ~1 GiB can't be appended in a single atomic writev because kernel write_vectored calls cap at INT_MAX bytes on macOS and MAX_RW_COUNT (~2 GiB) on Linux. In that case put takes an exclusive file lock and issues plain write_all calls — still append-only, still repairable by an explicit amputate if a crash leaves a partial tail, but serialised against other writers for the duration of the append.
5. Read through a snapshot. SnapshotSource::snapshot refreshes the pile, then clones the memory map and persistent PATCH indices into a PileSnapshot. Blob bytes, collection records, and capability proofs all come from that one immutable prefix and can be read without further locking.

This lifecycle keeps pile usage predictable: open → operate (operations refresh as they run) → freeze immutable snapshots. If a process wants to scan for new appends between operations (for example, a background monitor that is not issuing a snapshot or record enumeration), it can explicitly call refresh to pick up external writers without blocking them for long. If corruption is ever reported, surface it to the operator; truncating is a decision, not a default.

Immutability Assumptions

A pile is treated as an immutable append-only log. Once a record sits below a process's applied offset, its bytes are assumed permanent. The implementation does not guard against mutations; modifying existing bytes is undefined behavior. Only the tail beyond the applied offset might hide a partial append after a crash, so validation and repair only operate on that region. Each record's validation state is cached for the lifetime of the process under this assumption, avoiding repeated hash verification for frequently accessed blobs.

Hash verification only happens when blobs are read. Opening even a very large pile is therefore fast while still catching corruption before data is used.

Every newly written record begins with the generic marker, kind ID, and span described above. The sections below illustrate each kind-specific body.

Usage

A pile typically lives as a .pile file on disk. Applications open it through Pile::open and load it with refresh (directly or via the first operation that refreshes internally). Multiple threads may share the same handle thanks to internal synchronisation, making a pile a convenient durable store for local development. Blob appends use a single O_APPEND write. Each handle remembers the last offset it processed and, after appending, scans any gap left by concurrent writes before advancing this applied_length. Writers may race and duplicate blobs, but content addressing keeps the data consistent. Each handle tracks hashes of pending appends separately so repeated writes are deduplicated until a refresh.

use std::error::Error;
use std::path::PathBuf;

use anybytes::Bytes;
use triblespace::prelude::*;
use triblespace::core::repo::pile::ReadError;
use triblespace::core::repo::BlobStoreMeta;

fn add_blob(bytes: &[u8]) -> Result<(), Box<dyn Error>> {
    let path = PathBuf::from("data.pile");
    let mut pile = Pile::open(&path)?;
    // Load and validate the existing records. This FAILS LOUD on a corrupt
    // or torn record and never mutates the file. Unknown envelope kinds are
    // skipped as opaque; unknown legacy markers remain unsupported.
    match pile.refresh() {
        Ok(()) => {}
        Err(err @ ReadError::UnsupportedRecord { .. }) => return Err(err.into()),
        Err(err @ ReadError::CorruptPile { .. }) => return Err(err.into()),
        Err(other) => return Err(other.into()),
    }

    // Insert a blob and obtain a handle pointing at the on-disk bytes.
    let handle = pile.put(Bytes::from_source(bytes.to_vec()))?;

    // One immutable snapshot owns every read capability for this pile prefix.
    let snapshot = pile.snapshot()?;
    if let Some(meta) = snapshot.metadata(handle)? {
        println!("stored {} bytes at {}", meta.length, meta.timestamp);
    }
    drop(snapshot);
    pile.close()?;
    Ok(())
}

This pattern illustrates the typical flow: open, load with refresh, append through the storage traits, then hand a PileSnapshot to read-only consumers. refresh acquires a shared lock so it cannot race with an explicit amputate, which takes an exclusive lock before truncating a corrupted tail.

Filesystems lacking atomic write/vwrite appends—such as some network or FUSE-based implementations—cannot safely host multiple writers for records below the ~1 GiB atomic-write threshold and are not supported in that mode. (Records above the threshold use the exclusive-lock fallback and don't rely on filesystem atomicity.) Using an atomicity-lacking filesystem for small records risks pile corruption.

Bounded refresh snapshots

Replay snapshots the observed file length once per pass and decodes exactly that bounded prefix. A shared-lock writer's single append syscall can expose an intermediate file length, so a shared replay parse failure is rechecked once under an exclusive lock with a fresh length. This completion barrier waits for participating writers; a persistent malformed or unsupported record remains an error, without truncation or retrying any write. Shared blob-writer preflight uses the same recheck. Post-write readback instead stops at the completed blob (or an earlier concurrent duplicate) before a later writer's partial tail. Appends beyond a successful bounded observation remain for the next refresh.

PileSnapshot receives persistent PATCH roots when it is created. Later refreshes can extend the pile without changing existing snapshots. Blob replay maintains one segmented PATCH relation keyed by hash || offset_be. It retains every physical occurrence in file order, while its zero-copy view at the 32-byte segment boundary is the semantic resident-blob set used for listing, membership, differences, and cover intersection. Reads walk the offsets for one hash lazily when an earlier payload fails content validation. The validation byte lives in the persistent occurrence leaf, so snapshots share cached verdicts without a separate map or per-handle Arc-linked duplicate chain. StoreSnapshot::changes_since compares the Blob, collection-record, and capability-proof PATCH roots directly. For Blobs the physical occurrence root means adding a duplicate fallback or replacing a corrupt same-handle candidate is visible while unrelated appended records are not. Classification is therefore constant in the number of semantic components rather than a scan of either snapshot; component-specific consumers can reuse unchanged derived state.

Tools that need the raw log rather than the collapsed state—reflogs, consolidation, forensics—should use PileRecords, an iterator over every record in a pile file in log order. It shares its decoder with the replay path described above, so it understands every record format ever written; do not hand-roll a parser against the layouts documented in this chapter. An unknown envelope kind is yielded as PileRecordContent::Opaque with its declared boundary; callers can preserve its exact bytes through the iterator's raw file view. An unknown unenveloped marker is reported as UnsupportedRecord, while a malformed or truncated record is reported as CorruptPile.

For an operator-facing view of one exact boundary, use trible pile diagnose record-at <pile> <offset>. The command is read-only: it walks the same canonical decoder from byte zero, rejects offsets that land inside a record, and prints the physical marker, classification, known span, next offset, and the fields the current reader can safely decode. In particular, an unsupported unenveloped marker asks for a newer reader and never suggests amputation; only a malformed or torn known record presents the explicit destructive-repair command.

Semantic Pile and Yard reads may continue across opaque records, but destructive retention cannot guess their ownership. Pile::rewrite_retained_into carries each byte-distinct opaque frame unchanged and retains every resident blob in the observed prefix, regardless of the caller's narrower roots. Repeated copies use BLAKE3 digests to avoid appending the same frame again; physical offsets and duplicate occurrences are not preserved. trible pile compact uses this conservative path and reports the opaque frames it carries.

Yard collection, compaction, and reclaim still refuse before mutation when any opaque record is present. Semantic reframe also refuses unknown kinds at its preflight. Neither can infer a smaller safe live set from unknown ownership.

The retired V4 collection DERIVE kind is deliberately not opaque. Its former fields and semantics are known, and a derivation carried neither ownership nor authoritative state: it is a computation whose artifact is rebuilt under the current recipe. Replay exposes it as RetiredCollectionDeriveV4, projects no collection record, and semantic rewrites may discard it. This distinction lets the unknown-kind fence stay strict without letting known inert computations block semantic compaction forever.

Blob Records

Each blob record carries:

• Record kind – identifies blob semantics inside the generic envelope.
• Timestamp – milliseconds since the Unix epoch when the append occurred.
• Payload length – the unpadded byte length of the blob.
• Hash – the digest produced by the pile's hash protocol (BLAKE3 by default) and used as the blob handle.
• Reserved – zeroed padding to the fixed 256-byte header length; not part of the content hash.

The payload follows at record_start + 256 and is post-padded to the next 256-byte boundary. The Pile Blob Metadata chapter explains how to query these fields through the PileSnapshot API.

Native Collection Records

CollectionStore is a grow-only set of signed typed collection-calculus records: COMMIT assertions and witness-bound MERGE and DERIVE endorsements. The pile stores them directly as fixed enveloped records: COMMIT occupies 256 bytes; MERGE and DERIVE each occupy 512. COMMIT's signed bytes and kind are unchanged. The current equation kinds are distinct from both their unsigned and their signed payload-only predecessors. They are not blob records, have no following payload, and carry no insertion timestamp. They are also distinct from operational wants and historical pins: collection records have no head, tombstone, or last-writer-wins update. Their exact canonical value is the semantic record; a collection record is not a trible entity. The pile indexes a record by the collection and member it names and the frame it sits in; a full-width BLAKE3 fingerprint exists for anything that needs a fixed-width identity, such as the network delta.

The collection itself is identified by a canonical SimpleArchive descriptor. Its 32-byte blob handle is the sole CollectionHandle. Records carry this handle directly; there is no definition record or registry. Consequently a transferred collection record names the exact descriptor bytes needed to interpret it, using the ordinary blob store.

The descriptor archive holds a descriptor entity carrying:

• metadata::tag, the descriptor kind;
• one anchor — an attached UTF-8 collection_name on a root, or collection_source on a derivation, naming by handle the collection it derives from. A reader holding the pile can resolve a root's human-readable name without needing the code that minted an opaque id. A derivation carries no name of its own: its source already anchors it. Naming the source by handle rather than by a shared label means a descriptor cannot claim a lineage it does not have;
• repeated resource_policy links to self-contained policy entities, each naming an exact capability_handle. The standard READ and WRITE definitions and any application-specific definitions share this relation. Consumers query the bound definition's invocation-action facts to select the policy for a requested action; a proof's grant handle need not equal the binding handle. Existing bindings and standard definition blobs retain their byte identities. An open policy needs no proof. A quorum policy carries a canonical nonempty set of Ed25519 roots and one semantic quorum threshold. The byte-compatible descriptor may still contain the earlier optional delegation-threshold fact, but authorization ignores it; a signed proof edge's capability definition states which actions its recipient may delegate onward, independently of which it may invoke. Roots and derivations state both policies independently; source walking never supplies authority. Their ordinary facts participate directly in the descriptor handle;
• collection_representation, naming the canonical member encoding. The encoding owns validation and the intra-encoding join;
• on a derivation, collection_mapping, linking a concrete mapping entity. Canonical builders derive its id, but readers validate the entity's facts rather than its minting history. It names one mapping_algorithm and carries its concrete parameters as ordinary tribles — which attribute an observed set observes, which pair a register is ordered by, or the complete automaton a path summary uses.

A mapping algorithm id names the reusable computation, while the linked mapping entity names one parameterization of it. The complete descriptor archive—not a requirement that the entity id was intrinsically minted—binds those facts into the target collection identity. The parameters remain queryable rather than being hidden behind an opaque hash-only API.

The archive may also carry the describe fragments of the encoding and mapping algorithm, so the descriptor states what its bytes and conversion are rather than only naming them. A bare id is legible only to someone who already holds the code that minted it, which is exactly the reader — a peer receiving a collection it has never seen — that most needs the answer.

The magic markers below identify the compact pile representation. They are storage-envelope markers, distinct both from the stable semantic kind IDs used in fingerprints/signature domains and from the one-byte versioned tags used by generic dense record stores. There is no equivalent SimpleArchive form for these algebra records.

The new MERGE and DERIVE description anchors above were minted with installed trible genid on 2026-09-14. Their signed semantic kind IDs, independently minted in the same session, are 96AB9DE7389DD621D44C55E593898A69 and 529D8F57A20FCA0BC921121313DC619B, respectively. The descriptions are named pile-collection-merge-v6 and pile-collection-derive-v7; their handles above are pinned by the description-recomputation test. COMMIT retains semantic kind B34817308188C4515A3C51967A91A603 and its historical version-2 transcript.

Generic stores and collection repair use a one-byte dense tag followed by the same ordered 32-byte body fields, without native framing or reserved padding:

Tags 2/3 are retired unsigned layouts; tags 4/5 are retired signed payload-only layouts. None is an alias for a witness-bound endorsement. The combined equation/AUTH-v5 wire epoch uses ALPN /triblespace/pile-sync/26. An equation signs its domain bytes (triblespace.collection.merge.endorsement or triblespace.collection.derive.endorsement), the corresponding new semantic kind ID, author public key, and its ordered collection/operand/result/witness fields. The newly minted kind versions this grammar; there is no extra equation version integer. COMMIT's historical domain, kind, version integer, field order, and signature bytes are unchanged.

Public dense from_bytes constructors perform strict signature verification, including canonical non-weak author keys. Network ingress uses that checked boundary once. Locally signed constructors build their typed values directly; typed store inserts, adapters, local repair indexes, and outbound encoding do not re-verify. Native replay and explicit raw audit decode structurally. The object-store backend is likewise trusted persistence: it checks canonical structure and the exact content-addressed key without repeating signatures. Foreign pile/import bytes need an explicit checked ingress or a user-approved trusted-source boundary; structural open alone is not an import verifier.

An equation producer endorses the mathematical result. What a result stands for is the coverage index's answer: the monotone union of everything believed beneath it, foundation COMMITs and admitted DERIVE leaves reached through the MERGEs the store's own host key signed. Attachment applies WRITE to the target's candidate foundations and reads their rows; it needs the descriptor lineage and the producing records to be available, and a payload no believed record produces is reported rather than taken for empty support. A later admitted foundation or host MERGE about the same payload extends that answer; another key's MERGE, or an unadmitted record, changes nothing. The fold compares a MERGE's key with the host and checks no signature, so this holds for records whose signatures were checked when they entered the store, as above. Only the selected output and its encoding-required blob dependencies need be resident for materialization; historical inputs may have been evicted. This is producer trust, not proof of mathematical correctness: an authorized dishonest producer, or a bug in the host's own merge, can still endorse a wrong result.

MAP(attached; node -> attachment) is the fourth native record: a host's own index of one node of a parent collection's lattice, its attachment mapped from the node's own bytes. The MAP kind was minted with trible genid on 2026-09-28: semantic kind 329669AA662709605053C123671DF1D2, pile description anchor ED7B98CD20AD9A4D172F037A08CC47F1 (description pile-collection-map-v1, handle above pinned by the description-recomputation test), signed over the domain 4A5191C36898D0B453FEFF99ED0E3852 || 4AADD87ED8EB8620BD39623B5221B5BE. It carries no witness and adds no foundation, and it means nothing to another store: a store believes a MAP only when its own host key signed it, keeps it as the attachment of its node, and a reader uses it only for a node it reaches in the parent's lattice. A Rank9-accelerated member is an ordinary blob root plus its portable raw child, attached to the same node as that child's Succinct attachment; neither attached collection has MERGE records of its own. The accelerated root's first 32 bytes name the raw child, so generic blob traversal can follow the dependency without a special pile index. An attached read treats an attachment as usable only when its root and required representation closure are resident; otherwise it descends to the attachments of the nodes beneath. Typed materialization then defensively validates the selected raw/index pair. The unpublished mapping-evidence record kind was clean-cutover removed after a scan found no live records requiring migration.

Every reserved byte must be zero; a nonzero reserved byte makes replay fail as corrupt rather than silently assigning meaning to a format extension. Merge inputs are sorted by payload digest, so swapping operands cannot create a second representation of the same endorsement.

There is no synthetic trible entity or separately stored ID for the record. On replay, the decoder reconstructs its exact dense typed payload: 192 bytes for a commit, 224 bytes for a merge, and 192 bytes for a derive. Every current payload includes the public key and both signature components. A CollectionRecordFingerprint, the BLAKE3 digest of the stable semantic kind ID followed by every canonical payload byte, names the actual persisted record wherever a fixed-width identity is needed, such as the network delta PATCH; it names a record, not a blob, invented entity, or member of Support. The pile computes none at replay.

Pile replay indexes every collection frame, by the collection it names, the member it produces, and the frame's offset. Inserting a record the file already holds appends nothing: the producers of the same member are the only records it could be, and they are compared as records. Replay does not make that comparison. Done per frame it is quadratic in a member's producers, and on a 68 GB pile it cost 294 million record decodes per open while collapsing nothing, because that pile's 1,367,796 collection frames were all distinct. Since insert never writes a repeat, a record can only occupy two frames when piles are concatenated, and then each frame gets its own index entry until a compaction rewrites the pile through insert. Reads hide the repeat: every copy of a record sits under the same collection | member prefix, so a walk in key order meets them together and hands the record out once, and the coverage fold is idempotent. Concatenating piles therefore gives set-union semantics for collection records: append order and duplicate copies do not change the discovered collection calculus. Current operational WANTs are likewise a grow-only set. Historical pins remain ordered evidence; retired WANT logs are only explicit migration input and do not participate in ordinary replay.

The immutable record read surface also provides raw relationship selectors. Collection(C) selects every record naming C; ProducedMember(C, H) selects every COMMIT data, MERGE result, or DERIVE output matching the exact collection and payload; CommitMember, MergeCollection and DeriveTarget narrow those by kind. One snapshot-shared PATCH keyed C | member | frame offset (72 bytes) answers all of them by prefix, and the record itself is read from its frame. Mixed selectors return each record once, in index order; unindexed backends retain the same scan-based contract. The index only projects stored record fields: refresh performs no capability admission, signature recheck, or blob read for it, and changes no record bytes or encoding identities.

Retired payload-only equations and reader cutover

The first signed payload-only MERGE kind was 9D9B962D46FA42168AB3A11FB367AC14692D4F51B5190196F2BDB08D5BC2BA07 (anchor BC68266A511EC292D815A30C8DFBA82D, semantic kind 1E5277B23D177FD692B074FBF0EF28F1). Its dense tag 4 carried 224 bytes and its native frame occupied 512 bytes. DERIVE used B2EE8382C70161379E387D692B822946A60B602A909EED66B7D6DA2A62F36232 (anchor 7FDDB25BB2B40E002A7B0EC40E316232, semantic kind CE6A838612D0AA0812C05A50CCD11DC1), dense tag 5 with 192 bytes and a 256-byte frame. Their signature domains ended in .transcript, not .endorsement. They authenticated payload equations but no particular input-record route; their old signatures cannot be reinterpreted as the new endorsement. Ordinary native replay crosses these exact framed kinds as opaque evidence and does not put them in the current collection record index or repair stream.

The earlier unsigned MERGE kind was 0CEE320DE0BDA40A6A6F52221C5E4E4D2CE3B165B69C858673FD13D98F655379, with collection/low/high/result at 64..192 and zeros through byte 256. DERIVE was 7ACE1ED10F3EBC632627058CC461DC1CC171CD2E56C52E5DCE60EA4C8DC23C36, with target/input/output at 64..160 and zeros through byte 256. Their retired dense tags 2/3 carry 128/96 payload bytes and retain historical semantic kinds 5F20FFC64313969B7E046A7677874D39 / 46C621338B6DD5B71C8E1E6DD74B087C. The earlier 36-byte-envelope forms, and unenveloped MERGE V4, are recognized as the same unsigned evidence rather than current equations.

PileSnapshot::legacy_unsigned_collection_equations() exposes their exact equations for explicit migration, separate from CollectionRead and ordinary repair. GC and semantic reframe preserve their raw source frames, and retained rewrites preserve all resident descriptor/input/output reference closures. Object-store listing omits these equations from current semantics while leaving their immutable objects untouched and checking their historical content keys. An endorsement migration must append a new signature by a writer authorized for that exact collection (the target for DERIVE). It must not use every receiver's key or delete unsigned evidence that cannot yet be endorsed.

No shipped command restates them any more. The endorse-unsigned-equations migration that re-signed retired equations under a current writer's key was applied to the live piles and then removed together with the witness-bound record kinds it produced; a pile that still holds unrestated frames keeps them as inert evidence that a current reader crosses without interpreting. Old signed opaque frames retain the ordinary conservative copying rule: preserve their bytes and all resident blobs; Yard and semantic reframe still refuse opaque ownership. Use trible pile verify <PILE> for the separate native-signature audit before accepting an unfamiliar pile as a trusted local store. That audit does not prove the mathematical correctness of historical unsigned equations.

An older reader that understands the generic native frame can open a pile with new signed kinds: it crosses them as opaque, at their declared spans. This is not rewrite compatibility. Old retained GC refuses opaque ownership, and the old semantic reframe implementation may silently drop unknown records. Do not use an old installed reframe on signed-equation piles. The new implementation preflights truly unknown kinds and refuses before writing output; it explicitly preserves known unsigned evidence.

Native Capability Proof Records

CapabilityProofStore is a second grow-only native set. Each member is the canonical self-contained prefix-signed proof body

magic32 | resource32 | root32 |
    (capability_handle32 | delegate32 | signature64)+

Its logical key is CapabilityProofId = BLAKE3(body). The ID is reconstructed during replay and is not duplicated in the frame.

There is no proof-specific envelope, inner magic, or separate byte-length field. The proof starts with the record kind itself. Its header is exactly 96 bytes:

Each following edge is exactly 128 bytes:

The body length must be exactly 96 + 128n for 1 <= n <= 255. Replay parses every Ed25519 key, requires canonical non-weak principals, requires the declared span to be the smallest span containing the body, and rejects any nonzero padding byte as corruption. There are no inline modes, flags, or validity bytes. A one-edge body is 224 bytes, so its pile record fits one 256-byte block exactly. The generic framing declares only the block span. To recover the exact proof, the decoder walks fixed-width edges until the zero suffix: a valid edge cannot be all zero because an all-zero delegate key is weak. A bounded scan, the minimal-span check, and an entirely zero remainder make this unambiguous even when padding occupies one complete edge. Zero bytes at the end of a signature remain part of the proof. Odd edge counts fill their blocks exactly; even edge counts leave 128 bytes of padding.

This grammar's description is named pile-auth-proof-v5. Its anchor above was minted with trible genid on 2026-09-14. Its incompatible wire identity changes proof bytes, not existing collection descriptors or their policy bindings.

Insertion of identical bytes is idempotent. Different bytes reconstructing to the same proof ID are a collision and fail. Exact lookup is only by proof ID; the store does not infer a proof from a key, resource, or request, and record presence grants no authority. Each signature is last and covers every earlier body byte through its own delegate, including the record kind and prior signatures. Generic framing and block padding are excluded. An exact prefix ending after any signature is therefore a complete proof for that intermediate delegate.

Pile and MemoryRepo insertion verify every signature; trusted native replay checks framing and keys without repeating cryptography. Byte-only verification does not acquire or interpret capability definitions. Generic authorization separately reads those definitions: capability_action permits invocation and capability_delegate_action permits conferring invocation or onward delegation. For child Q following P, I(Q) ∪ D(Q) ⊆ D(P). There is no same-handle rule.

Conservative rewrites preserve every current canonical proof record and each resident capability definition it references, without fetching absent blobs or authoring WANTs. The opaque resource field is not interpreted as a blob handle by storage. Semantic verification needs the external trust root, expected subject, resource/action request, and readable capability definitions, not a generic clock. Applications may interpret their own restrictions before quorum counting, as described in Resource Capability Proofs. Physical retention grants no authority.

The earlier K(S,C,K)+, 145-byte-edge action-ID, and 161-byte-edge handle/mode/validity proof kinds remain structurally recognizable as inert frames so append-only piles can be traversed safely. Fresh writers do not emit them, replay projects no authority from them, and semantic rewrites may drop them. Their signatures cover other grammars and cannot be mechanically migrated; the relevant private keys must reissue authority.

Retired Team-Era Records

Earlier network hosts used PeerStore as a grow-only set of routing hints. Each historical PEER member was written as PEER(team_public_key, peer_public_key). Those names describe the old writer's intent; the current reader treats both 32-byte fields as uninterpreted historical bytes. Because the record is inert, replay validates only its fixed framing, span, kind, and reserved zeros—not whether either field encodes an Ed25519 point. The historical dense body was the 64-byte concatenation team || peer; its optional physical selector was a domain-separated BLAKE3 identity over that body.

The same generation also wrote a fixed STORE_SCOPE record:

Current replay validates both layouts and exposes dedicated retired record variants to raw inspection, but builds no index or repository state from either. Current collection-scoped networking does not create, synchronize, or route from them: DHT referrals, liveness, and provider leases are process-local soft state, first contacts come from the keys a collection's records and grants name and from the addresses the pile's own sync daemon records in its configuration collection, and collection policy is the sole admission and disclosure authority. Semantic reframe, retained rewrite, Yard reclaim, and trible pile compact deliberately drop both retired kinds. Genuinely unknown records remain opaque. Retained Pile copying preserves them and all resident blobs; semantic reframe and Yard reclamation still fail closed.

Retired: Collection Publication Grants

A grant was an author-signed, irrevocable permission to redistribute that author's commits in one exact collection, stored as a fourth pile record kind beside the collection calculus. It is gone, and nothing in a pile refers to one: 21.2 GB across six piles were scanned for its record marker before removal, and the same scan found ordinary commit records, so the absence was the grant's rather than the scan's.

An intermediate design replaced grants with a collection_reach descriptor attribute. That design is retired too. Current descriptors carry mandatory, independent READ and WRITE policies; collection-scoped repair proves READ(C), and commit admission proves WRITE(C). Exact-content provider leases are instead derived from every served resident H and are independent of collection policy. There is no ambient full-team inventory or separate durable gossip permission.

The semantic kind ID 9BB5B1F4D6FD8FB850B494C2CF51B5CA (minted 2026-08-12, retired 2026-08-21) and its record kind 5E18D982337466E65CB8278658CF53027FC109385456B49D35C4E66D6E9CE599 are recorded here and in collection::records::KIND_COLLECTION_GOSSIP_V1 so neither is minted a second time. A pile is not expected to contain one; if any did, it would read as an unknown record rather than a permission.

Legacy unenveloped V4 collection records

Before the legacy envelope, the same three V4 kind IDs occupied bytes 0..16, followed immediately by the semantic fields. These postdate v0.46.4, so they are read only for reframe.

Legacy V3 collection records

V3 encoded a collection by a separate definition record with a 16-byte intrinsic entity ID. Its V1 commit signature transcript and equations therefore do not identify the current descriptor-handle semantics. The reader recognizes all four old markers so it can validate record boundaries and preserve their bytes during conservative rewrites, but treats them as inert physical evidence: they never enter CollectionStore, assert membership, or retain blobs. Semantic reframe preserves these raw V3 headers too. This narrower historical guarantee does not reconstruct their definition-ID semantics or add GC roots; the descriptor-handle unsigned equations above have the stronger direct-reference retention needed by the signed-equation endorsement transition.

Legacy Pin Records (head / tombstone)

These historical records map a pin identifier to a blob hash or tombstone that mapping. Current code decodes them into immutable snapshots for explicit migration, capability reachability, diagnostics, and byte-preserving retained rewrites. No current publication API appends, advances, or tombstones a pin. The legacy encoding never required the referenced blob to be resident locally.

Retired Local Cell Records

These kinds are never written under the current framing, so they exist only in the legacy envelope and the unenveloped V3 form:

These markers belonged to an experimental named last-writer-wins value API. That API and its writers were removed before release: a whole-value replacement was not invariant under pile concatenation and made independently edited policy silently order-dependent. The markers are retired permanently and must never be assigned new meaning.

Current readers recognize both the legacy enveloped form above and the fixed-width unenveloped V3 form solely to preserve migration evidence. They expose either form as PileRecordContent::Opaque, do not project a value into current state, and do not treat its referenced archive as a retention root. Raw tooling through PileRecords can still copy or explicitly migrate the exact bytes. Because their former ownership semantics are no longer interpreted, Pile::rewrite_retained_into and Yard collection/reclaim refuse a destructive rewrite while any such record remains.

Want Records

The root id was minted with trible genid on 2026-09-02. One frame is one element of a grow-only set keyed by the exact canonical 97-byte WantRequest below. Repeating a request is idempotent. There is no timestamp and no retraction kind: concatenation is set union. Retired frames are inert, so stale pre-cutover history can neither remove nor resurrect current demand.

The canonical key is tag || A || B || C: byte 0 is the versioned tag, 1..33 is A, 33..65 is B, and 65..97 is C. The Blob and Derive decoders reject nonzero unused fields. Merge inputs must be in lexicographic order (low <= high), so operand order cannot create a second request key.

The three request kinds deliberately share durability but not policy. Blob(H) is the sole exact-content fetch/retention intent. Merge and Derive are durable questions about reproducible collection work; an answer is the corresponding native MERGE or DERIVE receipt already defined by CollectionStore, not mutable state inside the want. The storage format persists those questions independently of whether a local or remote worker eventually supplies the receipt.

Forgetting is a physical storage-policy operation. Every retained WANT owns each independently resident handle named by its request recursively. Yard does not weaken this edge through a recency or byte budget; a deliberate physical rewrite may instead omit the WANT and its ownership edges. No negative record is appended.

Four former current-frame kinds—the blob assertion/retraction pair EC1C024C04AF08243DB3AE318C93FA500355C74395C0F553CFFC0AF0A4BA0346 / ACCB531FC7489357C40FCEF0DDE8BD9088F2AC1924A652EA211ADD5C30B95B46 and typed assertion/retraction pair 65EE9E4279FFE01D263E75A8E2DF6289B6DE403CB4468098A0EAB925F81C28ED / A57C866A83A90635090A947D92464B19D9F898C0C961AB7A91C79A979F9F1483—plus their legacy-envelope and unenveloped weak-pin forms are now retired. Ordinary replay recognizes their exact framing but treats them as inert. The explicit monotone-wants migration and whole-pile reframe are the only operations that interpret their file order: both resolve the former per-request LWW log; migration appends missing positive members under the current kind, while reframe emits that final projection into its destination. Historical typed tag 3 encoded a derive as (source, target, input); resolution keeps all three fields in its historical identity, then projects each active key to current Derive(target, input). The current kind accepts only canonical tag 4.

Legacy unenveloped records

The legacy envelope's bodies all began at byte 36, four bytes short of a 32-byte boundary; the tables above give the current offsets, and the legacy ones are each exactly 28 lower.

Unenveloped V3 blob, branch, and retired blob-WANT records place their kind ID directly in 0..16. Their semantic bodies begin at byte 16 rather than byte 36: a V3 blob stores timestamp at 16..24, byte length at 24..32, and hash at 32..64; branch IDs occupy 16..32; branch values occupy 32..64; and retired WANT handles occupy 16..48. All have a 256-byte header and remain structurally readable byte-for-byte; retired WANT records have no ordinary replay effect. The two retired local-cell markers are the one deliberate exception to the unknown-unenveloped rule: their historical 256-byte boundary is known, so the reader crosses them and exposes them as opaque migration evidence. Their former cell ID occupied 16..32, and a replacement's archive handle occupied 32..64. The legacy V3 and V4 collection layouts are listed above.

Piles written before V3 contain 64-byte-aligned V1 records: a 64-byte blob header (marker, timestamp, length, hash) followed by a payload padded to a 64-byte boundary, and 64-byte branch / tombstone records. The reader recognises the V1 markers and reads these records byte-identical; they are never rewritten. V1 had no want records.

Recovery

refresh scans an existing file to ensure every record fits. It does not verify blob hashes. A malformed or truncated known or enveloped record reports the number of bytes that were valid so far using ReadError::CorruptPile. A complete unknown envelope kind is structurally accepted and semantically skipped; an unknown unenveloped marker reports its bytes and offset using ReadError::UnsupportedRecord, since the reader cannot infer that record's length. The retired cell markers are recognized as fixed 256-byte opaque records rather than guessed. Both errors leave the file untouched, and the reader never guesses any other legacy record length.

If the file shrinks between scans into data that has already been applied, the process aborts immediately. Previously returned Bytes handles would dangle and continuing could cause undefined behavior, so truncation into validated data is treated as unrecoverable.

refresh holds a shared file lock while scanning. This prevents a concurrent amputate call from truncating the file out from under the reader.

The amputate helper is the explicit, destructive repair path: it re-runs the same validation under an exclusive lock and truncates the file to the valid length if corruption is encountered, discarding incomplete data left by an interrupted write. It crosses complete opaque envelopes, truncates a torn one at its start, and propagates UnsupportedRecord for unknown unenveloped markers without truncating. Run it deliberately (e.g. via trible pile amputate <path> --truncate-to <byte-offset>)—never as a routine part of opening. The CLI refuses a boundary that differs from the current reader's result. Hash verification happens lazily only when individual blobs are loaded so that opening a large pile remains fast.

For more details on interacting with a pile see the Pile struct documentation.

Pile Blob Metadata

Every blob stored in a pile begins with a compact header. Besides the payload hash (covered in Pile Format), the header records when the blob was appended and how long the payload is. The Pile implementation surfaces this information so tooling can answer questions such as "when did this blob arrive?" without walking the raw bytes on disk.

Header fields at a glance

The current 256-byte generic envelope written ahead of every blob contains:

The pile snapshot also accepts the legacy envelope and legacy V1/V3 blob headers, and projects their timestamp, length, and hash through the same API.

BlobMetadata re-exposes the timestamp and length fields so callers can read when a blob was appended and how large the payload is.

BlobMetadata

BlobMetadata is a lightweight struct shared by blob-store implementations. It mirrors the timestamp/length pair in the header and leaves validation to the snapshot:

• timestamp: the write time stored in the blob header as a u64. A convenient way to turn this into a SystemTime is shown below. Pile::put records this value using SystemTime::now(), so it reflects wall-clock time and can move forward or backward if the system clock is adjusted.
• length: the size of the blob payload in bytes. Padding that aligns current entries to 256-byte boundaries is excluded from this value, so it matches the slice returned by PileSnapshot::get.

Looking up blob metadata

PileSnapshot::metadata accepts the same Inline<Handle<_, _>> that other blob store APIs use. The snapshot consults its persistent in-memory index and, on the first request for a handle, lazily hashes the payload to confirm the bytes match the handle. Subsequent metadata lookups for the same handle reuse that cached validation result. When the payload passes validation the method returns Some(BlobMetadata); otherwise it yields None.

PileSnapshot is the immutable prefix frozen by Pile::snapshot. Request another snapshot to refresh the pile and observe blobs appended afterwards. PileSnapshot::metadata never fails for valid snapshots—its error type is Infallible.

use std::time::{Duration, SystemTime, UNIX_EPOCH};

use anybytes::Bytes;
use triblespace::core::blob::encodings::UnknownBlob;
use triblespace::core::blob::Blob;
use triblespace::core::repo::pile::Pile;
use triblespace::core::repo::{BlobStoreMeta, BlobStorePut, SnapshotSource, StorageClose};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut pile = Pile::open("/tmp/example.pile")?;

    let blob = Blob::<UnknownBlob>::new(Bytes::from_source(b"hello world".to_vec()));
    let handle = pile.put(blob)?;

    let snapshot = pile.snapshot()?;
    if let Some(meta) = snapshot.metadata(handle).unwrap() {
        let appended_at = UNIX_EPOCH + Duration::from_millis(meta.timestamp);
        println!(
            "Blob length: {} bytes, appended at {:?}",
            meta.length, appended_at
        );
    }

    drop(snapshot);
    pile.close()?;
    Ok(())
}

Failure cases

metadata returns None in a few situations:

• the handle does not correspond to any blob stored in the pile;
• the snapshot predates the blob (create a new snapshot to refresh and see later writes);
• validation previously failed because the on-disk bytes did not match the recorded hash, for example after the pile file was corrupted before this process opened it.

When None is returned, callers can treat it the same way they would handle a missing blob from get: the data is considered absent from the snapshot they are reading. Validation is cached in the physical occurrence's persistent PATCH leaf, which immutable snapshots share. Later calls therefore reuse the verdict; a future snapshot can recover when the pile acquires another occurrence of the same handle whose payload validates.

For additional background on the binary layout and how the header interacts with padding, see the Pile Format chapter.

Glossary

This chapter collects the core terms that appear throughout the book. Skim it when you encounter unfamiliar terminology or need a refresher on how concepts relate to one another in TribleSpace.

Action

An uninterpreted 128-bit identifier naming one exact operation. Actions do not form a hierarchy and never imply one another. For example, ACTION_WRITE and ACTION_READ are separate atoms even when they concern the same collection.

Attribute

A property that describes some aspect of an entity. Attributes occupy the middle position in a trible and carry the InlineEncoding (or blob-handle encoding) that interprets and validates the value. Modules mint them with the attributes! macro, so they behave like detached struct fields: each attribute remains independently typed even when many are combined to describe the same entity, preserving its individual semantics. Provide an explicit 128-bit id in the macro when you need a canonical column shared across crates or languages; omit the literal to derive a deterministic id from the attribute name and encoding (the macro wraps the name + encoding id in an entity!{} fragment and takes the root for you), which is handy for short-lived or internal attributes.

Blob

An immutable chunk of binary data addressed by the hash of its contents. Blobs store payloads that do not fit in the fixed 32-byte value slot—long strings, media assets, archived TribleSets, commit metadata, and other large artifacts. Each blob is tagged with a BlobEncoding so applications can decode it back into native types.

Blob Store

An abstraction that persists immutable content-addressed blobs. Implementations back local piles, in-memory collections, or remote object stores while exposing small capability traits for insertion, retrieval, metadata, and enumeration.

Capability Proof

A canonical self-contained, prefix-signed byte string. Its header contains a grammar magic, opaque resource, and one Ed25519 root. Each edge then contains a capability-definition handle, delegate, and strict Ed25519 signature over the complete prefix through that delegate. Every signed prefix is therefore a proof for its intermediate subject. Definitions provide independent invocation and delegation action sets: a child's invocation and delegation actions must both be delegated by its parent. Its BLAKE3 digest is the proof ID used for exact physical lookup. Interpretation receives a definition reader, external trust root, expected subject, and resource/action request, with no generic clock or same-handle restriction. Proof presence alone grants no authority.

Capability Proof Store

A grow-only native set of canonical capability proofs. It supports deterministic enumeration and exact lookup by proof ID, but no discovery by key or semantic request. Storing a proof preserves evidence but does not make the proof authorized or root any blob closure.

Commit

A signed native collection membership assertion. A CollectionCommit names the exact collection descriptor, data element, mandatory metadata archive, and author. Its exact canonical 192-byte value is the assertion; it has no synthetic entity ID. Commits are independent leaves rather than snapshots in a parent chain.

Capability Presentation

One owned CapabilityProof paired with the exact subject key the caller expects it to establish. The expectation prevents a valid prefix or proof for another principal from silently becoming an admission decision; no companion bundle or claim blobs are needed.

Collection

A self-describing grow-only join semilattice. Signed commits introduce members; stored merge records describe materialized joins within the lattice; derivation records describe materialized mappings into another collection. Warm resolution reuses those equations without executing their algebra again. A collection has no distinguished head. In Rust, Collection<E> is the cheap descriptor handle after the runtime member encoding has been validated against the CollectionEncoding type E.

Cover

One exact point in a collection lattice, represented by a typed collection descriptor and a PATCH set of distinct Handle<E> payload handles. Signatures, authors, and metadata are optional provenance fibers queryable from the store, not part of cover identity or required for replay, so several commits over identical data collapse to one member. Distinct covers may have the same support: a stored merge records that {a, b} and {a⊔b} denote the same join. Cover construction is opaque; admission and stored collection algebra produce them rather than accepting caller-forged hash sets.

Collection Admission

The read-time signer decision performed by collection.admitted(&store_snapshot). Each WRITE-policy root supplies its own share; resident proof paths rooted in the policy's canonical root set are considered for ACTION_WRITE on the collection's exact descriptor handle. A writer is admitted only when it has the policy's required distinct root support through resident definitions. Invalid or irrelevant candidates grant nothing without poisoning other evidence; collection membership has no generic expiry.

Collection Descriptor

A canonical SimpleArchive describing a collection's UTF-8 root name or exact derived source, member encoding, and independent READ and WRITE admission policies. Each policy is open or a canonical quorum over capability roots with one semantic threshold. A derived descriptor also links one concrete mapping entity carrying its algorithm and parameters. Its content handle is the CollectionHandle, so every native record which names a collection can resolve its meaning through the ordinary blob store. A derived descriptor states its own policies and never inherits them through its source.

Collection Encoding

A BlobEncoding with one canonical member validation rule, exposed by CollectionEncoding. Every member is an ordinary typed blob, and every collection encoding defines one canonical associative, commutative, and idempotent member-join operation. When one blob cannot hold the result, a finer Cover<E> represents the same join, so the collection lattice remains total. Derived collections are full lattices connected by mappings, not projection-only representations.

Collection Member

One ordinary typed Blob<E> admitted into a Collection<E>. A source-bound encoding may name another blob in its bytes; validation and cover-aware views follow that handle through the same immutable store snapshot rather than wrapping the member in another runtime artifact. For Rank9-accelerated SuccinctArchive, the root embeds its exact raw source handle. Its canonical join names the corresponding raw union as an immutable dependency and succeeds once that blob is resident.

Collection Derivation

A parameterized source-to-target conversion owned by the target encoding's CollectionDerivation implementation. Its associated Source fixes the incoming encoding, while Argument carries the concrete runtime parameters in the target descriptor's mapping entity. It maps ordinary source blobs to ordinary target blobs. The mathematical contract is a join homomorphism over their logical values: f(a ⊔ b) = f(a) ⊔ f(b).

When neither encoding crate can own that implementation, DeriveMapping is the coherence-safe explicit extension seam. The derive_with, ensure_with, and maintain_with operations select such a mapping value or type; both surfaces execute the same derivation engine and persist the same descriptor facts.

Attached Collection

An index attached to a root collection: its descriptor names the parent root and a mapping, and carries no policy. It has no lattice of its own. A MAP record, signed by the store's host, names the attachment built from one node of the parent's lattice; a store believes only its own host's MAPs, so one that reaches another store stays inert. A read takes, for each frontier node widest first, its usable attachment -- bytes and dependencies resident -- and otherwise descends through the host's merges; the foundations no attachment reaches are the read's residual. Succinct, Rank9, id-set, latest-state, last-writer-wins, BM25 and path-summary indexes are attached.

MAP

The signed native record MAP(attached, node, attachment): the host's claim that attachment is the attached collection's mapping of the parent node node. It carries no witness and is believed only when signed by the store's host.

Collection Store

A grow-only set of native COMMIT, MERGE, DERIVE, and MAP records. Insertion is idempotent by exact canonical record value; combining two stores is set union. Physical fixed-width indexes may key that value by its full-width fingerprint.

Store Snapshot

One immutable, coherent known-prefix observation produced by SnapshotSource::snapshot. A snapshot owns all blob, collection-record, and capability-proof reads for that prefix and implements StoreSnapshot::changes_since for conservative local invalidation. Collection admission reads the coverage index in it and produces a semantic Cover<E>, the union of what the collection's frontier stands for; Collection::read and CollectionSnapshotExt::collection attach that frontier from the same index and reconstruct an eager value or a lazy sharded view from the nodes whose bytes are resident, through the same snapshot.

Collection READ

The exact ACTION_READ capability over one collection descriptor handle. A node pushes a collection's records, proofs and held set only to a peer that passes READ(C) under its own evidence and the credentials the peer presented when it asked to peer. Knowing the collection handle permits finding holders of its descriptor and asking them to peer, but does not reveal records, proofs, counts, or blobs. Open READ needs no proof. Exact immutable content is a separate bearer system: every served resident H may be advertised under opaque KDF(H), and exact GET neither names a collection nor consults READ(C).

Constraint

The trait that every query operator implements. Its methods—variables, estimate, propose, confirm, satisfied, and influence—let the Atreides solver navigate the search space without a separate planner. propose and confirm take a frontier: a whole batch of parent bindings, of which a single binding is the width-1 case. Constraints are stateless: every method receives the bindings it needs as a parameter, so the engine can backtrack, batch, and split without telling anyone. Estimates guide variable ordering and never change results; confirm may only kill candidates, never add or revive them. Custom data sources and application predicates participate in queries by implementing this trait.

Entity

The first position in a trible. Entities identify the subject making a statement and group the attributes asserted about it. They are represented by stable identifiers so multiple facts about the same subject cohere.

In practice you pick an identifier policy:

• Extrinsic ids (for example ufoid, fucid, genid) track a conceptual subject across edits and versions. Use these when you intend to accumulate additional facts over time.
• Intrinsic ids (content-derived hashes) are recomputed from the entity's asserted fields. The entity! macro uses this policy when you omit the explicit id @ prefix (or when you write _ @), so identical records unify naturally.

Ownership policies and schemas determine who may mint new facts for a given identifier.

Fragment

A self-contained bundle of exported IDs, content facts, descriptive metafacts, and one content-addressed blob store shared by both fact sets. entity! and import pipelines return fragments; entity! carries descriptions for the attributes that actually emitted facts. Fragments compose via += without mixing descriptions into ordinary queries. Use Fragment::root() to extract derived IDs, Fragment::empty() to start accumulation, and spread (*) to pass child fragments into parent entities, giving Merkle trees for free.

Derive

A signed endorsement of an exact mapping from one source element into a derived collection. The target descriptor names both source and concrete mapping, so the record carries the target, input, output, signer, and signature. Target WRITE admits its producer; the source's WRITE policy does not. Derivations are reusable materialized work, not new membership. Every successful mapping is stored with its equation; Yard/GC decides later whether its bytes remain resident.

Merge

A signed endorsement of a ⊔ b = c inside one collection. A resident stored result can replace its inputs in a physical cover without changing the logical value or creating new membership. Warm resolution admits the producer by that collection's WRITE policy and does not recompute the join.

PATCH

The Persistent Adaptive Trie with Cuckoo-compression and Hash-maintenance. A single PATCH stores one ordering of a trible set in a 256-ary trie whose nodes use byte-oriented cuckoo hash tables and copy-on-write semantics. A TribleSet maintains six PATCH instances — one per permutation of entity, attribute, and value. Shared leaves keep permutations deduplicated, rolling hashes let set operations skip unchanged branches, and queries only visit the segments relevant to their bindings, further described in the deep-dive chapter.

Pile

An append-only collection of blobs, native collection records, native capability proofs, WANT records, and legacy pin evidence stored in one file. Retired PEER and STORE_SCOPE records remain structurally readable but do not enter repository state and disappear under semantic compaction. Piles are memory mapped, recoverable after interrupted appends, and mergeable by byte concatenation. Legacy pin records remain decodable only for conservative retention and explicit migration.

Encoding

The byte-layout contract for a typed value. Encodings assign language-agnostic meaning to the raw bytes — they are not the concrete Rust types — so any implementation that understands the encoding can interpret the payloads consistently. Inline encodings map the fixed 32-byte payload of a trible to native types; blob encodings describe arbitrarily long payloads so tribles referencing those blobs stay portable. The corresponding traits are InlineEncoding and BlobEncoding.

Policy Root

One Ed25519 key named by an admission policy. Roots have inherent support for their policy's action and may issue capability paths. A collection may name several roots and require one threshold; READ and WRITE have independent root sets and thresholds. Whether one root share remains delegable is carried by the delegation-action facts of definitions signed into that path, not derived from sibling proofs. A root contributes one share, including to itself; it is not exempt from a multi-root threshold. A root is not a network namespace, routing scope, roster, or mutable owner.

Collection WRITE

The exact ACTION_WRITE capability over one collection descriptor handle. Signed COMMITs are active only when their author satisfies the descriptor's WRITE policy in the observed proof set. Local stores may retain inactive commits; synchronization and concatenation remain monotone because later proof evidence can activate them without retracting bytes.

Push

One collection tree, its records, its authorization evidence or its held set, sent to one peered neighbour the sender sends C to, on a walk/1 stream the sender opens and on a per-collection timer of two to sixty seconds. The sender walks its pinned tree; the receiver prunes it with HELD bitmaps, asks for the values it lacks and lands them as they arrive, and its LANDED confirms the root, so an unchanged tree is ROOT then DONE next time. The held set travels only between two Full neighbours. A push is never forwarded and is not the source of truth.

Peering

Two keys' agreement, on the one connection between them, to sync one collection. Each side asks only for collections its pile selects. The asked side accepts when it selects the collection too and the asker passes READ(C), or passes WRITE(C) with its send flag set. Each side's send flag says whether the collection flows from it to the other. The keys asked are the ones the collection's records and grants name, then the DHT's providers of its descriptor; no configured list of peers exists. Peerings belong to their connection and form no roster.

Held Set

The resident blobs reachable from a collection's seeds -- its descriptor, its foundations' direct references and its proofs' capability definitions -- as a positive PATCH kept by the store for collections replicated in full. Two Full neighbours push it to each other like any other tree, pruned by what both hold. It is a local observation, not membership.

Sync Selection

The register in a pile's own configuration collection that says, for each collection, whether the pile syncs it. trible pile net select and unselect write it and selection lists it; an unset or conflicted register selects nothing. The same configuration records where the pile's sync daemon listens, so a process that opens the pile reaches the daemon first.

Trible

A three-part tuple of entity, attribute, and value stored in a fixed 64-byte layout. Tribles capture atomic facts, and query engines compose them into joins and higher-order results.

TribleSpace

The storage model which organises tribles across blobs, PATCHes, and native collections. It emphasizes immutable content-addressed data, monotone set semantics, and reproducible derived representations.

Inline

The third position in a trible. Values store a fixed 32-byte payload interpreted through the attribute’s schema. They often embed identifiers for related entities or handles referencing larger blobs.

WANT

A durable operational request for a blob or for an existing merge/derive result. Blob(H) is the sole exact-content request and can be fulfilled through collection-independent KDF(H) discovery. WANT is operational policy: it neither adds collection authority nor changes a collection's logical value.

Formal Verification Roadmap

This roadmap captures the initial strategy for driving the triblespace crates toward comprehensive formal verification. It unifies model checking, symbolic execution, fuzzing, and deterministic simulation so we can reason about both the low-level data structures and high-level collection workflows with stronger correctness guarantees.

Verification Stack Overview

• Model checking with Kani explores bounded but exhaustive state spaces for invariants that must never be violated.
• Symbolic execution with Miri exposes undefined behaviour (UB) and aliasing issues across regular unit tests without requiring new harnesses.
• Coverage-guided fuzzing stresses APIs with randomized input sequences to uncover emergent behaviours that formal proofs might miss.
• Deterministic simulations replay realistic collection and storage workflows so we can audit higher-level semantics and regression-test subtle interplays between subsystems.

Each technique complements the others; together they provide layered assurance that keeps regressions from reaching downstream users.

Goals

• Protect the fundamental algebraic properties of TribleSet, PATCH, and the native collection ledger.
• Exercise serialization, deserialization, and zero-copy data views under adversarial inputs.
• Detect behavioural regressions in query heuristics, constraint solving, and collection resolution before they reach downstream users.
• Integrate the tooling into CI so proofs and regression checks run automatically for every change.
• Preserve a contributor-friendly workflow where verification steps are discoverable, well documented, and quick to reproduce locally.

Current Foundation

• proofs/ already contains Kani harnesses for query, value, and variable-set behaviour. They provide examples of bounded nondeterministic data generation (kani::any, Inline::new) and assume/guarantee reasoning that new harnesses can reuse.
• ./scripts/preflight.sh is the aggregation point for formatting and tests; adding verification steps here keeps contributor workflows consistent.

Invariant Catalogue

The roadmap anchors future work around the following invariants. Each row tracks the subsystem we care about, the guarantees we want to encode, and a rough sketch of how to exercise them in Kani, Miri, or fuzzing harnesses.

Expansion Plan

Phase 1 – Harden the Existing Kani Coverage

1. Catalogue crate-level invariants and map them to concrete Kani harnesses. Start with:
   • TribleSet operations preserving canonical ordering and deduplication.
   • Join heuristics in atreides ensuring variable bindings never alias conflicting values.
   • Collection resolution preserving explicit payload support under alternate covers.
2. Extract shared helpers for generating bounded arbitrary data (e.g. Vec::bounded_any) so harnesses remain expressive without exploding the search space.
3. Adopt a per-module harness layout (proofs/<module>_harness.rs) registered from proofs/mod.rs to make maintenance predictable.
4. Configure scripts/verify.sh to run targeted cargo kani --harness <name> invocations in parallel, then wire it into CI with caching to keep runtimes manageable.

Phase 2 – Symbolic Execution with Miri

1. Enable cargo miri test for the default test suite to surface undefined behaviour (UB) and aliasing bugs that regular tests may miss.
2. Gate flaky or unsupported tests with cfg(miri) guards so the suite stays deterministic under the interpreter.
3. Document the workflow in scripts/preflight.sh and optionally expose a dedicated scripts/miri.sh for local runs when developers need deeper debugging.

Contributor Workflow

1. Run ./scripts/preflight.sh before every commit; it aggregates formatting, testing, and (eventually) targeted verification checks.
2. Use cargo kani --harness <NAME> locally when iterating on a new proof. Start from the harness templates in proofs/ so generators and assumptions stay consistent.
3. Execute cargo miri test after modifying unsafe code, pointer logic, or concurrency primitives; it catches UB bugs that normal tests cannot surface.
4. Kick off fuzz targets with cargo fuzz run <TARGET> when touching boundary code (deserializers, planners, collection sync). Store new corpus inputs in version control if they expose bugs or tricky behaviours.
5. Record findings, gaps, and future work in INVENTORY.md so the roadmap evolves alongside the implementation effort.

Phase 3 – Fuzzing and Property Testing

1. Introduce a cargo fuzz workspace targeting:
   • PATCH encoders/decoders with binary corpus seeds generated from integration tests.
   • Join-order heuristics to explore combinations of constraint graphs and filter predicates.
   • Collection sync workflows by fuzzing unions of commits, merges, derivations, missing blobs, and WANTs.
2. Reuse structured generators from proptest where deterministic shrinking is valuable, and bridge them into fuzz harnesses when possible to keep the state space constrained.
3. Automate nightly or on-demand fuzz campaigns via CI artifacts, storing any found counterexamples alongside minimised reproducers.

Phase 4 – Deterministic Simulation Testing

1. Model collection replication scenarios with deterministic event queues to explore sparse evidence, garbage collection, and concurrent writers.
2. Encode the simulations as regular unit tests backed by recorded execution traces so they can double as documentation for expected behaviour.
3. Capture simulation scenarios discovered during fuzzing to prevent regressions.

Milestones & Reporting

• Track coverage for each invariant in a shared dashboard (CI summary or INVENTORY.md) so contributors can quickly spot gaps.
• Celebrate major wins—like a new harness landing or a bug found via verification—in the CHANGELOG to reinforce the value of the effort.
• Review and refresh this roadmap at least once per release cycle to keep the guidance aligned with the architecture.

Tooling Integration

• Track verification status in CI badges and documentation so contributors know which guarantees currently hold.
• Extend INVENTORY.md with follow-up work items whenever new invariants or subsystems are identified.
• Keep verification-specific configuration (Kani property files, fuzz corpora, deterministic seeds) under version control to make runs reproducible.

Next Steps

1. Break the invariant catalogue into GitHub issues that reference the planned harness files (proofs/tribleset_harness.rs, proofs/patch_harness.rs, etc.).
2. Prototype the PATCH harness that drives Branch::modify_child through insertion/growth cycles so we can assert the displacement planner and table_grow never drop entries; wire the run into scripts/verify.sh.
3. Evaluate CI capacity to determine how frequently Kani proofs, cargo miri, and fuzzers can run without blocking contributors, documenting the cadence directly in INVENTORY.md.

This roadmap should evolve alongside the codebase—update it whenever new verification opportunities or obstacles appear.