Codify the "internal artifact vs interop artifact" doctrine — it is already load-bearing, and written down exactly once

Источник: dev/fera-research/docs/oc4/idea-04-internal-vs-interop-doctrine.md (read-only; сгенерировано scripts/sync-docs.mjs)

Codify the “internal artifact vs interop artifact” doctrine — it is already load-bearing, and written down exactly once

• kind: idea
• area: architecture / storage / modules
• evidence:
  • ~/dev/fera/project-design/context/RESEARCH-OQ-003.md §2.2 — the doctrine, stated once, applied once: “deflate-per-block in Lattice (OQ-013) was chosen for ubiquity in an interop container read by foreign tools, whereas the sticker cache is internal, rebuildable, content-hash-keyed, never leaves the machine — ‘Lattice’s choice does not set a precedent here.’”
  • ~/dev/fera/project-design/context/ADR-024:424 — “Companion to OQ-013’s scope decision that Lattice (the CAR/Merkle container) is for WASM plugins only — native modules remain plain signed dylibs”
  • ~/dev/fera/project-design/context/NAMING-UX.md:36-37 — Lattice is “a separate standalone project … plugins/runtimes only — native modules stay out of it”
  • ~/dev/fera/project-design/context/OQ.md:872-881 — OQ-024(d): pin the rustc version “so our own modules compile without #[repr(C)] — a fixed rustc version makes the Rust ABI itself stable enough between our modules” — only sound because those modules are ours
  • ~/dev/fera/project-design/context/ADR.md:784-786 (ADR-020) — “Flat, simple, hot-path types are #[repr(C)] structs passed by value” — the general, always-safe rule
  • ~/dev/fera/project-design/context/OPTIM-IDEAS-4-oq24.md §7 — the anti-list forbids “extern "Rust"/Rust ABI types across independently built plugins”
  • ~/dev/fera/project-design/context/ADR.md:665-674 (ADR-031) — deterministic-HMAC searchable index, frequency-analysis limitation accepted because it is opt-in secret-chat-only, while the main index stays plaintext-at-rest with disk encryption “by default (speed)”
  • ~/dev/fera/project-design/context/TECHSPEC.md:196-204 — sticker cache format: raw vs compressed decided by CPU/disk tradeoff, never leaving the machine
  • ~/dev/fera/project-design/context/OQ.md:713-724 — OQ-021: Turso .ttso files are derived, rebuildable copies; RON is the source of truth; mismatch = warning
  • ~/dev/fera/fera/core/crates/storage/src/memory.rs:18-26 — MemoryEngine is a real, tested, internal implementation whose job is to be replaced by a durable one

Суть / Thesis

One sentence in RESEARCH-OQ-003.md §2.2 contains a general design doctrine the project has been quietly running for months:

An artifact read by foreign tools optimizes for ubiquity and stability. An artifact that never leaves the machine optimizes for speed and can be rebuilt.

Every hard-won decision in the corpus is downstream of that split, and none of them are labelled with it:

• Lattice uses deflate, not zstd — interop artifact.
• The sticker cache uses zstd −1, not deflate — internal artifact.
• OQ-024(d) drops #[repr(C)] via a pinned rustc — sound only for our own modules (internal), and deliberately forbidden across independently built plugins (interop), per OPTIM-IDEAS-4 §7.
• The main search index is plaintext-at-rest with disk encryption for speed, while the secret-chat index pays for HMAC tags — both internal, chosen by a different axis (ADR-031’s own opt-in design).
• Turso .ttso indexes are explicitly derived and rebuildable, so a mismatch is a warning, not data loss — internal artifact.

The doctrine is real, it is already the tiebreaker, and it has no name, no recorded statement, and no place to be checked. That means every future format question gets re-litigated from scratch, and — more dangerously — it means the #[repr(C)] decision (ADR-020’s blanket rule) and the “no repr(C)] for our own modules” decision (OQ-024(d)) look like a contradiction in the docs when they are actually two applications of one rule.

Варианты / Options (with tradeoffs)

A. Leave it. Each future decision re-derives it from first principles.

• Cost: low per-decision; the corpus already has one contradiction-shaped pair (ADR-020 vs OQ-024(d)) that will read as a conflict to any future reader, including future agents.

B. Write it as a section in RFC.md.

• Pros: discoverable.
• Cons: RFC is “system design / what and how”. A rule about tradeoffs belongs with the decisions it explains.

C. A short ADR (proposed) — “classify every artifact as internal or interop, then optimize for the right axis” — with the §2.2 sentence promoted to Context, the seven worked examples as Evidence, and a five-question test as Decision.

Proposed solution

The ADR’s operative content: a five-question test

For any format, container, index, cache, or ABI, ask:

1. Who reads it? Only this project → internal. Anything outside (other tools, other languages, third parties, a format proposal) → interop.
2. Can it be rebuilt from a source of truth? Yes → internal. No → interop (this is why OQ-021 can treat a Turso mismatch as a warning).
3. Does it ever leave the machine? Never → internal. Transmitted, uploaded, exported → interop.
4. Is a change to it survivable? A new version of an internal artifact is a rebuild; a new version of an interop artifact is a migration someone else has to implement.
5. Who is the adversary? For internal artifacts, the project’s own future self (it may delete the code that reads it — so prefer regenerable). For interop artifacts, a third party on the other side of the format — so prefer stable, documented, boring.

Then the payoff: five concrete decisions it re-derives or unblocks

Artifact	Classification	What the doctrine says that the corpus currently argues for
Lattice .lttc	interop (foreign tooling, Rust build-script + CLI consumers)	keep deflate; keep canonical-RON; the standalone repo + CLI (§13) is not optional for an interop format
Sticker frame cache	internal (rebuildable from the .tgs, never leaves)	zstd −1 + per-frame codec tag, exactly as RESEARCH-OQ-003 §4 concluded
A trace/replay container (see idea-02)	internal	reuse lattice’s discipline, not its format — its appendability/interop costs buy nothing
The plugin UI buffer (see idea-03)	— it is both (produced by an untrusted third party, consumed by the host)	this is the hard case, and the doctrine says: it is interop-shaped (must be validated byte-for-byte against a frozen spec, version-skew-safe, no engine types) living inside an internal pipeline. That is precisely why chroix-contract’s “plain data, no engine types, append-only, skippable unknown ops” rules are the right rules — and it retro-justifies idea-03’s whole shape.
repr(C) vs pinned-rustc Rust ABI	interop for plugin/foreign boundaries; internal between our own modules	ADR-020’s blanket #[repr(C)] and OQ-024(d)’s drop-it are the same rule applied to two classifications. The “contradiction” disappears.

And one place the doctrine should not be applied — stated, so it isn’t misapplied later

The trust manifest. ADR-003 chose JSON “for external-tooling interoperability”. The manifest is classified as interop, so the doctrine predicts: boring, stable, documented, externally implementable — and not optimized for parse speed. That is the right answer, and it is worth writing down explicitly, because “internal, we can use rkyv” is exactly the kind of argument someone will make the first time manifest parsing shows up in a profile — and it would be wrong, for a reason now on record.

Cost

One ADR, about a page. It resolves a doc-shaped contradiction retroactively, it retro-justifies two already-made decisions (zstd for stickers, the plugin buffer’s contract discipline), it settles the container-format question for traces before anyone builds one by accident, and it gives every future format question a 30-second test instead of a re-derivation.

This is a proposal about process, which is unusual, and the justification is that this corpus already contains the answers — it just does not contain the rule, and the rule is cheaper to keep than the answers are to re-derive.