Lattice reference implementation — plan

Источник: dev/fera/lattice/TODO.md (read-only; сгенерировано scripts/sync-docs.mjs)

Lattice reference implementation — plan

For agentic workers: REQUIRED SUB-SKILL: use superpowers:executing-plans (or subagent-driven-development) to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: working core of the .lttc format: write/read containers, content addressing (FastCDC + BLAKE3), Ed25519 sign/verify — proven by a green pack → unpack → verify round-trip, clippy-clean.

Architecture: workspace of crates per CONTRIBUTING.md — lttc-format (pure byte-exact types + spec-contract crypto derivations, no I/O), lttc-writer (pack: chunk → FS metadata → layout → sign), lttc-reader (canonical parse order §3.2 → verify → unpack), later lttc-cli, lttc-build. The §5.3/§5.4 message constructions (super-node, sig_input, key_id) live in lttc-format::sig so writer and reader share one source of truth.

Tech Stack: Rust (edition 2021, stable-compatible), blake3 1, ed25519-dalek 3, fastcdc 5 (ronomon = the v2020 variant §8 pins), thiserror 2, hex 0.4; Phase C adds ron (see OPEN note) + toml; Phase D clap 4.

Spec: SPEC.md (Draft v1, frozen) — canonical for every byte layout, constant and error below. Section refs (§N) are to it.

Global Constraints

Frozen values copied from SPEC.md — every task inherits these:

• Magic b"LTTC_BUNDLE >^<\0" (16 B); superblock 4096 B; signed prefix superblock[0..160]; 4 sig slots × (64 B sig + 32 B key_id) at 160+96·k; detached-sig ptr at 544/552; reserved tail 560..4096 zero.
• All integers little-endian. Block-table entry 64 B; Merkle node 72 B; pointer-table entry 48 B. On-disk CID = bare 32-byte digest (§6.1).
• Hash BLAKE3 (hash_alg 0x02); signatures Ed25519 only (sig_alg 0x01).
• key_id = BLAKE3("lttc-key-v1" || ed25519_pubkey) (§5.4).
• sig_input = BLAKE3_keyed(ctx "lttc-sig-v1") over "lttc-slot" || slot u8 || subtree_name_utf8 || superblock[0..160] || super_node[32].
• super_node = BLAKE3_keyed(ctx "lttc-supernode-v1") over, for each present root in order [plugin, translations, themes]: name_len u16 || name_utf8 || digest[32] (§5.3).
• Chunking FastCDC v2020 fastcdc::ronomon::FastCDC, min 8 KiB / avg 16 KiB / max 64 KiB; files < min = one block (§8).
• FS rules (§5.2): entries sorted by raw name bytes; names UTF-8, no normalization, MUST NOT contain /, \, NUL, be ./..; no mtime anywhere; odd-leaf rule duplicate-last; binary trees only.
• Caps (§11): manifest ≤ 16 MiB; block length_uncompressed ≤ 64 MiB; blocks ≤ 2²⁴; Merkle depth ≤ 64; path ≤ 4096 B. Alignment padding zero-filled, excluded from hashes.
• pack MUST be deterministic: same tree + key ⇒ byte-identical file (§13).
• Reader runs §3.2 checks in order 1–9 (cheap arithmetic before any hash / signature / attacker-sized allocation).
• Error taxonomy + exit codes: LttcError (§12) is the contract; do not invent variants.
• #![forbid(unsafe_code)] in every crate.
• Commits: conventional, English, do not push.

Writer conventions to ratify (proposed — OPEN round needed)

Spec gaps the core writer must fill; each gets an OPEN-DECISIONS.md round, answers marked proposed until the author ratifies (CLAUDE.md: decisions live in the repo, not chat):

1. Root composition. §5.2 names roots (Plugin → /code+/data+ /manifest, Translations → /localization, Themes → /themes) but does not define how a multi-directory root combines into one “subtree digest”. Proposed: plugin = digest of the FS root dir node (covers the whole tree); translations / themes = digests of /localization / /themes top-level dir nodes when present, else absent (omitted from super-node).
2. Subtree names in sig_input / super-node: lowercase "plugin", "translations", "themes"; slot 3 name TBD in Phase D (unused in core).
3. v1-core writer emits no pointer table and no compression (both optional per §5.2/§7; flags = 0). Reader MUST still accept/parse them where the spec requires (table trust rule) — deferred to Phase D+, absent tables are legal.
4. Manifest region is opaque bytes in core (typed canonical RON = Phase C); it is still length-capped (16 MiB) and its locators are inside the signed prefix.

Review Focus

Failure modes the spec implies; each pinned by a test in the owning task:

1. Hostile superblock arithmetic (huge/overflowing offset+length) must be rejected by §3.2 steps 1–8 before any read/alloc — Task B4 test rejects_out_of_bounds_regions.
2. Malicious block table (unsorted, duplicate digests, block offsets outside the content region) — Task B4 tests rejects_unsorted_table, rejects_duplicate_digests.
3. Signature cross-slot replay: a slot-0 signature must not verify in slot 1 (sig_input binds slot index + subtree name) — Task A4 test sig_input_binds_slot_and_name.
4. FS name attacks (/, \, NUL, ./.. in names, >4096-byte paths) — Task A1 tests + B4 unpack_rejects_bad_names via crafted node.
5. Tamper detection end-to-end: one flipped byte in the content region or superblock fails verify with the right §12 error — Task B5 tests tamper_content_detected, tamper_superblock_detected.
6. Truncated / size-mismatched file → SizeMismatch, not a panic — Task B4 test rejects_truncated_file.

---

Phase A — finish lttc-format (crate currently does not compile)

Audit result: lib.rs declares mod fs_node; and mod pointer_table; but the files were never committed. Type names are pinned by the existing pub use lines: DirEntry, FileEntry, NodeKind, PointerEntry, POINTER_ENTRY_SIZE. Existing modules (superblock, block_table, merkle, cid, error) look spec-conformant; verify with golden tests, don’t rewrite.

Task A1: fs_node.rs — FS-metadata nodes (§5.2)

Files: create crates/lttc-format/src/fs_node.rs; tests inline (#[cfg(test)] mod tests).

Interfaces:

• Consumes: crate::cid::Digest, crate::error::LttcError.

• Produces (pinned by lib.rs):

  • pub enum NodeKind { Dir = 0x01, File = 0x02 } (+ from_byte).
  • pub struct DirEntry { pub name: Vec<u8>, pub kind: NodeKind, pub mode: u32, pub child_digest: Digest } with encode(&self) -> Vec<u8> / decode(&[u8]) -> Result<(Self, usize), LttcError> (returns bytes consumed; node = entries back-to-back until buffer end).
  • pub struct FileEntry { pub name: Vec<u8>, pub mode: u32, pub uncompressed_size: u64, pub chunk_digests: Vec<Digest> } with the same encode/decode shape (kind byte always 0x02; chunk_count u32 = chunk_digests.len()).
  • pub fn validate_name(name: &[u8]) -> Result<(), LttcError> — UTF-8, non-empty, no /, \, NUL, not ./.., ≤ 4096 total path budget handled by caller; violation → MalformedFsEntry, over-long → PathTooLong.

• Byte layouts exactly §5.2: dir name_len u16 || name || kind u8 || mode u32 || child[32]; file name_len u16 || name || kind u8 || mode u32 || size u64 || chunk_count u32 || digests[32·n].

• Step 1: failing tests — golden bytes for one dir entry and one file entry (hand-computed offsets per layout above); decode round-trip; validate_name rejects a/b, a\\b, a\0b, ., .., non-UTF-8; decode of truncated buffer → MalformedFsEntry.

• Step 2: run cargo test -p lttc-format → fails (module missing; also fixes the crate-wide compile error).

• Step 3: implement fs_node.rs.

• Step 4: cargo test -p lttc-format → green.

• Step 5: commit feat(format): FS-metadata dir/file node encoding (§5.2).

Task A2: pointer_table.rs — pointer-table entry (§5.2)

Files: create crates/lttc-format/src/pointer_table.rs; inline tests.

Interfaces:

• Produces: pub const POINTER_ENTRY_SIZE: usize = 48; pub struct PointerEntry { pub node_digest: Digest, pub node_offset: u64, pub depth: u16, pub flags: u16 } with encode() -> [u8; 48] / decode(&[u8; 48]) -> Self; layout node_digest[32] || node_offset u64 || depth u16 || flags u16 || reserved u32.

• Table size caps as consts: POINTER_TABLE_MIN: usize = 2048, POINTER_TABLE_MAX: usize = 65536.

• Step 1: failing tests — golden bytes, round-trip, reserved u32 zeroed.

• Step 2: run tests → fail. Step 3: implement. Step 4: green.

• Step 5: commit feat(format): pointer-table entry encoding (§5.2).

Task A3: golden vectors for the existing modules (§3, §6.2, §5.2)

Files: create crates/lttc-format/tests/golden.rs.

Interfaces: consumes only the public API (Superblock, BlockTableEntry, MerkleNode, consts).

• Step 1: write tests/golden.rs: superblock with distinctive field values → assert exact byte offsets (magic@0, header_size=4096@16, major@20, minor@24, hash_alg@28=0x02, sig_alg@29=0x01, flags@32, manifest@40/48, block_table@56/64, [email protected], license@104/112, pointer@120/128, content@136, total@144, slot k sig@160+96k / key_id@224+96k, detached@544/552, tail 560..4096 zero); decode(encode(x)) == x; decode on bad magic → BadMagic; check_reserved_tail catches a single non-zero byte with correct offset; block-table entry 64 B golden; Merkle node 72 B golden (incl. depth/kind bytes at 64..67, reserved zero); FeatureFlags:: lowest_unknown_bit (bit 4 → Some(4), vendor bit 32 → Some(32), known bits → None); LttcError::exit_code mapping 1..17.
• Step 2: run → expect all green immediately (types already written); any failure is a real bug in the snapshot — fix the module, not the test.
• Step 3: commit test(format): golden byte vectors for frozen layouts.

Task A4: sig module — spec-contract crypto derivations (§5.3, §5.4)

Files: create crates/lttc-format/src/sig.rs; add mod sig; + re-exports in lib.rs; add ed25519-dalek = "3" to workspace deps and to lttc-format/Cargo.toml.

Interfaces:

• Produces (all pure, no I/O):

  • pub const SUPERNODE_CONTEXT: &str = "lttc-supernode-v1"; pub const SIG_CONTEXT: &str = "lttc-sig-v1"; pub const KEY_ID_PREFIX: &[u8] = b"lttc-key-v1"; pub const SLOT_CODE: u8 = 0; … SLOT_SBOM: u8 = 3;
  • pub fn root_entry(name: &str, digest: &Digest) -> Vec<u8> — name_len u16 || name_utf8 || digest[32].
  • pub fn super_node_digest(present_roots: &[(&str, Digest)]) -> Digest — keyed BLAKE3 over root_entry concatenation in the order given (callers pass the fixed [plugin, translations, themes] order).
  • pub fn key_id(pubkey: &[u8; 32]) -> Digest — blake3(KEY_ID_PREFIX || pubkey).
  • pub fn sig_input(slot: u8, subtree_name: &str, signed_prefix: &[u8], super_node: &Digest) -> Digest — keyed BLAKE3 per §5.4; panics-free: signed_prefix must be exactly SIGNED_PREFIX_LEN (else debug_assert + truncate-safe error is not possible for a pure fn → take &[u8; SIGNED_PREFIX_LEN]).
  • pub fn sign_slot(key: &SigningKey, slot: u8, subtree_name: &str, signed_prefix: &[u8; SIGNED_PREFIX_LEN], super_node: &Digest) -> (Signature, Digest) — returns (64-byte sig, key_id).
  • pub fn verify_slot(pubkey: &[u8; 32], slot: u8, subtree_name: &str, signed_prefix: &[u8; SIGNED_PREFIX_LEN], super_node: &Digest, sig: &Signature) -> bool.

• Step 1: failing tests — fixed test key (SigningKey::from_bytes(&[7u8;32])): sign/verify round-trip true; wrong message byte → false; sig_input_binds_slot_and_name: sig made for (0,“plugin”) does not verify as (1,“plugin”) nor (0,“themes”); key_id = hand-computed blake3(b"lttc-key-v1"||pubkey); super-node order sensitivity ([a,b] ≠ [b,a]) and omission sensitivity.

• Step 2: run → fail. Step 3: implement. Step 4: green.

• Step 5: cargo clippy --workspace --all-targets clean; commit feat(format): §5.3/§5.4 super-node, key_id, slot sign/verify derivations.

Phase B — writer + reader core (the graded milestone)

Task B1: workspace scaffolding + deps

Files: modify Cargo.toml (workspace deps: ed25519-dalek = "3", fastcdc = "5", hex = "0.4"); create crates/lttc-writer/{Cargo.toml,src/lib.rs}, crates/lttc-reader/{Cargo.toml,src/lib.rs} (stub libs, #![forbid(unsafe_code)], depend on lttc-format via { path = "../lttc-format" }; reader dev-depends on writer for round-trip tests).

• Step 1: stubs compile: cargo clippy --workspace --all-targets clean.
• Step 2: commit chore: scaffold lttc-writer and lttc-reader crates.

Task B2: lttc-writer — chunking + FS tree + block store

Files: create crates/lttc-writer/src/{chunk.rs,tree.rs,writer.rs}.

Interfaces:

• Consumes: lttc_format::* (entries, sig, consts), fastcdc::ronomon, blake3.

• Produces:

  • chunk.rs: pub fn chunk_digests(data: &[u8]) -> Vec<(Digest, usize, usize)> (digest, offset, length) via FastCDC::new(data, 8192, 16384, 65536); empty file → zero chunks (FileEntry with chunk_count = 0, size 0).
  • tree.rs: pub struct FsBuilder — add_file(rel_path: &str, mode: u32, data: &[u8]) -> Result<(), WriterError>, finish() -> Result<(Digest, BTreeMap<Digest, Vec<u8>>), WriterError> — root dir-node digest + every content-addressed block (file chunks and encoded dir nodes), deduped, ordered by digest. Internals: dirs as BTreeMap<Vec<u8>, Child> (sorted by raw name bytes); a dir node = its entries concatenated per §5.2, ascending raw name bytes: subdirs encode as DirEntry (kind 0x01, child_digest = the subdir node’s digest), files as FileEntry (kind 0x02, chunk digests inline). Node digest = BLAKE3 of its encoded bytes; empty dir = zero-entry node. Per-file binary Merkle trees are NOT emitted: §5.2’s FileEntry carries chunk digests inline and no file-root field exists (plan-scan ruling #2 — MerkleNode/duplicate-last stay pinned by A3 golden tests for a future pointer-table phase). Path budget: 4096 accumulated over components → PathTooLong; names validated via fs_node::validate_name → MalformedFsEntry.
  • writer.rs: pub struct PackOptions { pub signing_key: Option<ed25519_dalek::SigningKey>, pub manifest_bytes: Vec<u8> }, pub struct PackReport { pub root_digest: Digest, pub super_node: Digest, pub block_count: usize, pub file_count: usize, pub total_size: u64 }, pub fn pack_dir(input: &Path, out: &Path, opts: &PackOptions) -> Result<PackReport, WriterError>; pub fn pack_files(files: &[PackedFile], manifest: &[u8], out: &Path, opts: &PackOptions) -> Result<PackReport, WriterError> where pub struct PackedFile { pub path: String, pub mode: u32, pub data: Vec<u8> } (test/CLI entry point; pack_dir walks sorted, skips symlinks — v1 has none — and reads files).
  • WriterError: thiserror enum wrapping LttcError + std::io::Error (+ ManifestTooLarge enforced here too).

• Layout algorithm (writer.rs), canonical §2 order: superblock(4096) → manifest (uncompressed, ≤16 MiB) → block table → content blocks (each start 4 KiB-aligned, zero padding, padding NOT hashed). Pointer table & license region absent (offsets 0, flags 0). Blocks = dedup’d union of {file chunks, FS dir-node blobs} (no separate Merkle-internal-node blocks in v1-core — ruling #5); block-table entries: digest-sorted, length_uncompressed == length_on_disk (no compression), flags 0; block offsets absolute. Roots per convention #1: plugin = FS root digest; translations/themes from top-level /localization, /themes when present; super_node via sig::super_node_digest; all signature slots zero in B2 (the signing_key in PackOptions is carried but ignored — slot-0 signing is B3, ruling #4); feature_flags = 0 (single root) or MULTI_ROOT (roots.len() > 1); total_file_size = final size; fs_metadata_root_digest = FS root digest.

• Step 1: failing unit tests — chunk_digests on <8 KiB input = 1 chunk equal to blake3(data); on ~200 KiB pseudo-random input: chunks within [8 KiB, 64 KiB] except possibly last, offsets contiguous, cover input; determinism (same input twice → same digests); dedup: two identical files → one chunk-block set; empty file → zero chunks and FileEntry chunk_count = 0; add_file rejects bad names / over-long paths (plan-scan ruling #5: no per-file Merkle-tree tests here — trees are not emitted in v1-core, MerkleNode encoding stays pinned by A3 golden tests only).

• Step 2: run → fail. Step 3: implement chunk.rs + tree.rs. Step 4: green.

• Step 5: failing test for pack_files: produces a file whose first 16 bytes are MAGIC, size = total_file_size, block table sorted & unique (re-parse with lttc_format types only). Step 6: implement writer.rs. Step 7: green; commit feat(writer): deterministic pack — FastCDC chunks, FS Merkle metadata, §2 layout.

Task B3: writer signing integration

Files: modify crates/lttc-writer/src/writer.rs.

• Step 1: failing test — pack with fixed key: slot 0 non-zero, slots 1–3 zero, key_id = sig::key_id(pubkey); unsigned pack: all slots zero.
• Step 2: implement (encode superblock with zero sigs → take [0..160] → sig::sign_slot → re-encode with sig). Step 3: green.
• Step 4: determinism test — same tree+key twice ⇒ byte-identical files.
• Step 5: commit feat(writer): Ed25519 slot-0 signing over the signed prefix (§5.4).

Task B4: lttc-reader — canonical parse, verify, unpack

Files: create crates/lttc-reader/src/{parse.rs,verify.rs,unpack.rs}.

Interfaces:

• Consumes: lttc_format::* (incl. sig), blake3.

• Produces:

  • parse.rs: pub struct Container { /* superblock + mmap-less file handle */ } — pub fn open(path: &Path) -> Result<Container, ReaderError> running §3.2 steps 1–8 in order on the raw 4096-byte header + file size: magic → header_size (≥ 4096 known span, ≤ file size) → major == 1 → minor ≤ 0 → feature_flags.lowest_unknown_bit() → reserved tail (known minor) → total_file_size == actual (step 7; also reject actual < 4096 early) → region bounds & pairwise non-overlap using checked arithmetic (u64::checked_add; overflow ⇒ RegionOverlap); content region = [content_blocks_offset, total_file_size). pub fn block_table(&self) -> Result<Vec<BlockTableEntry>, ReaderError> — read + decode + one O(n) pass: strictly ascending digests, no dups (MalformedBlockTable), count ≤ 2²⁴ (TooManyBlocks), each block [offset, offset+length_on_disk) inside content region, uncompressed ⇒ length_uncompressed == length_on_disk. pub fn read_block(&self, digest: &Digest) -> Result<Vec<u8>, ReaderError> — binary search, bounds-checked, read, BLAKE3, mismatch ⇒ HashMismatch { digest }; length_uncompressed > 64 MiB ⇒ BlockTooLarge (checked before read); a block with the COMPRESSED flag is not supported by the core reader — return ReaderError::Unsupported("deflate blocks") (the flag bit is known per spec, so UnknownFeatureBit would be wrong; full deflate lands with Phase D).
  • verify.rs: pub fn verify(c: &Container, pubkey: &[u8; 32], slot: u8) -> Result<VerifyReport, ReaderError> — hash every block in the table (HashMismatch on any); rebuild FS root node from fs_metadata_root_digest, recompute every dir/file digest bottom-up, require equality with the stored root; recompute roots set (convention #1) → sig::super_node_digest; verify slot signature via sig::verify_slot over signed_prefix → false ⇒ SignatureInvalid { slot }; pub struct VerifyReport { pub super_node: Digest, pub block_count: usize }. pub fn verify_all_present_slots(c: &Container, keys: &[(u8, [u8;32])]) -> … thin wrapper (core: slot 0 only).
  • unpack.rs: pub fn unpack(c: &Container, dest: &Path) -> Result<u64, ReaderError> — walk FS tree from root (validating names via fs_node::validate_name on every entry — hostile node ⇒ MalformedFsEntry/PathTooLong; path budget 4096 accumulated), reassemble files from chunk digests via read_block (per-chunk hash check = lazy verification §6), create dirs, apply unix mode bits; refuse entries escaping dest (validated names make this structurally impossible — defense in depth: canonicalize check).
  • ReaderError: thiserror wrapper { #[from] LttcError, Io(std::io::Error), Unsupported(&'static str) } with exit_code() delegating to LttcError where present.

• Step 1: failing tests (dev-dep lttc-writer, tempdir via std::env::temp_dir + unique subdir — no extra deps): open rejects truncated file (SizeMismatch), bad magic; rejects_out_of_bounds_regions (hand-crafted header: manifest_offset+length overflow ⇒ RegionOverlap); rejects_unsorted_table / rejects_duplicate_digests (craft header + table bytes directly); verify on a writer-packed unsigned container with slot absent ⇒ SignatureInvalid-free path (verify only checks present slots; absent ⇒ Ok with note) — pinned: verify with explicit pubkey on absent slot ⇒ SignatureInvalid { slot: 0 }.

• Step 2: run → fail. Step 3: implement parse.rs. Step 4: green. Step 5: implement verify.rs + unpack.rs with tests. Step 6: green.

• Step 7: cargo clippy --workspace --all-targets clean; commit feat(reader): §3.2 canonical parse order, full verify, unpack.

Task B5: round-trip integration test (the graded gate)

Files: create crates/lttc-reader/tests/roundtrip.rs.

• Step 1: fixture builder — temp tree with: nested dirs (code/wasm/ components/main/component.wasm), empty dir, file > 64 KiB (multi-chunk), two files with identical content (dedup), unicode + space-containing names, 0-byte file.
• Step 2: the test: pack_dir (fixed key) → open → verify (slot 0, derived pubkey) → unpack to fresh dir → assert byte-equality of every file + same dir structure + modes; PackReport.root_digest equals recomputed root; pack again → byte-identical container (determinism §13); tamper_content_detected: flip one byte in the content region → verify ⇒ HashMismatch or SignatureInvalid; tamper_superblock_detected: flip a byte in 560..4096 ⇒ NonZeroReserved, flip total_file_size low byte ⇒ SizeMismatch/SignatureInvalid (assert it is an LttcError, code per §12).
• Step 3: run → expect green against B2–B4 (any failure = real bug, fix the crate). Step 4: cargo test --workspace + cargo clippy --workspace --all-targets -- -D warnings + cargo fmt --check all clean.
• Step 5: commit test(reader): pack → verify → unpack round-trip + tamper detection.

Task B6: docs & decisions

• OPEN-DECISIONS.md: open rounds for conventions #1–#2 (root composition, subtree names) with proposed answers; note ron 0.8 vs structural_names discrepancy for Phase C (§4.1 says 0.8.x; field exists only ≥ 0.9 — needs author decision: bump spec reference or pin an older serialization style).
• README status line: core writer/reader landed.
• Commit docs: record writer conventions as OPEN rounds; status.

Phase C — manifest (outline; detail when reached)

Typed Manifest (§4.2) + canonical RON serialization (§4.1 dialect — blocked on the ron-version OPEN above), TOML↔RON conversion, manifest hashed under /manifest/ per §4, MULTI_ROOT from roots.len(), per-subtree slots 1–2, signatures[] mirrors, issued_at/expires_at anti-downgrade.

Phase D — CLI + hardening (outline)

lttc-cli (clap 4): pack (TOML manifest in), unpack (TOML out), verify, inspect, sign (keyfile handling, slot selection), diff; exit codes = LttcError::exit_code() (§12/§13); deflate per-block (§7 + zip-bomb cap); pointer-table writer/reader with the §5.2 trust rule; detached-signature blob (§5.4 LTD1); license/SBOM region (§10.2); lttc-build build.rs integration; wasmtime target smoke test.

Status log

• 2026-10-17: audit done (crate incomplete: fs_node/pointer_table missing; deps verified against crates.io: ed25519-dalek 3.0, fastcdc 5.0, blake3 1.8.7, ron 0.8.1 lacks structural_names). Plan written. Phase A starting.