Every load-bearing architecture decision rests on unmeasured numbers, and the benchmark that would produce them is a 15-line stub that cannot run against the current code

Источник: dev/fera-research/docs/oc5/perf-claims-unmeasured-gate-is-stub.md (read-only; сгенерировано scripts/sync-docs.mjs)

Every load-bearing architecture decision rests on unmeasured numbers, and the benchmark that would produce them is a 15-line stub that cannot run against the current code

• kind: risk
• area: perf / architecture / process
• severity: mid
• evidence:
  • ~/dev/fera/project-design/context/ADR.md:61 — “The ~30% performance overhead of WASM is an accepted tradeoff for portability + sandboxing on the plugin tier only”
  • ~/dev/fera/project-design/context/ADR.md:123 (ADR-005 Context) — “Crossing a dylib boundary per-call is measurably more expensive than an in-process call”
  • ~/dev/fera/project-design/context/ADR.md:576-577 (ADR-014) — “Performance: criterion benchmarks, scoped per Engine module, run against a fixed benchmark scene”
  • ~/dev/fera/project-design/context/OQ.md:892-894 (OQ-024 final) — “every change lands only with a reproducible benchmark (CPU/RAM/energy/latency; idle/burst/load/hot-swap; 1/10/100/500 plugin scenarios) on target hardware — pre-registered-falsifiers style”
  • ~/dev/fera/project-design/context/OQ.md:882 — “Fork only after Phase 0 baseline + profiling”
  • ~/dev/fera/project-design/context/OQ.md:840 — “Fork stays gated behind Phase 0 (baseline metrics per OPTIM-IDEAS-4)”
  • ~/dev/fera/fera/tools/bench-scene/src/main.rs — 15 lines total; fn main() prints "fera-bench-scene: fera-scene-v0 generator skeleton"
  • grep across the ecosystem for criterion / [[bench]] / harness = false → zero hits; find -type d -name benches → none
  • only CI workflow in the ecosystem: ~/dev/fera/fera/.github/workflows/ci.yml — build/test/clippy/fmt only, no benchmark job, no perf artifact
  • ~/dev/fera/project-design/context/RESEARCH-OQ-003.md:117,128,140 — the codec tables are honestly marked **[est]** and drive the frozen scene’s sticker parameters (OQ.md:146-147)
  • ~/dev/fera/chroix-core/src/lib.rs:283-286 — assert!(next < CELL_BUDGET) where CELL_BUDGET = 128 (chroix-core/src/lib.rs:195)
  • ~/dev/fera/project-design/context/OQ.md:146 — frozen scene: “Chat list size | 500 chats (30 with unread)”

Substrate

The corpus has an unusually disciplined stated performance method and essentially no performance data.

Three unmeasured numbers are carrying architectural weight

1. ~30% WASM overhead (ADR.md:61) is a stated consequence of ADR-002 — the two-tier trust model. It is the justification for shipping first-party modules as native dylibs at all. If the real number is materially different (and WASM has narrowed the gap substantially since the Component Model landed), the cost/benefit of the whole first-party-native tier changes — and with it the entire panic-boundary problem in no-panic-containment-at-dylib-boundary.md and the reclamation problem in hazard-pointers-cannot-allow-dylib-unmap.md. Two of my high-severity findings rest on a premise nobody has measured.

2. “measurably more expensive” across a dylib boundary (ADR.md:123) is the entire rationale for ADR-005 — batch APIs plus the host-side LRU cache. ADR-005’s cache is, in turn, the unclassified plaintext message store identified in storage-seam-frozen-wrong-shape.md. So the causal chain is: an unmeasured number → a batch/cache design → a plaintext cache that the mandatory ADR-031 tier audit never classifies. Measure the number and it is possible the cache is not worth its exposure.

3. RESEARCH-OQ-003’s decode-throughput tables are explicitly **[est]** (RESEARCH-OQ-003.md:117,128,140) and are used to justify the frozen scene’s stress parameters, which are then cited as the basis for the benchmark everything else is gated on. Estimates presented as estimates are good practice — the problem is what is built on top of them, not that they are estimates.

The gate is a stub, and the stub cannot run against the current code

• No measurement infrastructure exists. Zero criterion, zero benches/ directories, zero benchmark step in the only CI workflow. ADR-014’s mandated methodology (criterion, per Engine module, against the fixed scene) is unimplemented.
• bench-scene is a skeleton. 15 lines that print a string. The “frozen” fera-scene-v0 — the artifact every performance decision in the project is defined relative to — does not exist.
• Even once implemented, it would panic. The frozen scene specifies a 500-chat list (OQ.md:146). Keyed cells are one-per-entry (“created on demand when a key appears in a collection”, chroix-core/src/lib.rs:302-305), the budget is 128 (chroix-core/src/lib.rs:195), and the guard is assert! — not a Result. The sanctioned benchmark scene exceeds the cell budget by ~3.9×, and sweep only retires keyed cells whose keys vanish, on the slow lane. So the first thing the Phase 0 baseline harness does is panic in the render loop. See no-panic-containment-at-dylib-boundary.md.

The circular dependency

OQ-024 makes the entire optimization program conditional on a Phase 0 baseline: the nylon-ring fork is “gated behind Phase 0 (baseline metrics)” (OQ.md:840), “Fork only after Phase 0 baseline + profiling” (OQ.md:882), and every change “lands only with a reproducible benchmark … pre-registered-falsifiers style” (OQ.md:892-894). Meanwhile OPTIM-IDEAS-3 proposes LTO, panic=abort, a custom allocator, PGO/BOLT and debloating — all premised on having a baseline to compare against.

So the program is gated on a baseline that is gated on a generator that does not exist. That is fine as a plan. It stops being fine the moment anyone cites a number from these documents as a result — which is exactly what a reader of ADR.md:61 will do.

Options (tradeoffs)

1. Land a minimal harness before anything else. bench-scene emits the frozen scene as a fixture; one criterion benchmark measures the dylib-boundary vs in-process call cost. That single measurement validates or kills ADR-005’s premise and gives OQ-024 a Phase 0 baseline. Cost: small — it is a fixture generator and a bench.
2. Mark the numbers as estimates until measured. ADR.md:61 and ADR.md:123 get [unmeasured — see OQ-006 phase 0] the way RESEARCH-OQ-003 already does honestly. Cost: zero. Value: stops an unmeasured number from being cited as a result. This is the cheapest and highest-value action here.
3. Re-derive the WASM-overhead number before committing to the first-party-native tier. Cost: a spike. If WASM is within a few percent, ADR-002’s tier split — and two of my high-severity findings’ premises — change materially.
4. Fix CELL_BUDGET before the harness. Raise it, or make the guard a Result. Otherwise the benchmark cannot run. Non-negotiable regardless of which option is chosen.
5. Do nothing now. Legitimate if the corpus’s estimates are understood as estimates. Not acceptable if ADR.md:61 keeps being cited as a measured tradeoff.

Proposed solution

• Option 2 immediately (zero cost, stops the miscitation), then option 1 to make the citations real.
• Option 4 before option 1 — a benchmark that panics is not a benchmark. CELL_BUDGET needs to either exceed the frozen scene or the guard needs to become a Result. Note StorageEngine already has a per-impl Error channel (storage/src/engine.rs:14 per oc4), so the discipline has a precedent in the codebase.
• Option 3 as a Phase 0 item, explicitly: measure WASM-vs-native on the real scene before the nylon-ring fork is gated on a baseline that assumes the tier split is already correct.
• Add a CI note that benchmarks run on target hardware only — the 1/10/100/500 plugin scenarios matrix (OQ.md:893) implies hardware variance, so results need to record the machine or they are not comparable across commits.

Framing for the merge: this is not a claim that the estimates are wrong. It is that the corpus already uses careful [est] labelling in RESEARCH-OQ-003 but not in ADR.md, while the unlabelled numbers sit higher in the document hierarchy and carry heavier architectural consequences.