The determinism seam: inject Clock + entropy from day one, then get whole-app trace replay for free

Источник: dev/fera-research/docs/oc4/idea-02-determinism-seam-trace-replay.md (read-only; сгенерировано scripts/sync-docs.mjs)

The determinism seam: inject Clock + entropy from day one, then get whole-app trace replay for free

• kind: idea
• area: architecture / testing / tooling
• evidence:
  • ~/dev/fera/fera/core/crates/domain/src/outbox.rs:129,154,180 — let now = Instant::now(); called directly in the outbox retry scheduler (three sites)
  • ~/dev/fera/project-design/context/QA.md §Protocol layer — “reconnect backoff includes jitter; a test simulates N=100 simultaneous reconnect events and confirms requests are not synchronized to a single instant” (unreproducible: jitter reads the real RNG and the real clock)
  • ~/dev/fera/project-design/context/QA.md cross-cutting gate — the only offline-first gate is “a test that runs the app with network access blocked” (one scenario, binary on/off)
  • ~/dev/fera/project-design/context/PREDICT.md:220 — §8 Evaluation: “Replay harness (deterministic, seedable) so model changes are testable without live use” — planned, but scoped only to the predictor
  • ~/dev/fera/project-design/context/PREDICT.md §7 — “log the feature snapshot with the prediction, attach the label when the window closes; computing features at label time leaks the future” — an append-only, delayed-label log is already the design
  • ~/dev/fera/project-design/context/PREDICT.md:42 — Type A “System models … deterministic, explainable, testable, no training data needed”
  • ~/dev/fera/project-design/context/FUTURE-CHECKS.md §3 — the entire planned benchmark pass (sanakirja throughput, milli query latency, module-boundary overhead, sticker decode) is unspecified as to run-to-run variance control
  • ~/dev/fera/project-design/context/ADR.md ADR-015 — typestate/newtype discipline applied to IDs and states, but nothing to time or randomness
  • ~/dev/fera/project-design/context/ADR-016 — Argon2id cost parameters; QA.md requires them “documented with the rationale for the chosen cost” — which is unmeasurable without an injectable cost/time knob
  • ~/dev/fera/fera/tools/bench-scene/src/main.rs:1 — “the fera-scene-v0 generator (seeded, deterministic)” — the author already builds determinism into fixtures; the app itself has none
  • ~/dev/fera/project-design/context/NAMING-UX.md §1.5 — in-app bug-report button + self-hosted Sentry planned for ЗАТ

Суть / Thesis

Every source of nondeterminism in the system should reach the code through a seam, and there should be exactly two of them: time and entropy.

Today the system has neither. The evidence is a single line — outbox.rs reads the wall clock directly, three times, inside the retry scheduler. Multiply that by every Instant::now() in reconnect backoff, every rand::thread_rng() jitter draw, every Argon2 parameter, every nonce generation, every worker-pool size derived from available_parallelism(), and the consequence is:

Fera, as designed, cannot be tested end to end. ADR-014 lists six test suites (contract / actor / Elm / module-loading / visual / perf). Five of them test components. The sixth — QA.md’s “block the network and confirm cached history still renders” — is a binary switch on the one property the project names as its primary differentiator (ADR-010, RFC §1.2: official clients fail at exactly this and it is “a first-class design requirement”).

And the same missing seam blocks four other things the project has already committed to:

Committed thing	Blocked by
QA: “N=100 reconnect events are not synchronized to a single instant” (jitter test)	jitter reads real RNG + real clock
PREDICT §8: deterministic seedable replay harness	no injected clock/seed; actor scheduling is real
FUTURE-CHECKS §3: the whole benchmark pass	no variance control — numbers are not comparable run-to-run
QA: “Argon2id parameters documented with rationale for chosen cost”	a 200 ms KDF cannot be benchmarked without a cost knob

The key move: PREDICT.md already asks for the replay harness. It is scoped to predictor evaluation because that’s the only consumer anyone could name at the time. But a deterministic replay harness is not a predictor feature — it is a property of the whole application, and it only exists if the clock and the RNG are injectable from the first commit. Built later it is a project-wide sed of every timing-dependent line. Built now it is one trait.

Варианты / Options (with tradeoffs)

A. Status quo — real clock, real RNG.

• Zero up-front cost.
• Cost: end-to-end tests stay coarse; “runs sometimes” flakiness; benchmark deltas indistinguishable from noise; replay harness impossible; a user bug report is a stack trace with no way to reproduce it.

B. Inject only inside the outbox / protocol layer (local fix).

• Fixes the jitter test. Small.
• Cost: leaves 90% of timing surface unaddressed; the seam is half a seam, and the second half is a larger refactor than the first. Partial seams get abandoned.

C. Two root seams, plumbed from the start (proposed).

• trait Clock — now() -> Instant, now_wall() -> DateTime<Utc>, sleep_until(deadline) (async). Implementations: SystemClock (tokio), TestClock (virtual time, auto-advancing when all tasks are blocked — the FoundationDB/Tokio-test pattern), TraceClock (steps to the next recorded timestamp). One field in root state. Every actor gets it via the supervisor; no actor calls tokio::time::sleep, Instant::now, or SystemTime::now directly.
• trait Entropy — fill(&mut [u8]). Implementations: SystemEntropy (OsRng), SeededEntropy (ChaCha20 stream, per-actor substream derived from a trace seed + actor id, so actor scheduling order doesn’t change the draw). Used by: backoff jitter, nonce generation, ID minting, temp-file names, Argon2 salt, and — importantly — the Argon2 cost parameter is a constructor argument, not a constant, so KDF cost becomes a bench axis.

Tradeoff table:

Property	A (now)	C (proposed)
Code volume	0	~1 trait + 3 impls + ~30 mechanical call-site edits
Every future test	flaky/weak	deterministic
Refactor cost later	—	project-wide, grows with codebase
Review burden	none	a lint-able rule: “no Instant::now outside SystemClock” (deny-by-default via a workspace lint)

Proposed solution

1. Make the seams real and lint-enforced from the first commit

A workspace lint (clippy::disallowed_methods or an internal lint) banning Instant::now, SystemTime::now, tokio::time::sleep*, thread_rng, and available_parallelism outside the seam modules. This is the difference between a seam and a suggestion: it makes the leak a compile error, so the property cannot erode over the ~2 years the docs say this project will run.

2. Name the payoff early so it gets funded: fera-trace

With the seams in place, one idea unlocks four committed-but-blocked items:

A trace is a recorded sequence of external inputs — MTProto updates received, user actions (Msg), network conditions, and the clock readings they arrived at — plus the RNG seed. Replaying it feeds exactly those inputs back and produces a bit-identical run.

Because a trace records only inputs, it is small, and it is safe to attach to a bug report: the user’s message content need not be in it if the trace records chat_id + a content hash, with the payload optionally captured locally and never transmitted (or captured as a separate, user-consent-gated enclave).

This one artifact serves:

1. QA’s offline-first gate, generalized. “Block the network” becomes one script among many: drop-after-40%, 300 ms ± 120 ms RTT, 5% loss, DC switch mid-write, kill at outbox-pending, reconnect storm of 100. ADR-010’s promise becomes a parameterized test matrix, not one test.
2. Bug reports (NAMING-UX §1.5). An in-app “export replay” button. A trace reproduces the bug deterministically — categorically better than the stack-trace-plus-Sentry design currently planned, and it means the ЗАТ friends circle can report real bugs rather than “it crashed once”.
3. PREDICT §8’s replay harness — same machinery, predictor-specific assertions layered on. The predictor’s planned replay harness is this, scoped up. And PREDICT §7’s delayed-label log is naturally expressed as a trace with an evaluation window.
4. Fork/upgrade verification (see idea-01-fork-sync-xtask.md): a recorded trace corpus becomes the semantic-conformance corpus a fork bump is run against — the concrete form of ADR-014’s “contract tests are the load-bearing safety net” applied upstream.

3. On container format — and an honest scope note

The natural temptation is to reuse .lttc. Lattice’s own docs push back: NAMING-UX.md §1 scopes Lattice to “plugins/runtimes only”, and RESEARCH-OQ-003.md §2.2 makes the principle explicit —

“deflate-per-block in Lattice was chosen for ubiquity in an interop container read by foreign tools, whereas the sticker cache is internal, rebuildable, content-hash-keyed, never leaves the machine — ‘Lattice’s choice does not set a precedent here.’”

That distinction is worth promoting to a general rule, and it decides this case: a trace is internal, never leaves the machine, and is read only by Fera. So it should be a small sibling format with the same discipline (content-addressed blocks, BLAKE3, deterministic serialization) but tuned for what traces need — appendability during capture, and fast seek on replay — rather than for foreign-tool interop. Reusing .lttc would need an explicit OQ-013 scope amendment; proposing it here as a pattern costs nothing and breaks no decision.

4. Sequence

• Now, before more subsystems land: the two traits, the three Clock/two Entropy impls, the lint. This is a few hundred lines and it is the single cheapest high-value thing in this document.
• Next: convert QA’s existing actor tests to TestClock (they are currently time-dependent by accident — a latent flake source).
• Then: trace record/replay for the protocol + domain layers only. That is where every listed payoff lives; UI and media traces can come later or never.
• Explicitly not doing: simulating the network at packet level. The seam is at the protocol actor’s input, not the socket — MTProto is a stateful authenticated protocol and faking it below grammers buys nothing.

Why this is the highest-leverage idea in this document

Everything else in the project is either architecture (settled, 31 ADRs) or optimization (measurable, OQ-024 + fera-scene-v0). The one class of work that is both cheap and unblockable-by-nobody-else is test infrastructure — and the project’s own quality bar (“tests pin behavior; benchmarks prove optimizations”, community/README.md) is currently unreachable end-to-end because of two missing traits. Retrofit cost grows with codebase size, and this project is explicitly designed to live for years and churn constantly (ADR-001’s own Context).