No schema-migration policy exists, and ADR-024's N-1 rollback contradicts ADR-021's "a module swap does not imply a storage migration"

Источник: dev/fera-research/docs/oc1/017-no-schema-migration-policy-rollback-conflict.md (read-only; сгенерировано scripts/sync-docs.mjs)

No schema-migration policy exists, and ADR-024’s N-1 rollback contradicts ADR-021’s “a module swap does not imply a storage migration”

• kind: problem
• area: storage / modules
• evidence:
  • ADR.md:851-853 (ADR-021 Consequences) — “The on-disk data format is versioned independently of API editions — a module swap does not imply a storage migration, and vice versa.”
  • ADR.md:439-441 (ADR-024) — “Rollback: keep at most the N-1 module version on disk, and only while it remains host-compatible (per the manifest’s host.min/max_version); older versions are garbage-collected.”
  • ADR.md:452-454 — “Rollback can never strand the user on an incompatible module: the N-1 copy is validated against the current host before it’s trusted as a fallback.”
  • ADR.md:400-403 — the core module (all the code, including the storage layer) updates are restart-gated: “download → verify → replace on-disk file → apply on next launcher start”.
  • ADR.md:852-853 names the version axis but no ADR, OQ, TECHSPEC section or QA gate specifies how a schema version is stored, how a migration is discovered, when it runs, or what happens if it fails. Verified: grep for migration|migrate|schema version across the corpus → only ADR.md:480 (secret chats: “no key migration”), ADR.md:853, OQ.md:207, and OQ.md:695 (a migrations/ directory inside the Lattice container tree — unrelated).
  • TECHSPEC.md:454-456 — phase 1 is “raw StorageEngine/sanakirja (no encryption yet)”, i.e. the storage format is explicitly expected to change shape during development, with no migration story for the people doing it.
  • The StorageEngine trait has no migration hook and no transaction primitive: TECHSPEC.md:136-143 = get_messages, put_messages, get_outbox, “// …”.
  • Nothing specifies what happens to a database written by a beta build when the user runs a newer build after having downgraded — which ADR-024 makes routine (hourly update poll, ADR.md:427).
  • Repo reality: ~/dev/fera/fera/core/crates/storage = a StorageEngine contract + in-memory reference impl (commit 004540f) — so the on-disk engine has not been written yet. This is the cheapest moment in the project’s history to settle the policy.
• severity: high

Суть / Substance

The corpus has made two independent promises that cannot both hold:

1. ADR-021: a module swap never implies a storage migration; the data format versions independently of abi_version/api_edition/ module_version.
2. ADR-024: when a module update is bad, the host rolls back to N-1, and this must “never strand the user”.

If the core module update (restart-gated, ADR.md:400-403) carries a schema change — and it will, since the core module contains the storage layer and every new release adds fields — then rollback to N-1 means running old code against a new-schema store. ADR-021 says that situation “does not happen”; ADR-024 says it must be safe. Nothing says how.

The failure mode is the worst kind for an offline-first client where “local storage is the source of truth” (ADR.md:307-311): the user’s only copy of their history is in a store the current binary cannot read, and the previous binary has been garbage-collected or is itself incompatible. Message history is unrecoverable except by re-downloading everything, which for a large account may be impossible (Telegram does not let you re-download what you never fetched) — i.e. permanent data loss.

Three variants need answers, and none has one:

• Forward migration interrupted. New core module starts migrating, crashes halfway. Is the migration transactional (sanakirja is ACID/MVCC, so it can be) — and is the store then readable by the old binary? Only if the migration is version-stamped and the old binary refuses cleanly.
• Downgrade with a newer schema. See above.
• Module swap within one core-module version. A hot-swapped storage-sanakirja replacement (N vs N-1 both host-compatible) reading a store written by the other. ADR-005/ADR-014’s contract tests cover behaviour, not format — a contract suite cannot detect that one impl writes a field the other drops.

Варианты решения (tradeoffs)

• (A) Store a data-format version in the store itself; refuse to open on mismatch; never auto-migrate. Each schema change ships with an explicit forward migration, run at startup, transactional, version-stamped. Old binary + newer store = clean refusal with a clear message (“this data was written by a newer version”) rather than corruption. Rollback to N-1 is then only offered when the N-1 binary’s supported range includes the current store version — which ADR-024’s existing “validated against the current host” check can carry, if “host” is extended to include the data-format range. Cheapest correct option; requires a real “downgrade blocked” UX (and means rollback is not always available).
• (B) Always-downward-compatible schema (additive-only, never drop or rename; deprecate instead). Rollback always works; but schema cruft accumulates forever, and encryption-tier reclassification (ADR-031) or the index-format change (see finding 004) may force a non-additive change. Also blocks the (A)-style clean refusal.
• (C) Snapshot-and-retry: before any migration, snapshot the store (sanakirja checkpoint) and keep it; on failure, restore. Cheap with MVCC, needs a retention policy for snapshots (which is itself a cache/quota question — finding 013) and doubles on-disk usage transiently.
• (D) Copy-on-write store swap (new file, migrate, atomic rename). Clean rollback in principle; doubles disk during migration; cross-device rename is not atomic; and with the cache quota also in play, peak disk use is a real constraint.

Предлагаемое решение

(A) + (C), decided now while the reference impl is still in-memory.

1. New ADR “Data-format versioning, migration and rollback” specifying:
   • a data_format_version stored inside the store (not only in the manifest — the manifest can be rolled back, the store cannot);
   • an explicit supported range per binary, the same min/max shape as host.min_version (ADR.md:837), so ADR-024’s rollback validation can consult it;
   • migrations are transactional and forward-only, each shipped with the binary that needs it, run at startup before the store is used;
   • refusal, not best-effort, on version mismatch, with a UI message naming the version and offering update/rollback — never a partial read.
2. Amend ADR-021:851-853 to state the caveat explicitly: the independence of the axes does not extend to a schema change shipped in a core-module update; and amend ADR-024:452-454 so “host-compatible” for rollback includes the data-format range, i.e. rollback can be legitimately unavailable and that is a normal state with a defined UI.
3. Add the hooks to StorageEngine now: data_format_version(), migrate(to) (transactional), compact() (also needed by ADR-025’s TTL deletion), and transaction() as a primitive — otherwise every one of these decisions has to be re-argued against a trait that cannot express them (TECHSPEC.md:136-143).
4. Snapshot before migration (C), with the snapshot subject to the same quota policy as finding 013, and a “keep last successful snapshot” rule.
5. QA gate: “a store written by version N is either fully readable by the current binary or refused with a clear message — never partially read”, plus “downgrade from N+1 to N is offered iff the data-format ranges overlap, and the refusal path is exercised in a test”.
6. Note the cheap timing: this is free to fix while storage is an in-memory reference impl, and expensive the moment a real user has data.