RESEARCH — OQ-003: Sticker frame cache — raw vs. per-frame-compressed storage

Источник: dev/fera/project-design/context/RESEARCH-OQ-003.md (read-only; сгенерировано scripts/sync-docs.mjs)

RESEARCH — OQ-003: Sticker frame cache — raw vs. per-frame-compressed storage

Companion docs: OQ.md (OQ-003, OQ-006), TECHSPEC.md §4 (cache format), ADR.md (ADR-019 prefetch, frame-rate budgets in TECHSPEC §6), QA.md (media + frame-budget gates), RFC.md §4.6/§7.

Status: pre-bench analysis (2026-09-30). The benchmark scene (OQ-006) is not frozen and no implementation exists, so nothing here was measured on Fera hardware. Every number is either exact arithmetic (formula shown) or a literature-range estimate marked [est]. Purpose: fix a default now, with pre-registered falsifiers, so the bench (§5) can run immediately after OQ-006 freezes and either confirms or flips it.

1. Size arithmetic

1.1 One sticker, raw

Reference sticker (TECHSPEC §4 / Telegram animated-sticker profile): 512×512 RGBA, 30 fps, ~3 s ≈ 90 frames.

frame   = W × H × 4 = 512 × 512 × 4 = 1,048,576 B = exactly 1 MiB
sticker = frames × frame = 90 × 1 MiB ≈ 90 MiB

Header overhead is negligible: 4 + 2 + 2 + 8·N + 2·N bytes; at N = 90 that is 908 B (~0.001 % of payload). Format cost is not a factor.

corpus	raw @ 512²	raw @ 256² (¼ area)
1 sticker	90 MiB	22.5 MiB
50 concurrent (strawman scene)	4.4 GiB	1.1 GiB
100 stickers	8.8 GiB	2.2 GiB
1 sticker set (120)	10.5 GiB	2.6 GiB
500 stickers (one chat’s sticker history, plausible)	44 GiB	11 GiB

1.2 The source it caches

A .tgs file is gzipped Lottie JSON, typically 20–100 KiB; Telegram’s own animated-sticker requirements cap it around 64 KiB [est]. Blowup of a raw cache over its source:

90 MiB / 64 KiB ≈ 1440×

A user who has used three sticker sets carries ~32 GiB of cache. This is the decisive footprint fact: raw @ 512² cannot ship unbounded. It needs a quota + eviction policy (e.g. 1–2 GiB → 11–22 stickers at 512²), and a quota that tight means the cache covers only a small slice of the corpus — which undermines the cold-start benefit the cache exists for.

1.3 Per-frame compression, lossless

Expected ratios on 512² RGBA frames of Lottie-style flat vector art [est — must be measured on the real corpus, §5 M4]:

codec	ratio [est]	notes
deflate (zlib −6)	5–10×	strong ratio, slow decode (§2)
zstd −1	4–8×	decode cost ≈ level-independent (§2.2)
zstd −6…−19	6–12×	same decode speed, slower one-time build
LZ4 block	2–5×	no entropy stage; loves zero runs
Snappy	2–4×	dominated — see §2.2
RLE32 (4-byte pixel runs)	3–12×	content-dependent; worst case +0.2 %

Why sticker frames should compress well (hypotheses, verified by M4):

Large fully-transparent regions. Sticker art typically occupies a fraction of the canvas; alpha=0 zones become long zero runs, especially after canonicalizing RGB to 0 wherever alpha = 0 (lossless under premultiplied compositing — see §4).
Flat fills → long identical-pixel runs.
No sensor noise: vector art has near-zero entropy outside gradients and anti-aliased edges.

Two free multipliers that stay compatible with the TECHSPEC §4 header:

Frame dedup [est 10–30 % fewer unique frames]: Lottie “hold” frames decode to identical pixels; store the frame once and point several frame_offsets entries at it. This captures the only safe part of inter-frame redundancy. True inter-frame coding (delta frames) is rejected: O(1) random access by frame index is the whole point of the format.
Raster size: 256² is ¼ the bytes and matches actual message-list display size (~256–300 device px [est]) — but it is a quality decision (OQ-010 pixel-diff) as much as a size one. Kept as a bench variable, not a default (§4).

Compressed footprint at ~5× ratio: 90 MiB / 5 ≈ 18 MiB/sticker; 500 stickers ≈ 8.8 GiB; at 256² raster ≈ 2.2 GiB. Five times more corpus under the same disk quota and the same page cache.

2. Decompression cost

2.1 The budget, derived

Sticker path budget: 60–75 fps (TECHSPEC §6), its own frame loop, and it must not degrade the 24/45 fps paths (QA gate).

Stickers are 30 fps content; at a 60 fps display tick each visible sticker advances on every other tick. Strawman stress scene (50 concurrent):

advances/s              = 50 × 30            = 1500
required decode output  = 1500 × 1 MiB       ≈ 1.5 GB/s   (of decompressed bytes)

That second number is the key one: codec speeds are conventionally quoted as output-rate GB/s, so the comparison is direct. A realistic (non-stress) scene of ≤ 10 animating stickers needs only ≤ 0.3 GB/s.

Per-frame CPU at 1 MiB/frame, single thread, mid-tier x86 [est]:

codec	decode [est]	ms/frame	CPU @ 1500/s (stress)	CPU @ 150/s (5 stickers)
deflate (zlib)	150–400 MB/s	2.6–7.0	3.9–10.5 cores	0.4–1.1 core
zstd −1	1–2 GB/s	0.5–1.0	0.8–1.6 core	0.08–0.16 core
LZ4 block	3–7 GB/s	0.15–0.35	0.22–0.53 core	0.02–0.05 core
RLE32 scalar	2–8 GB/s	0.13–0.52	0.2–0.8 core	0.02–0.08 core
raw (mmap)	—	0	0	0

Deflate resuscitation via zlib-ng / libdeflate raises decode to ~0.7–1.2 GB/s [est] — which is exactly zstd-1’s territory at a worse ratio. Not worth it: zstd already occupies that point. (Note the asymmetry with OQ-013/Lattice, where deflate-per-block was chosen for ubiquity in an interop container read by foreign tools. The sticker cache is internal, rebuildable, keyed by content hash, and never leaves the machine — no ubiquity argument applies. Lattice’s choice does not set a precedent here.)

The structural observation: with 1 frame of look-ahead at 30 fps content, each frame has a 33 ms deadline, not a 16.7 ms tick — latency is a non-issue; this is purely a CPU-occupancy question. A small worker pool (2 threads) decouples codec speed from the display tick entirely: it must sustain 1.5 GB/s output in aggregate, i.e. zstd needs 0.8–1.6 cores of pool capacity (37–80 % of a 2-worker pool), LZ4 11–27 %, RLE 10–40 %, deflate 200–525 %. On a mid-tier 6–8 core machine, zstd is feasible but expensive; LZ4/RLE are comfortable; deflate is dead on arrival.

2.2 Codec-by-codec verdicts

deflate — rejected. Decode 3–10× too slow for the stress scene even realistically (0.4–1.1 core for 5 stickers); its ratio edge over zstd is small and decode-side cost dominates this design.
zstd −1 — viable with look-ahead. Ratio strong (4–8×), decode 1–2 GB/s, and a property that matters here: zstd decode speed is (nearly) independent of compression level — the level only prices the one-time cache build. Encode [est]: −1 ~300–500 MB/s, −3 ~150–300, −6 ~60–150, −19 ~10–30. A 90 MiB sticker compresses at −6 in ~0.6–1.5 s of background build time, once.
LZ4 — the safe fast choice. Ratio mediocre (2–5×), decode 3–7 GB/s; long zero-runs (transparent regions) hit its best case.
Snappy — dominated; expected to lose on both axes. zstd −1 beats it on ratio (4–8× vs 2–4×) and matches/beats it on decode (1–2 GB/s vs ~1–1.5 GB/s [est]); LZ4 beats it on speed outright. Keep one confirmation row in the microbench (M4), then drop it.
RLE32 — the special case this content invites. Format: control byte (run/literal flag, count ≤ 127) over 4-byte pixel units. Transparent and flat regions become runs; worst case (no runs at all) is +1 byte per 127 pixels = +0.2 % bloat. Scalar decode is memset/copy-bound (2–8 GB/s [est]) and SIMD-trivial. Risk: ratio collapses on gradient- or detail-dense stickers → mitigated by the per-frame codec tag (fall back to zstd, or to raw, per frame — §4).

2.3 Costs beyond the codec (the honest part of “raw is free”)

GPU texture upload: 1500 MiB/s at the stress scene (512²), format-independent — raw pays it too. That is ~10–12 % of usable PCIe 3.0 ×16 [est]. Raw’s “zero CPU” claim removes only the decompress slice, not the dominant per-frame copy.
DRAM traffic: ~3 GB/s aggregate (decompress writes + upload reads) at stress [est] — ~8 % of a mid-tier DDR4 machine. L3 pollution is bounded by a per-worker scratch ring (1–2 MiB double buffer).
Warm raw is only free while resident. The stress scene’s raw corpus is 4.4 GiB, which crowds an 8–16 GiB machine. When evicted, re-faulting 256 pages/frame from NVMe at QD1 4K [est 60–100 µs/page] costs 15–25 ms/frame — ~20–35× worse than a zstd decompress — unless saved by proactive madvise(WILLNEED) / PrefetchVirtualMemory prefetch of contiguous ranges. The compressed corpus for the same scene is ~0.9 GiB, comfortably resident, and its refaults cost 5× less. Under memory pressure, compressed-mmap beats raw-mmap; the bench must test this explicitly (M6).
Cold first-frame (the cache’s actual purpose): raw ≈ 0.4–1 ms (1 MiB page-in + 256 soft faults); zstd ≈ 0.3–0.8 ms (~0.2 MiB read + up to 1 ms decompress). Both are an order of magnitude under a live tlottie re-decode [est 2–10 ms/frame]. Parity on the metric OQ-003 names — compression does not meaningfully hurt cold start. Prefetching a whole sticker on scroll-in favors compressed even more: 18 MiB vs 90 MiB from disk.

2.4 A wording hazard in QA.md

The QA gate “re-displaying a previously cached animated sticker does not re-invoke the decoder” is satisfied by the compressed design only if “decoder” means the Lottie decoder (tlottie). A decompressor is re-invoked per access. The gate’s intent (no Lottie re-render) holds; the warm-CPU metric (M3) is what proves the distinction is immaterial. Record that reading when this OQ resolves.

3. Hybrids

H1 — promote cold→hot by rewriting raw on disk

Promotion cost: decompress + write 90 MiB ≈ 30–100 ms of SSD I/O + CPU [est]. Benefit: 0.15–1.0 ms saved per subsequent frame display.
Break-even: ~40–650 frame displays (codec- and SSD-dependent [est]) — reachable for stickers that stay on screen, but the same benefit is available from RAM (H2) without the write, the transient double disk usage, or the file-lifecycle complexity. Dominated.

H2 — compressed disk + bounded RAM decompress-ahead ring

Keep the next K frames (K = 2–4) decompressed per animating sticker, filled by a 2-thread worker pool against 30 fps deadlines; evicted when the sticker leaves view.
RAM: 50 × K MiB = 100–200 MiB at 512² (25–50 MiB at 256²).
This converts the synchronous per-tick codec budget (~160 µs/frame if only 4 ms of the tick is spare) into a worker-throughput budget (≥ 0.67 ms/frame per worker) — the structural move that makes zstd viable and LZ4 trivial.
This is also what “raw for hot” was trying to be, minus the disk.

H3 — raw files for in-view stickers, compressed for the rest

Dual representation: worst-of-both disk (90 + 18 MiB per sticker), or re-encode on demotion (write churn). Dominated by H2 — RAM is fungible, disk is not.

H4 (orthogonal) — VRAM loop-resident textures

Upload each frame once per loop-entry, keep all N textures GPU-resident: the only true zero-CPU steady state. Cost: 90 MiB VRAM per 512² sticker (4.4 GiB at stress — no); 22.5 MiB at 256² (1.1 GiB at stress — borderline; a VRAM-quota’d few). This is an engine-side decision, independent of the disk format; worth one bench row at 256² only.

The equivalence that collapses the hybrid space

“Raw on disk for hot stickers” ≈ “decoded frames in RAM”, because the OS page cache is RAM — except the disk copy is permanent, quota’d in GiB, and ~5× larger per sticker. The only thing disk uniquely provides is cold start after a process/system restart, and §2.3 shows compressed serves that at parity. Hence every hybrid that burns disk for hotness (H1, H3) is dominated by H2.

4. Recommendation

Default (pre-registered, to be confirmed by the bench): per-frame compressed storage — zstd at a low level (−1…−6 chosen by build-time budget; decode cost is level-independent) — plus a per-frame codec tag with raw fallback, frame dedup, alpha=0 canonicalization, a bounded RAM decompress-ahead ring (K = 2–4) on a small worker pool, and prefetch hints (madvise(WILLNEED) / PrefetchVirtualMemory) on the next frame’s byte range.

Rationale, in priority order:

Footprint is the binding constraint. 90 MiB/sticker is a ~1440× blowup over the 64 KiB source; unbounded raw is a support incident (disk fills), and a quota tight enough to make raw safe (1–2 GiB) covers so little corpus that the cache stops doing its job. Compression buys 5× at ≤ 1 core worst-case CPU.
The CPU cost fits, given the look-ahead structure. 1.5 GB/s required decode output at the stress scene vs 1–2 GB/s per zstd core on a 2-worker pool → 1.3–2.7× margin [est]. If the bench shows the margin too thin, LZ4 is the pre-registered fallback (4–9× margin) — the codec is one tag value, swappable per frame.
Cold-start parity (§2.3): both variants beat live re-decode by roughly an order of magnitude, so the cache’s purpose survives compression.
Under memory pressure compressed wins outright (§2.3): 5× smaller working set, proportionally cheaper refaults; raw’s steady state degrades to 15–25 ms/frame re-faults exactly when the machine is busiest.
Raw’s steady-state advantage — zero decompress CPU, i.e. ~0.35 core at stress / ~0.03 core realistically — is real but small, and it coexists with the same 1.5 GB/s GPU upload either way.

Not recommended: raw as a storage mode. Raw survives as codec tag 0 for incompressible frames (measured zstd ratio < ~1.05) — a per-frame escape hatch, not a policy. This is also the honest literal resolution of TECHSPEC §4’s <raw or per-frame-compressed RGBA frames> placeholder.

Pre-registered falsifiers — any one flips the default back toward raw (or toward 256²):

M3 (warm CPU): compressed variant misses the 60 fps frame-advance p99 budget on mid-tier hardware with the look-ahead ring in place.
M1 (footprint): measured corpus ratio < 2× (compression buys too little to justify the CPU).
M6 (pressure): compressed refault-under-pressure p95 exceeds raw by more than 2 ms/frame.
M2 (cold): compressed first-frame p95 exceeds raw by more than 2× (not expected, §2.3).

TECHSPEC §4 amendments to record once the bench confirms (all small, all backward-compatible with the written header):

File magic + format version before the header — currently absent, so a truncated/corrupt cache file is indistinguishable from a valid one. Cheap insurance for a rebuildable cache.
frame_codecs: [u8; frame_count] (0 = raw, 1 = zstd, 2 = LZ4, 3 = RLE32 …), so the per-frame fallback policy is data, not code paths.
frame_lengths: [u32; frame_count] — make compressed lengths explicit (offsets alone only imply them for fixed-size raw frames). A zero length + shared offset marks a dedup’d frame; resolve at load into an offset→frame map, O(N).
Frame blocks 64-byte-aligned (SIMD-friendly decode; prefetch hints take byte ranges, so page alignment is unnecessary).
State the canonical alpha representation (premultiplied vs straight) — it decides whether alpha=0 → RGB=0 canonicalization is exactly lossless and it fixes the GPU blend setup. Decide with the engine path.

Deliberately left to the bench: 512² vs 256² raster. 256² quarters footprint, upload and DRAM traffic, and matches actual message-list display size — but it is as much a quality decision (OQ-010 pixel-diff against the 512 reference) as a size one, and large views (emoji picker, fullscreen) would need live re-decode from the retained .tgs source (keep it: ~64 KiB [est] per sticker is free). Not adopted by default here.

5. Bench protocol (run after OQ-006 freezes)

Scene: reference the frozen fixture by exact name/version from /tools/bench-scene/ (per the OQ-006 resolution — never re-describe it inline). Until freeze, use the strawman (50 concurrently-animating stickers) for protocol dry-runs only, marked provisional.

Corpus: real Telegram sticker sample, ≥ 200 stickers [est], stratified: flat cartoon / gradient-heavy / high-detail / dense-motion; durations 0.5–3 s. This deliberately reuses the OQ-004 requirement of a real corpus, not synthetic — codec ratios on synthetic buffers lie. The corpus file list is part of the fixture.

Variants matrix:

id	variant
V0	raw @ 512²
V1	raw @ 256²
V2	zstd −1 @ 512² (sub-rows: −3, −6)
V3	zstd −1 @ 256²
V4	LZ4 @ 512²
V5	RLE32 @ 512² (scalar and SIMD rows)
V6	deflate-fast @ 512² (flate2 + zlib-ng) — reference only
V7	Snappy @ 512² — single confirmation row, expected to lose (§2.2)
V8	BC7 @ 512² — exploratory, lossy, flagged: fixed 4:1, zero CPU

  decode, GPU-native; only consumable by the GPU upload path (CPU
  compositing would need a BC7 decode); include only if the engine
  path is GPU-locked |

All compressed variants share: frame dedup ON, alpha canonicalization ON, per-frame codec-tag fallback, 64-B alignment. Report dedup savings and canonicalization effect as separate A/B columns — measure, don’t assume.

Hardware: mid-tier, mirroring OQ-011’s calibration philosophy (mid-tier, not strongest or weakest, reasonable power draw). Minimum: one Windows + one Linux run (the madvise vs PrefetchVirtualMemory paths differ — verify what memmap2 actually does on Windows before trusting the prefetch numbers). Report CPU model, RAM, disk (NVMe mandatory; one HDD run optional as a floor).

Metrics:

M1 footprint — total cache bytes for the corpus; per-sticker p50/p95; ratio vs raw; dedup %; canonicalization %. Gate [provisional]: ≤ 2 GiB / 100 stickers @ 512², ≤ 0.5 GiB / 100 @ 256².
M2 cold first-frame — process restart + page cache dropped (bench VM reboot; on Windows, OS-level cache eviction is not reliably scriptable — use a VM or a dedicated machine). p50/p95 of first frame and of “first loop complete”. Gate [prov]: p95 ≤ 5 ms either variant; compressed within 2× of raw.
M3 warm steady-state — 50 concurrent, 60 and 75 fps ticks, look-ahead ring on, 2 workers. Report frame-advance p99 end-to-end, decompress CPU-ms/s, dropped-tick count, and the QA cross-gate (UI-chrome 24 fps / message 45 fps paths unaffected). Gate [prov]: decompress ≤ 0.3 core sustained; zero budget misses at 60 fps.
M4 codec microbench — per-codec encode MB/s, decode GB/s, ratio on the corpus frames. Confirms or refutes every [est] in §2.
M5 GPU upload control — MiB/s and ms/frame for the upload path alone, format-independent; separates upload cost from codec cost in M3.
M6 memory-pressure ladder — run the scene while allocating 0/50/80/95 % of RAM; per-frame refault p95 and FPS degradation curves, raw vs compressed. This is raw’s last stand; measure it honestly.
M7 prefetch effectiveness — WILLNEED/PrefetchVirtualMemory hit rate; sync-fault vs async-fault counts after the hint.
M8 power (optional) — CPU package power during M3 if exposed; mid-tier-laptop thermals decide whether a full core of zstd is acceptable.

Decision rule (pre-registered): the default is the smallest-footprint variant that passes the M2 + M3 gates on mid-tier hardware. Among tied-footprint variants, prefer fewer moving parts (zstd over a custom RLE). Raw-as-mode is adopted only on a §4 falsifier. Record the outcome as an ADR addendum (sticker cache storage format) plus OQ-003 → Resolved → see ADR-XXX, mirroring the OQ-004/OQ-008 resolution style, and fold the TECHSPEC §4 amendments into the spec text in the same change.