gabriel / muse public
feat BREAKING snapshot dev
AI Agent gabriel · 143 days ago · Apr 26, 2026 · Diff

feat(snapshot): schema_version field + zstd at-rest compression

schema_version (int=1): - Added to SnapshotDict TypedDict and SnapshotRecord dataclass - Persisted in every new snapshot file; defaults to 1 when reading pre-migration files that lack the key - Intentionally excluded from the snapshot-ID hash — it is metadata, not content; changing schema_version never invalidates a snapshot ID - Constant _SNAPSHOT_SCHEMA_VERSION = 1 (bump only on breaking layout changes)

zstd at-rest compression: - Added zstandard>=0.22.0 to pyproject.toml core dependencies - _zstd_compress / zstd_decompress_if_needed helpers in store.py - _write_msgpack_atomic gains compress=True flag; write_snapshot passes it - Files are compressed only when packed msgpack exceeds _ZSTD_COMPRESS_THRESHOLD (8 KB) — tiny snapshots stay as raw msgpack, no overhead for small repos - Detection is self-describing via 4-byte zstd magic (\x28\xb5\x2f\xfd) so all existing uncompressed snapshot files remain readable without migration - _read_msgpack calls zstd_decompress_if_needed transparently before unpacking - Both raw GC snapshot reads in gc.py updated to decompress before unpackb

Indices dir: reserved; not yet used for snapshot indexing.

Tests: test_snapshot_schema_version_and_compression.py — 33 tests across unit, integration, E2E, stress, state, integrity, performance, security tiers

sha256:3a73d3b15384972008e0fb501f92506d480b1f493570cfeeee607996b71de437 sha
sha256:8834372cf47334de37bf920bdca52290b6816f0f3a2d8b67f27ba717ee1418b0 parent
+65 ~16 symbols
sha256:8bb16c25332657cb35c0a7dc388355b5c31a487bbd6c60fca88d0fafef514427 snapshot
+65
symbols added
~16
symbols modified
0
dead code introduced
Semantic Changes 81 symbols
~ tests/test_snapshot_schema_version_and_compression.py .py 59 symbols added
+ TestBackwardCompat class class TestBackwardCompat L334–365
+ test_mixed_compressed_uncompressed_in_same_dir method method test_mixed_compressed_uncompressed_in_same_dir L351–365
+ test_old_uncompressed_snapshot_still_readable method method test_old_uncompressed_snapshot_still_readable L335–349
+ TestCliCompression class class TestCliCompression L262–309
+ test_cli_commit_writes_compressed_snapshot method method test_cli_commit_writes_compressed_snapshot L263–309
+ TestCompressionRoundtrip class class TestCompressionRoundtrip L200–254
+ test_compressed_roundtrip_record_is_identical method method test_compressed_roundtrip_record_is_identical L225–235
+ test_large_snapshot_on_disk_is_zstd_compressed method method test_large_snapshot_on_disk_is_zstd_compressed L201–214
+ test_schema_version_survives_roundtrip method method test_schema_version_survives_roundtrip L248–254
+ test_small_roundtrip_record_is_identical method method test_small_roundtrip_record_is_identical L237–246
+ test_small_snapshot_on_disk_is_not_compressed method method test_small_snapshot_on_disk_is_not_compressed L216–223
+ TestConstants class class TestConstants L79–102
+ test_compress_threshold_is_positive method method test_compress_threshold_is_positive L95–97
+ test_compress_threshold_is_reasonable method method test_compress_threshold_is_reasonable L99–102
+ test_snapshot_schema_version_is_int method method test_snapshot_schema_version_is_int L80–83
+ test_snapshot_schema_version_is_one method method test_snapshot_schema_version_is_one L85–88
+ test_zstd_magic_is_correct method method test_zstd_magic_is_correct L90–93
+ TestIntegrity class class TestIntegrity L373–409
+ test_gc_finds_objects_in_compressed_snapshot method method test_gc_finds_objects_in_compressed_snapshot L398–409
+ test_tampered_compressed_manifest_is_rejected method method test_tampered_compressed_manifest_is_rejected L383–396
+ test_verify_snapshot_id_passes_after_compression method method test_verify_snapshot_id_passes_after_compression L374–381
+ TestPerformance class class TestPerformance L417–429
+ test_1000_file_roundtrip_under_2s method method test_1000_file_roundtrip_under_2s L418–429
+ TestSchemaVersionField class class TestSchemaVersionField L141–192
+ test_default_schema_version_is_one method method test_default_schema_version_is_one L142–145
+ test_from_dict_defaults_schema_version_for_missing_key method method test_from_dict_defaults_schema_version_for_missing_key L187–192
+ test_from_dict_reads_schema_version method method test_from_dict_reads_schema_version L182–185
+ test_from_msgpack_defaults_schema_version_for_old_files method method test_from_msgpack_defaults_schema_version_for_old_files L173–180
+ test_from_msgpack_reads_schema_version method method test_from_msgpack_reads_schema_version L165–171
+ test_schema_version_excluded_from_snapshot_id_hash method method test_schema_version_excluded_from_snapshot_id_hash L154–163
+ test_to_dict_includes_schema_version method method test_to_dict_includes_schema_version L147–152
+ TestSecurity class class TestSecurity L437–479
+ test_schema_version_cannot_alter_snapshot_id method method test_schema_version_cannot_alter_snapshot_id L460–466
+ test_symlinked_snapshot_dir_not_written method method test_symlinked_snapshot_dir_not_written L468–479
+ test_zstd_bomb_rejected method method test_zstd_bomb_rejected L438–458
+ TestStress class class TestStress L317–326
+ test_1000_file_snapshot_compress_decompress method method test_1000_file_snapshot_compress_decompress L318–326
+ TestZstdHelpers class class TestZstdHelpers L105–133
+ test_compress_is_smaller_than_input_for_repetitive_data method method test_compress_is_smaller_than_input_for_repetitive_data L130–133
+ test_compressed_output_starts_with_magic method method test_compressed_output_starts_with_magic L114–118
+ test_decompress_noop_on_empty method method test_decompress_noop_on_empty L126–128
+ test_decompress_noop_on_plain_bytes method method test_decompress_noop_on_plain_bytes L120–124
+ test_zstd_roundtrip method method test_zstd_roundtrip L106–112
+ _init_repo function function _init_repo L67–71
+ _make_snapshot function function _make_snapshot L57–64
+ _obj_id function function _obj_id L53–54
+ MAX_MSGPACK_BYTES import import MAX_MSGPACK_BYTES L38–38
+ SnapshotRecord import import SnapshotRecord L38–38
+ annotations import import annotations L28–28
+ compute_snapshot_id import import compute_snapshot_id L37–37
+ content_hash import import content_hash L45–45
+ datetime import import datetime L30–30
+ msgpack import import msgpack L34–34
+ pathlib import import pathlib L31–31
+ pytest import import pytest L35–35
+ read_snapshot import import read_snapshot L38–38
+ snapshot_path import import snapshot_path L38–38
+ time import import time L32–32
+ write_snapshot import import write_snapshot L38–38
~ muse/core/gc.py .py 1 symbol added, 6 symbols modified
+ _zstd_decompress_if_needed import import _zstd_decompress_if_needed L77–77
~ _collect_reachable_objects
~ _collect_reachable_snapshots
~ _commit_path
~ _snapshot_path
~ get_all_commits
~ read_snapshot
~ muse/core/store.py .py 5 symbols added, 8 symbols modified
+ _SNAPSHOT_SCHEMA_VERSION variable variable _SNAPSHOT_SCHEMA_VERSION L647–647
+ _ZSTD_COMPRESS_THRESHOLD variable variable _ZSTD_COMPRESS_THRESHOLD L657–657
+ _ZSTD_MAGIC variable variable _ZSTD_MAGIC L652–652
+ _zstd_compress function function _zstd_compress L660–674
+ zstd_decompress_if_needed function function zstd_decompress_if_needed L677–693
~ SnapshotDict
~ SnapshotRecord
~ from_dict
~ from_msgpack
~ to_dict
~ _read_msgpack
~ _write_msgpack_atomic
~ write_snapshot
~ pyproject.toml .toml 2 symbols modified
~ project
~ dependencies
← Older Oldest on dev
All commits
Newer → Latest on dev

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:3a73d3b15384972008e0fb501f92506d480b1f493570cfeeee607996b71de437 --body "your comment"