gabriel / muse public
refactor BREAKING shelf dev
AI Agent gabriel · 143 days ago · Apr 26, 2026 · Diff

refactor(shelf): migrate storage from shelf.json to per-entry msgpack layout

- Add shelf_dir() path helper to muse.core.paths (single source of truth for .muse/shelf/ location; shelf_json_path() marked deprecated) - Add shelf_entry_path(), write_shelf_entry(), read_shelf_entry(), list_shelf_entries(), delete_shelf_entry() to muse.core.store mirroring commit/snapshot layout: .muse/shelf/<algo>/<hex>.msgpack - write_shelf_entry: symlink-swap attack guard on .muse/shelf/ dir; atomic temp-rename write via _write_msgpack_atomic - list_shelf_entries: newest-first sort, skips corrupt/oversized files - Rewrite shelf.py _load_shelf/_save_shelf: _load_shelf now delegates to list_shelf_entries; _save_shelf removed entirely; all callers updated - GC _collect_shelf_objects: glob */*.msgpack instead of reading shelf.json - Update test_cmd_shelf.py: replace all _save_shelf / shelf.json references with write_shelf_entry / _load_shelf; 154 tests pass - Add test_shelf_msgpack_storage.py: 68 tests across 8 tiers (unit, integration, e2e, stress, state, integrity, performance, security)

sha256:8834372cf47334de37bf920bdca52290b6816f0f3a2d8b67f27ba717ee1418b0 sha
sha256:259e39800828109e5f314d63af912ee26e96f6dea92001ffa1a90d95612214c0 parent
+121 ~52 −14 symbols
sha256:d063fe93c3e86bf203ae98dc7f2e22920bbeab5ab0073d5dd2ac0a416d588d43 snapshot
+121
symbols added
~52
symbols modified
−14
symbols removed
0
dead code introduced
Semantic Changes 187 symbols
~ tests/test_shelf_msgpack_storage.py .py 105 symbols added
+ TestCliShelfSaveMsgpackLayout class class TestCliShelfSaveMsgpackLayout L468–558
+ test_drop_removes_msgpack_file method method test_drop_removes_msgpack_file L514–527
+ test_list_returns_saved_entry method method test_list_returns_saved_entry L529–542
+ test_pop_removes_msgpack_and_restores_file method method test_pop_removes_msgpack_and_restores_file L544–558
+ test_save_creates_msgpack_file method method test_save_creates_msgpack_file L471–485
+ test_save_does_not_create_shelf_json method method test_save_does_not_create_shelf_json L487–498
+ test_save_json_output_has_id method method test_save_json_output_has_id L500–512
+ TestDeleteShelfEntry class class TestDeleteShelfEntry L424–460
+ test_delete_existing_returns_true method method test_delete_existing_returns_true L427–431
+ test_delete_nonexistent_returns_false method method test_delete_nonexistent_returns_false L440–442
+ test_delete_one_leaves_others method method test_delete_one_leaves_others L444–453
+ test_delete_removes_file method method test_delete_removes_file L433–438
+ test_delete_twice_returns_false_second_time method method test_delete_twice_returns_false_second_time L455–460
+ TestEntryIdDerivation class class TestEntryIdDerivation L279–299
+ test_different_name_different_id method method test_different_name_different_id L287–290
+ test_id_has_sha256_prefix method method test_id_has_sha256_prefix L292–294
+ test_id_hex_is_64_chars method method test_id_hex_is_64_chars L296–299
+ test_same_content_same_id method method test_same_content_same_id L282–285
+ TestListShelfEntries class class TestListShelfEntries L375–421
+ test_empty_dir_returns_empty_list method method test_empty_dir_returns_empty_list L378–380
+ test_missing_shelf_dir_returns_empty_list method method test_missing_shelf_dir_returns_empty_list L382–386
+ test_no_shelf_json_created method method test_no_shelf_json_created L416–421
+ test_single_entry_returned method method test_single_entry_returned L388–394
+ test_sorted_newest_first method method test_sorted_newest_first L405–414
+ test_two_entries_returned method method test_two_entries_returned L396–403
+ TestMsgpackRoundTrip class class TestMsgpackRoundTrip L234–276
+ test_bool_fields_survive method method test_bool_fields_survive L259–263
+ test_empty_dict_domain_state_survives method method test_empty_dict_domain_state_survives L271–276
+ test_list_fields_survive method method test_list_fields_survive L265–269
+ test_nested_snapshot_survives method method test_nested_snapshot_survives L252–257
+ test_none_fields_survive method method test_none_fields_survive L245–250
+ test_string_fields_survive method method test_string_fields_survive L237–243
+ TestShelfDirPathHelper class class TestShelfDirPathHelper L166–187
+ test_does_not_create_directory method method test_does_not_create_directory L179–183
+ test_is_child_of_muse_dir method method test_is_child_of_muse_dir L175–177
+ test_name_is_shelf method method test_name_is_shelf L185–187
+ test_returns_dot_muse_shelf method method test_returns_dot_muse_shelf L169–173
+ TestShelfEntryPathHelper class class TestShelfEntryPathHelper L190–231
+ test_algo_extracted_from_prefix method method test_algo_extracted_from_prefix L202–210
+ test_different_ids_produce_different_paths method method test_different_ids_produce_different_paths L222–226
+ test_does_not_create_directory method method test_does_not_create_directory L228–231
+ test_extension_is_msgpack method method test_extension_is_msgpack L212–215
+ test_parent_is_shelf_dir method method test_parent_is_shelf_dir L217–220
+ test_sha256_path_shape method method test_sha256_path_shape L193–200
+ TestShelfStateMachine class class TestShelfStateMachine L631–672
+ test_empty_to_save method method test_empty_to_save L634–639
+ test_listing_after_no_writes_is_empty method method test_listing_after_no_writes_is_empty L659–662
+ test_overwrite_same_entry_is_stable method method test_overwrite_same_entry_is_stable L664–672
+ test_save_to_drop_to_empty method method test_save_to_drop_to_empty L641–646
+ test_two_saves_then_one_drop method method test_two_saves_then_one_drop L648–657
+ TestShelfStorageIntegrity class class TestShelfStorageIntegrity L680–747
+ test_corrupt_entry_skipped_by_list method method test_corrupt_entry_skipped_by_list L719–730
+ test_empty_msgpack_file_skipped_by_list method method test_empty_msgpack_file_skipped_by_list L732–737
+ test_file_path_encodes_entry_id method method test_file_path_encodes_entry_id L683–693
+ test_read_back_id_matches_filename method method test_read_back_id_matches_filename L695–704
+ test_tampered_bytes_causes_rejection method method test_tampered_bytes_causes_rejection L706–717
+ test_write_creates_no_temp_files method method test_write_creates_no_temp_files L739–747
+ TestShelfStoragePerformance class class TestShelfStoragePerformance L755–804
+ test_delete_entry_under_10ms method method test_delete_entry_under_10ms L797–804
+ test_list_50_entries_under_500ms method method test_list_50_entries_under_500ms L780–795
+ test_read_entry_under_10ms method method test_read_entry_under_10ms L769–778
+ test_write_entry_under_50ms method method test_write_entry_under_50ms L758–767
+ TestShelfStorageSecurity class class TestShelfStorageSecurity L812–872
+ test_entry_id_cannot_escape_shelf_dir method method test_entry_id_cannot_escape_shelf_dir L827–838
+ test_non_dict_msgpack_rejected_on_read method method test_non_dict_msgpack_rejected_on_read L853–862
+ test_oversized_msgpack_rejected_on_read method method test_oversized_msgpack_rejected_on_read L840–851
+ test_shelf_dir_not_traversable_via_list method method test_shelf_dir_not_traversable_via_list L864–872
+ test_symlinked_shelf_dir_rejected_on_write method method test_symlinked_shelf_dir_rejected_on_write L815–825
+ TestStressShelfStorage class class TestStressShelfStorage L566–623
+ test_100_entries_all_listable method method test_100_entries_all_listable L585–597
+ test_100_entries_all_written method method test_100_entries_all_written L569–583
+ test_concurrent_writes_no_corruption method method test_concurrent_writes_no_corruption L599–623
+ TestWriteReadRoundTrip class class TestWriteReadRoundTrip L307–372
+ test_basic_round_trip method method test_basic_round_trip L310–319
+ test_creates_algo_subdirectory method method test_creates_algo_subdirectory L330–335
+ test_creates_msgpack_file_at_correct_path method method test_creates_msgpack_file_at_correct_path L321–328
+ test_none_fields_preserved method method test_none_fields_preserved L337–343
+ test_read_nonexistent_returns_none method method test_read_nonexistent_returns_none L369–372
+ test_resumable_true_preserved method method test_resumable_true_preserved L345–350
+ test_tags_preserved method method test_tags_preserved L352–357
+ test_write_is_idempotent method method test_write_is_idempotent L359–367
+ _delete_shelf_entry function function _delete_shelf_entry L86–88
+ _init_repo function function _init_repo L95–112
+ _list_shelf_entries function function _list_shelf_entries L81–83
+ _make_entry_dict function function _make_entry_dict L115–149
+ _read_shelf_entry function function _read_shelf_entry L76–78
+ _shelf_dir function function _shelf_dir L59–62
+ _shelf_entry_path function function _shelf_entry_path L65–68
+ _write_object function function _write_object L152–158
+ _write_shelf_entry function function _write_shelf_entry L71–73
+ Any import import Any L47–47
+ Mapping import import Mapping L46–46
+ annotations import import annotations L39–39
+ blob_id import import blob_id L52–52
+ content_hash import import content_hash L52–52
+ fake_id import import fake_id L52–52
+ json import import json L41–41
+ long_id import import long_id L52–52
+ msgpack import import msgpack L49–49
+ object_path import import object_path L53–53
+ os import import os L42–42 ← moved from muse/cli/commands/shelf.py
+ pathlib import import pathlib L43–43
+ pytest import import pytest L50–50
+ threading import import threading L44–44
+ time import import time L45–45
~ muse/cli/commands/shelf.py .py 4 symbols added, 7 symbols removed, 13 symbols modified
− _SHELF_FILE variable variable _SHELF_FILE L105–105
− _SHELF_MAX_BYTES variable variable _SHELF_MAX_BYTES L104–104
− _save_shelf function function _save_shelf L322–343
− assert_not_symlink import import assert_not_symlink L97–97
− load_json_file import import load_json_file L84–84
− os import import os L77–77 → moved to tests/test_shelf_msgpack_storage.py
− tempfile import import tempfile L80–80
+ _delete_shelf_entry import import _delete_shelf_entry L93–93
+ _list_shelf_entries import import _list_shelf_entries L93–93
+ _read_shelf_entry import import _read_shelf_entry L93–93
+ _write_shelf_entry import import _write_shelf_entry L93–93
~ _load_shelf
~ _shelf_pop_programmatic
~ _shelf_push_programmatic
~ JsonValue
~ Manifest
~ content_hash
~ get_head_snapshot_manifest
~ now_utc_iso
~ read_current_branch
~ sanitize_display
~ run_drop
~ run_pop
~ run_save
~ muse/core/gc.py .py 1 symbol added, 3 symbols removed, 8 symbols modified
− _MAX_SHELF_BYTES variable variable _MAX_SHELF_BYTES L84–84
− _shelf_json_path import import _shelf_json_path L67–67
− load_json_file import import load_json_file L66–66
+ _shelf_dir import import _shelf_dir L67–67
~ _collect_shelf_objects
~ _commits_dir
~ _heads_dir
~ _remotes_dir
~ _snapshots_dir
~ _tags_dir
~ long_id
~ short_id
~ muse/core/paths.py .py 1 symbol added, 1 symbol modified
+ shelf_dir function function shelf_dir L130–141
~ shelf_json_path
~ muse/core/store.py .py 6 symbols added, 9 symbols modified
+ delete_shelf_entry function function delete_shelf_entry L2312–2331
+ _shelf_dir import import _shelf_dir L175–175
+ list_shelf_entries function function list_shelf_entries L2279–2309
+ read_shelf_entry function function read_shelf_entry L2250–2276
+ shelf_entry_path function function shelf_entry_path L1374–1400
+ write_shelf_entry function function write_shelf_entry L2215–2247
~ _commits_dir
~ _head_path
~ _heads_dir
~ _ref_path
~ _releases_dir
~ _remotes_dir
~ _repo_json_path
~ _snapshots_dir
~ _tags_dir
~ tests/test_cmd_shelf.py .py 4 symbols added, 4 symbols removed, 21 symbols modified
− test_assert_not_symlink_in_load method method test_assert_not_symlink_in_load L366–368
− test_fsync_called_in_save method method test_fsync_called_in_save L362–364
− test_multiple_entries_ordered method method test_multiple_entries_ordered L370–381
− test_save_creates_shelf_json method method test_save_creates_shelf_json L277–287
+ test_fsync_called_in_write method method test_fsync_called_in_write L367–372
+ test_multiple_entries_sorted_newest_first method method test_multiple_entries_sorted_newest_first L380–392
+ test_save_creates_msgpack_file method method test_save_creates_msgpack_file L277–288
+ test_symlink_guard_in_write method method test_symlink_guard_in_write L374–378
~ TestDataIntegrity
~ test_content_address_id_matches_recomputed
~ test_deleted_paths_tracked
~ test_snapshot_contains_all_tracked_files
~ test_snapshot_id_is_sha256_prefixed
~ TestLoadSaveShelf
~ test_roundtrip_preserves_all_fields
~ test_save_is_atomic_no_temp_files
~ TestObjectIntegrity
~ test_apply_with_missing_object_exits_3
~ test_drop_succeeds_even_with_missing_objects
~ test_pop_with_missing_object_exits_3
~ TestPerformance
~ test_list_50_entries_under_2s
~ TestSecurity
~ test_ansi_in_branch_name_sanitized
~ test_ansi_in_file_path_sanitized_in_read
~ test_ansi_in_intent_sanitized
~ test_snapshot_values_must_be_strings
~ TestStress
~ test_save_load_large_snapshot
← Older Oldest on dev
All commits
Newer → Latest on dev

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:8834372cf47334de37bf920bdca52290b6816f0f3a2d8b67f27ba717ee1418b0 --body "your comment"