gabriel / muse public
feat patch object-store task/object-store-algo-dir #1 / 2
AI Agent gabriel · 160 days ago · Apr 18, 2026 · Diff

feat(object-store): algo-dir layout + iter_stored_objects canonical walker

- object_path() now returns objects/sha256/<shard>/<rest> — algorithm is a directory component, mirroring the sha256: prefix on object IDs - Add iter_stored_objects() — single canonical walker replacing six inline iterdir loops across gc, maintenance, verify_object, prune, count_objects - _object_path_with_fallback() retains 2↔4-char shard-prefix migration only; legacy no-algo-dir layout support removed entirely - cleanup_stale_object_temps() updated for the 3-level structure - Fix restore_object empty-object comparison (sha256: prefix was missing) - All six callers (gc, maintenance, verify_object, prune, count_objects ×3) replaced with iter_stored_objects; no inline iterdir loops remain - Test helpers that manually constructed legacy paths updated to use write_object/object_path; legacy test classes removed

sha256:41df9d174a062c036619a5afbd7f4653788d8b2db013ce19213d2ece2c47b0f4 sha
sha256:61bae996bc0a7c2313d680a6cd078ff5c84e8c5db19f6ca8d7a2312a6fc09b97 parent
+222 ~32 symbols
sha256:e854e88b072a276a8ed1c1d42f3e9e2ea2d80ee63a7cf0d7e0bd325bc4560730 snapshot
+222
symbols added
~32
symbols modified
0
dead code introduced
Semantic Changes 254 symbols
~ tests/test_object_store_algo_layout.py .py 36 symbols added
+ TestCleanupStaleTempNewLayout class class TestCleanupStaleTempNewLayout L224–243
+ test_cleanup_finds_temps_in_sha256_shards method method test_cleanup_finds_temps_in_sha256_shards L227–243
+ TestFallbackShardPrefix class class TestFallbackShardPrefix L192–216
+ test_finds_canonical_layout method method test_finds_canonical_layout L195–202
+ test_has_object_canonical method method test_has_object_canonical L204–209
+ test_read_object_canonical method method test_read_object_canonical L211–216
+ TestIterStoredObjectsNewLayout class class TestIterStoredObjectsNewLayout L110–184
+ test_empty_store_yields_nothing method method test_empty_store_yields_nothing L113–115
+ test_ids_are_sha256_prefixed method method test_ids_are_sha256_prefixed L146–151
+ test_no_duplicates method method test_no_duplicates L153–160
+ test_skips_non_hex_filenames method method test_skips_non_hex_filenames L175–184
+ test_skips_symlinks method method test_skips_symlinks L162–173
+ test_yielded_path_exists method method test_yielded_path_exists L127–133
+ test_yields_all_objects method method test_yields_all_objects L135–144
+ test_yields_written_object method method test_yields_written_object L117–125
+ TestObjectPathAlgoDirectory class class TestObjectPathAlgoDirectory L46–102
+ test_algo_directory_is_inside_objects method method test_algo_directory_is_inside_objects L57–62
+ test_filename_is_remaining_hex method method test_filename_is_remaining_hex L73–80
+ test_four_char_prefix_still_nested_under_sha256 method method test_four_char_prefix_still_nested_under_sha256 L82–92
+ test_path_contains_sha256_directory method method test_path_contains_sha256_directory L49–55
+ test_shard_prefix_still_correct method method test_shard_prefix_still_correct L64–71
+ test_write_object_lands_in_sha256_dir method method test_write_object_lands_in_sha256_dir L94–102
+ _repo function function _repo L36–38
+ _object_path_with_fallback import import _object_path_with_fallback L19–19
+ annotations import import annotations L12–12
+ blob_id import import blob_id L18–18
+ cleanup_stale_object_temps import import cleanup_stale_object_temps L19–19
+ has_object import import has_object L19–19
+ iter_stored_objects import import iter_stored_objects L19–19
+ object_path import import object_path L19–19
+ objects_dir import import objects_dir L19–19
+ os import import os L246–246
+ pathlib import import pathlib L14–14
+ pytest import import pytest L16–16
+ read_object import import read_object L19–19
+ write_object import import write_object L19–19
~ muse/cli/commands/count_objects.py .py 26 symbols added
+ _CountObjectsJson class class _CountObjectsJson L93–102
+ _ShardInfo class class _ShardInfo L87–90
+ _collect_reachable_ids function function _collect_reachable_ids L163–217
+ _count_by_shard function function _count_by_shard L135–160
+ _count_loose_objects function function _count_loose_objects L110–132
+ ExitCode import import ExitCode L70–70
+ TypedDict import import TypedDict L67–67
+ _json import import _json L60–60
+ annotations import import annotations L57–57
+ argparse import import argparse L59–59
+ deque import import deque L66–66
+ get_head_commit_id import import get_head_commit_id L73–73
+ iter_stored_objects import import iter_stored_objects L71–71
+ logging import import logging L61–61
+ long_id import import long_id L69–69
+ objects_dir import import objects_dir L71–71
+ os import import os L62–62
+ pathlib import import pathlib L63–63
+ read_commit import import read_commit L73–73
+ read_snapshot import import read_snapshot L73–73
+ require_repo import import require_repo L72–72
+ sys import import sys L64–64
+ time import import time L65–65
+ logger variable variable logger L79–79
+ register function function register L225–256
+ run function function run L264–322
~ muse/cli/commands/maintenance.py .py 38 symbols added
+ _ALL_TASKS variable variable _ALL_TASKS L121–121
+ _CONFIG_FILE variable variable _CONFIG_FILE L124–124
+ _DEFAULT_PERIOD_HOURS variable variable _DEFAULT_PERIOD_HOURS L123–123
+ _DEFAULT_TASKS variable variable _DEFAULT_TASKS L122–122
+ _MaintenanceErrorJson class class _MaintenanceErrorJson L163–167
+ _MaintenanceRunJson class class _MaintenanceRunJson L132–140
+ _MaintenanceScheduleJson class class _MaintenanceScheduleJson L154–160
+ _MaintenanceStatusJson class class _MaintenanceStatusJson L143–151
+ _TASK_RUNNERS variable variable _TASK_RUNNERS L297–300
+ _cmd_run function function _cmd_run L303–376
+ _cmd_schedule function function _cmd_schedule L408–441
+ _cmd_status function function _cmd_status L379–405
+ _config_path function function _config_path L175–177
+ _emit_error function function _emit_error L213–227
+ _now_iso function function _now_iso L204–205
+ _read_config function function _read_config L180–197
+ _run_gc function function _run_gc L235–252
+ _run_verify_objects function function _run_verify_objects L255–289
+ _write_config function function _write_config L200–201
+ Any import import Any L107–107
+ ExitCode import import ExitCode L110–110
+ TypedDict import import TypedDict L107–107
+ _DEFAULT_GRACE_PERIOD_SECONDS import import _DEFAULT_GRACE_PERIOD_SECONDS L111–111
+ _json import import _json L103–103
+ annotations import import annotations L100–100
+ argparse import import argparse L102–102
+ iter_stored_objects import import iter_stored_objects L112–112
+ load_json_file import import load_json_file L109–109
+ logging import import logging L104–104
+ long_id import import long_id L109–109
+ now_utc_iso import import now_utc_iso L109–109
+ require_repo import import require_repo L113–113
+ run_gc import import run_gc L111–111
+ sys import import sys L105–105
+ time import import time L106–106
+ logger variable variable logger L115–115
+ register function function register L449–538
+ run function function run L546–548
~ muse/cli/commands/prune.py .py 20 symbols added
+ _collect_all_reachable_ids function function _collect_all_reachable_ids L99–116
+ _find_prune_candidates function function _find_prune_candidates L119–157
+ ExitCode import import ExitCode L86–86
+ _collect_reachable_objects import import _collect_reachable_objects L87–87
+ _json import import _json L78–78
+ annotations import import annotations L75–75
+ argparse import import argparse L77–77
+ iter_stored_objects import import iter_stored_objects L88–88
+ logging import import logging L79–79
+ long_id import import long_id L85–85
+ objects_dir import import objects_dir L88–88
+ os import import os L80–80
+ pathlib import import pathlib L81–81
+ require_repo import import require_repo L89–89
+ short_id import import short_id L85–85
+ sys import import sys L82–82
+ time import import time L83–83
+ logger variable variable logger L91–91
+ register function function register L165–201
+ run function function run L209–303
~ muse/cli/commands/verify_object.py .py 25 symbols added
+ _CHUNK variable variable _CHUNK L87–87
+ _FORMAT_CHOICES variable variable _FORMAT_CHOICES L86–86
+ _ObjectResult class class _ObjectResult L90–94
+ _iter_all_object_ids function function _iter_all_object_ids L97–107
+ _verify_one function function _verify_one L110–164
+ ExitCode import import ExitCode L79–79
+ TypedDict import import TypedDict L76–76
+ annotations import import annotations L68–68
+ argparse import import argparse L70–70
+ hashlib import import hashlib L71–71
+ iter_stored_objects import import iter_stored_objects L80–80
+ json import import json L72–72
+ logging import import logging L73–73
+ long_id import import long_id L78–78
+ object_path import import object_path L80–80
+ objects_dir import import objects_dir L80–80
+ pathlib import import pathlib L74–74
+ require_repo import import require_repo L81–81
+ sanitize_display import import sanitize_display L82–82
+ short_id import import short_id L78–78
+ sys import import sys L75–75
+ validate_object_id import import validate_object_id L82–82
+ logger variable variable logger L84–84
+ register function function register L167–219
+ run function function run L222–312
~ muse/core/gc.py .py 29 symbols added
+ GcResult class class GcResult L87–112
+ _DEFAULT_GRACE_PERIOD_SECONDS variable variable _DEFAULT_GRACE_PERIOD_SECONDS L78–78
+ _HEX_CHARS variable variable _HEX_CHARS L357–357
+ _MAX_SHELF_BYTES variable variable _MAX_SHELF_BYTES L74–74
+ _collect_reachable_commits function function _collect_reachable_commits L222–285
+ _collect_reachable_objects function function _collect_reachable_objects L120–170
+ _collect_reachable_snapshots function function _collect_reachable_snapshots L288–349
+ _collect_shelf_objects function function _collect_shelf_objects L173–202
+ _is_hex function function _is_hex L360–362
+ _list_stored_msgpack function function _list_stored_msgpack L413–438
+ _list_stored_objects function function _list_stored_objects L365–410
+ _run_gc_inner function function _run_gc_inner L504–618
+ _strip_prefix function function _strip_prefix L210–219
+ annotations import import annotations L55–55
+ dataclass class import dataclass L62–62
+ deque import import deque L61–61
+ field import import field L62–62
+ get_all_commits import import get_all_commits L68–68
+ iter_stored_objects import import iter_stored_objects L67–67
+ json import import json L57–57
+ load_json_file import import load_json_file L66–66
+ logging import import logging L58–58
+ msgpack import import msgpack L64–64
+ pathlib import import pathlib L59–59
+ read_snapshot import import read_snapshot L68–68
+ short_id import import short_id L66–66
+ time import import time L60–60
+ logger variable variable logger L70–70
+ run_gc function function run_gc L446–501
~ muse/core/object_store.py .py 45 symbols added
+ _CLEANUP_MIN_AGE_SECS variable variable _CLEANUP_MIN_AGE_SECS L88–88
+ _DEFAULT_ALGO variable variable _DEFAULT_ALGO L74–74
+ _DEFAULT_SHARD_PREFIX_LEN variable variable _DEFAULT_SHARD_PREFIX_LEN L75–75
+ _HEX_CHARS variable variable _HEX_CHARS L81–81
+ _OBJECTS_DIR variable variable _OBJECTS_DIR L71–71
+ _OBJECT_MODE variable variable _OBJECT_MODE L79–79
+ _VALID_SHARD_PREFIX_LENS variable variable _VALID_SHARD_PREFIX_LENS L76–76
+ _cached_shard_prefix_len function function _cached_shard_prefix_len L137–152
+ _created_object_shards variable variable _created_object_shards L189–189
+ _ensure_object_shard function function _ensure_object_shard L192–220
+ _fsync_fd function function _fsync_fd L91–120
+ _fsync_path function function _fsync_path L123–133
+ _object_path_with_fallback function function _object_path_with_fallback L343–378
+ _shard_prefix_len function function _shard_prefix_len L155–165
+ cleanup_stale_object_temps function function cleanup_stale_object_temps L223–286
+ has_object function function has_object L449–463
+ Iterator import import Iterator L58–58
+ MAX_FILE_BYTES import import MAX_FILE_BYTES L61–61
+ MAX_OBJECT_WRITE_BYTES import import MAX_OBJECT_WRITE_BYTES L61–61
+ annotations import import annotations L46–46
+ assert_not_symlink import import assert_not_symlink L61–61
+ assert_write_inside_repo import import assert_write_inside_repo L61–61
+ blob_id import import blob_id L60–60
+ fcntl import import fcntl L48–48
+ functools function import functools L49–49
+ hash_file import import hash_file L60–60
+ hashlib import import hashlib L50–50
+ logging import import logging L51–51
+ long_id import import long_id L60–60
+ os import import os L52–52
+ pathlib import import pathlib L53–53
+ short_id import import short_id L60–60
+ shutil import import shutil L54–54
+ sys import import sys L55–55
+ tempfile import import tempfile L56–56
+ time import import time L57–57
+ validate_object_id import import validate_object_id L61–61
+ iter_stored_objects function function iter_stored_objects L381–446
+ logger variable variable logger L69–69
+ object_path function function object_path L310–340
+ objects_dir function function objects_dir L289–307
+ read_object function function read_object L634–691
+ restore_object function function restore_object L694–817
+ write_object function function write_object L466–548
+ write_object_from_path function function write_object_from_path L551–631
~ tests/test_cli_new_commands.py .py 2 symbols added, 6 symbols modified
+ object_path import import object_path L14–14
+ write_object import import write_object L14–14
~ TestGcCli
~ test_gc_dry_run_does_not_delete
~ test_gc_preserves_reachable
~ test_gc_removes_orphan
~ test_gc_verbose_lists_objects
~ _make_repo
~ tests/test_core_blame.py .py 1 symbol modified
~ _write_object
~ tests/test_integrity_I8_object_store_scale.py .py 1 symbol added, 23 symbols modified
+ iter_stored_objects import import iter_stored_objects L36–36
~ TestLargeScaleMode
~ test_100k_objects_all_0o444
~ TestObjectMode
~ test_multiple_objects_all_0o444
~ TestShardScaleSmoke
~ test_256_two_char_shards_coexist
~ TestStaleTempCleanup
~ test_cleanup_is_idempotent
~ test_cleanup_multiple_shards
~ test_cleanup_removes_obj_tmp_files
~ test_cleanup_removes_restore_tmp_files
~ _DEFAULT_SHARD_PREFIX_LEN
~ _OBJECT_MODE
~ _VALID_SHARD_PREFIX_LENS
~ _object_path_with_fallback
~ cleanup_stale_object_temps
~ has_object
~ object_path
~ objects_dir
~ read_object
~ restore_object
~ write_object
~ write_object_from_path
~ tests/test_stress_object_store.py .py 2 symbols modified
~ TestSharding
~ test_256_shards_can_all_be_created
Older fix: replace _sha256 bare-hex helpers with blob_id in test files sha256:61bae996bc0a7c2313d680a6cd078ff5c84e8c5db19f6ca8d7a2312a6fc09b97 All commits
Newer → Latest on task/object-store-algo-dir

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:41df9d174a062c036619a5afbd7f4653788d8b2db013ce19213d2ece2c47b0f4 --body "your comment"