gabriel / muse public
sprint-algo-prefix.md markdown
249 lines 16.8 KB
Raw
sha256:b636f72dcba9e190afb980bece906fa5b717fbde014b76ef023df8cb96e01eb9 docs: expand cache plan with all seven testing tiers and do… Sonnet 4.6 134 days ago

Sprint: Algorithm-Prefixed Storage Paths

Every content-addressed ID in Muse carries its algorithm as a prefix in the value (sha256:<hex>). The same principle must hold in the filesystem path: the algorithm must appear as a path segment, never as a prefix inside a filename. This makes storage algorithm-agnostic, future-proof, and Windows-safe (colons are not valid in filenames on Windows).

The object store already does this correctly: .muse/objects/sha256/<2-char>/<62-char> — every other storage area must follow the same pattern.

For nested resources (tags, releases) the path carries the algorithm of each ID independently:

.muse/tags/<repo-algo>/<repo-id>/<tag-algo>/<tag-id>.msgpack

This handles heterogeneous hashing across time — a repo created with sha256 can have tags created with blake3, and the path reflects both without ambiguity.

There is no backwards compatibility layer. All paths move to the new convention in a single atomic refactor.


repo_id: UUID → Genesis Fingerprint

repo_id is currently stored as a UUID in .muse/repo.json. UUIDs must not be used for any system-level entity. Every system entity must be identified by a content fingerprint computed at genesis: a one-time hash of the initialisation inputs (timestamp + domain + path), never re-hashed when content changes.

The code at init.py:481 already calls content_hash({"created_at": ..., "domain": ..., "path": ...}) — the correct shape. The stored value must also carry the algorithm prefix (sha256:<hex>), and all path segments derived from it must use the bare hex under an algo directory.


Agent Quick-Reference

Each row below is one atomic unit of work. An agent should complete one row per session, run only the tests listed for that row, and commit on the task branch for that row.

# Area Source file(s) New path shape Utility method Tests to run
1 Commits store.py .muse/commits/sha256/<hex>.msgpack commit_path(root, commit_id) test_cmd_commit_*.py, test_store_commits.py
2 Snapshots store.py .muse/snapshots/sha256/<hex>.msgpack snapshot_path(root, snapshot_id) test_cmd_snapshot*.py, test_store_snapshots.py
3 Tags store.py .muse/tags/sha256/<repo-hex>/sha256/<tag-hex>.msgpack tag_path(root, repo_id, tag_id) test_cmd_tag*.py, test_store_tags.py
4 Releases store.py .muse/releases/sha256/<repo-hex>/sha256/<release-hex>.msgpack release_path(root, repo_id, release_id) test_release_supercharge.py, test_store_releases.py
5 Harmony patterns harmony.py .muse/harmony/patterns/sha256/<hex>/ pattern_dir(root, pattern_id) test_harmony_*.py
6 Harmony resolutions harmony.py .muse/harmony/patterns/sha256/<p-hex>/resolutions/sha256/<r-hex>.json resolution_path(root, pattern_id, resolution_id) test_harmony_*.py
7 repo_id genesis store.py, init.py, repo.json stored as sha256:<hex>; path segments use bare hex under sha256/ repo_genesis_id(inputs) test_cmd_init*.py, test_store_repo.py

Detailed Grid

1 — Commits

Field Value
Current path .muse/commits/<64-hex>.msgpack
Target path .muse/commits/sha256/<64-hex>.msgpack
Current construction _commits_dir(root) / f"{commit_id.removeprefix('sha256:')}.msgpack"
Target construction commit_path(root, commit_id)
Utility method commit_path(repo_root: Path, commit_id: str) -> Path
Utility location muse/core/store.py — replaces current commit_path() at line 1381
Utility contract Accepts sha256:<hex> or bare <hex>. Strips prefix, extracts algo, returns _commits_dir(root) / algo / f"{hex}.msgpack". Raises ValueError for malformed IDs.
Algo extraction algo, _, hex_id = commit_id.partition(':') if : in id, else algo = "sha256"
Directory helper _commits_dir(root) at line 1359 — no change needed
All callers write_commit, read_commit, commit_exists in store.py; any bundle/pack path referencing commits directly
Glob pattern update _commits_dir(root).glob("*.msgpack")_commits_dir(root).glob("*/*.msgpack")
Docstring requirement Module-level docstring in store.py layout section; function docstring on commit_path explaining algo-in-path convention and why no colons in filenames
New test file tests/test_store_commits.py
Test cases test_commit_path_sha256, test_commit_path_bare_hex, test_commit_path_rejects_invalid, test_commit_path_algo_in_dir, test_write_read_commit_roundtrip, test_commit_path_windows_safe (no colon in any segment)

2 — Snapshots

Field Value
Current path .muse/snapshots/<64-hex>.msgpack
Target path .muse/snapshots/sha256/<64-hex>.msgpack
Current construction _snapshots_dir(root) / f"{snapshot_id.removeprefix('sha256:')}.msgpack"
Target construction snapshot_path(root, snapshot_id)
Utility method snapshot_path(repo_root: Path, snapshot_id: str) -> Path
Utility location muse/core/store.py — replaces current snapshot_path() at line 1386
Utility contract Same contract as commit_path: accepts prefixed or bare, extracts algo, returns _snapshots_dir(root) / algo / f"{hex}.msgpack".
All callers write_snapshot, read_snapshot, snapshot_exists, apply_manifest in store.py
Glob pattern update _snapshots_dir(root).glob("*.msgpack")_snapshots_dir(root).glob("*/*.msgpack")
Docstring requirement Function docstring explaining algo-in-path; cross-reference to commit_path
New test file tests/test_store_snapshots.py
Test cases test_snapshot_path_sha256, test_snapshot_path_bare_hex, test_snapshot_path_rejects_invalid, test_snapshot_path_algo_in_dir, test_write_read_snapshot_roundtrip, test_snapshot_path_windows_safe

3 — Tags

Field Value
Current path .muse/tags/<repo-id>/<tag-id>.msgpack (repo-id is UUID; tag-id has sha256: prefix embedded in filename — invalid on Windows)
Target path .muse/tags/sha256/<repo-hex>/sha256/<tag-hex>.msgpack
Current construction _tags_dir(root, repo_id) / f"{tag.tag_id}.msgpack" (line 2156)
Target construction tag_path(root, repo_id, tag_id)
Utility method tag_path(repo_root: Path, repo_id: str, tag_id: str) -> Path
Utility location muse/core/store.py — replaces inline construction at lines 2152, 2179
Utility contract Both repo_id and tag_id accepted with or without sha256: prefix. Extracts algo from each independently. Returns _tags_dir(root) / repo_algo / repo_hex / tag_algo / f"{tag_hex}.msgpack". Each algo segment comes from its respective ID. Raises ValueError for malformed IDs.
Two-level algo repo_algo may differ from tag_algo — path encodes both. Example: tags/sha256/<repo-hex>/blake3/<tag-hex>.msgpack is valid.
All callers write_tag, delete_tag, get_all_tags, get_tag_by_name in store.py
Directory helper update _tags_dir(root, repo_id) becomes _tags_dir(root) (no repo_id arg); scoping moves into tag_path
Glob pattern update tags_dir.glob("*.msgpack")tags_dir.glob("*/*/*/*.msgpack") to match <repo-algo>/<repo-hex>/<tag-algo>/<tag-hex>.msgpack
Docstring requirement Explain two-level algo, Windows-safety, and why colons are forbidden in filesystem paths
repo_id dependency After sprint row 7, repo_id will be sha256:<hex> instead of UUID — tag_path must handle both during the row-7 transition, then only the new format
New test file tests/test_store_tags.py
Test cases test_tag_path_both_sha256, test_tag_path_mixed_algos, test_tag_path_rejects_invalid_repo_id, test_tag_path_rejects_invalid_tag_id, test_tag_path_no_colon_in_any_segment, test_write_read_tag_roundtrip, test_delete_tag, test_list_tags

4 — Releases

Field Value
Current path .muse/releases/<repo-id>/<release-hex>.msgpack (repo-id is UUID)
Target path .muse/releases/sha256/<repo-hex>/sha256/<release-hex>.msgpack
Current construction _releases_dir(root, repo_id) / f"{release.release_id}.msgpack" (line 2223)
Target construction release_path(root, repo_id, release_id)
Utility method release_path(repo_root: Path, repo_id: str, release_id: str) -> Path
Utility location muse/core/store.py — replaces inline construction at lines 2223, 2232, 2293
Utility contract Same two-level algo contract as tag_path. Each ID's algo is extracted independently.
All callers write_release, read_release, list_releases, delete_release in store.py
Directory helper update _releases_dir(root, repo_id)_releases_dir(root) — scoping moves into release_path
Glob pattern update releases_dir.glob("*.msgpack")releases_dir.glob("*/*/*/*.msgpack")
Docstring requirement Same as tags; cross-reference tag_path for the two-level algo pattern
New test file tests/test_store_releases.py
Test cases test_release_path_both_sha256, test_release_path_mixed_algos, test_release_path_no_colon_in_any_segment, test_write_read_release_roundtrip, test_delete_release, test_list_releases

5 — Harmony Patterns

Field Value
Current path .muse/harmony/patterns/<64-hex>/
Target path .muse/harmony/patterns/sha256/<64-hex>/
Current construction patterns_dir(root) / _id_hex(pattern_id) where _id_hex strips sha256: prefix (line 599)
Target construction pattern_dir(root, pattern_id)
Utility method pattern_dir(repo_root: Path, pattern_id: str) -> Path
Utility location muse/core/harmony.py — replaces _pattern_entry_dir at line 599
Utility contract Accepts sha256:<hex> or bare hex. Extracts algo, returns patterns_dir(root) / algo / hex_id. Creates parent dirs on write. Raises ValueError for unknown algo or malformed ID.
_id_hex helper Retire _id_hex(); replace all call sites with explicit split_id(id)(algo, hex)
All callers Every function in harmony.py that calls _pattern_entry_dir or constructs pattern paths
Glob pattern update patterns_dir(root).glob("*/")patterns_dir(root).glob("*/*/")
Docstring requirement Explain that the pattern directory is NOT content-addressed (directory name = pattern_id hex, not hash of directory contents); the algo prefix is for future hash agility only
New test file tests/test_store_harmony_paths.py
Test cases test_pattern_dir_sha256, test_pattern_dir_bare_hex, test_pattern_dir_algo_in_path, test_pattern_dir_no_colon

6 — Harmony Resolutions

Field Value
Current path .muse/harmony/patterns/<p-hex>/resolutions/<r-hex>.json
Target path .muse/harmony/patterns/sha256/<p-hex>/resolutions/sha256/<r-hex>.json
Current construction Derived from _pattern_entry_dir(root, pid) / "resolutions" / f"{_id_hex(rid)}.json"
Target construction resolution_path(root, pattern_id, resolution_id)
Utility method resolution_path(repo_root: Path, pattern_id: str, resolution_id: str) -> Path
Utility location muse/core/harmony.py — new function, replaces inline construction
Utility contract Builds on pattern_dir. Extracts resolution algo from resolution_id. Returns pattern_dir(root, pattern_id) / "resolutions" / r_algo / f"{r_hex}.json".
All callers All resolution read/write/list/delete functions in harmony.py
Glob pattern update resolutions_dir.glob("*.json")resolutions_dir.glob("*/*.json")
Docstring requirement Cross-reference pattern_dir; note that resolution IDs and pattern IDs may use different algorithms
New test file Extend tests/test_store_harmony_paths.py
Test cases test_resolution_path_sha256, test_resolution_path_mixed_pattern_resolution_algos, test_resolution_path_no_colon, test_write_read_resolution_roundtrip

7 — repo_id: UUID → Genesis Fingerprint

Field Value
Current value UUID string, e.g. "51db136e-ae54-42c2-91c2-230f8171a488" in .muse/repo.json
Target value "sha256:<64-hex>" stored in repo.json; bare <64-hex> used in filesystem paths under a sha256/ segment
Current creation init.py:481content_hash({"created_at": ..., "domain": ..., "path": ...}) — already correct shape, but the stored value on disk is a UUID for this repo (legacy initialization)
Target creation Same content_hash(...) call; store the full sha256:<hex> string in repo.json
Utility method repo_genesis_id(created_at: str, domain: str, path: str) -> str
Utility location muse/core/store.py — new function wrapping content_hash, documents the genesis contract
Utility contract Computes content_hash({"created_at": created_at, "domain": domain, "path": path}). Returns sha256:<hex>. This ID is immutable after genesis — it does not change when content changes.
Validation update validation.pyvalidate_repo_id must accept sha256:<hex> (prefixed) and reject bare UUIDs. Update the validator and all call sites.
Scope repo.json schema, init.py, store.py, validation.py, and all 376 files that reference repo_id — this is a large cross-cutting refactor. Treat as its own multi-session sprint.
Path impact Tags and releases scoped under repo_id change from tags/<uuid>/ to tags/sha256/<repo-hex>/ — rows 3 and 4 depend on this being settled first, but can use a split_id() helper that handles both formats during transition
No UUID fallback No legacy UUID support. The repo is initialized fresh with the new format. Existing repos must be re-initialized or their repo.json updated manually.
Docstring requirement New repo_genesis_id function docstring must explain: (1) genesis = one-time computation, (2) not re-hashed on content change, (3) content-indexed not content-addressed in the traditional sense, (4) algo in prefix enables future migration
New test file tests/test_store_repo.py
Test cases test_repo_genesis_id_format, test_repo_genesis_id_is_deterministic, test_repo_genesis_id_is_prefixed, test_repo_id_in_path_uses_bare_hex, test_repo_id_validation_rejects_uuid, test_repo_id_validation_accepts_sha256_prefixed

Shared Utility: split_id()

Every row above needs to extract (algo, hex) from a potentially-prefixed ID. This must live in one place.

Field Value
Function split_id(object_id: str) -> tuple[str, str]
Location muse/core/store.py (or a new muse/core/ids.py if the module gets crowded)
Contract If : present: algo, _, hex_id = object_id.partition(':'). If absent: algo = "sha256", hex_id = object_id. Validates hex_id is 64 lowercase hex chars. Raises ValueError otherwise.
Usage Called by commit_path, snapshot_path, tag_path, release_path, pattern_dir, resolution_path, repo_genesis_id
Docstring Must document: (1) why algo is extracted not assumed, (2) the bare-hex fallback defaults to sha256 for legacy callers only, (3) the function is the single source of truth for ID parsing
Test file tests/test_store_ids.py
Test cases test_split_id_prefixed, test_split_id_bare_hex_defaults_sha256, test_split_id_rejects_empty, test_split_id_rejects_non_hex, test_split_id_rejects_wrong_length, test_split_id_rejects_colon_in_wrong_position

Object Store (reference — already correct)

.muse/objects/sha256/<2-char>/<62-char> — no changes needed. object_store.py:object_path() is the canonical model every other utility method should follow. Read it before implementing any new path utility.


Execution Order

Rows must be executed in this order due to dependencies:

split_id()          ← prerequisite for everything
    ↓
commits (1)         ← no dependencies
snapshots (2)       ← no dependencies
harmony (5, 6)      ← no dependencies
    ↓
repo_id (7)         ← must land before tags/releases can finalize
    ↓
tags (3)            ← depends on repo_id format being settled
releases (4)        ← depends on repo_id format being settled

split_id and rows 1, 2, 5, 6 can be parallelized. Row 7 is the blocker for rows 3 and 4. Do not start rows 3 or 4 until row 7 is merged to dev.

File History 1 commit
sha256:b636f72dcba9e190afb980bece906fa5b717fbde014b76ef023df8cb96e01eb9 docs: expand cache plan with all seven testing tiers and do… Sonnet 4.6 134 days ago