gabriel / muse public
commit-id-v2-rewrite.md markdown
272 lines 8.7 KB
Raw
sha256:b636f72dcba9e190afb980bece906fa5b717fbde014b76ef023df8cb96e01eb9 docs: expand cache plan with all seven testing tiers and do… Sonnet 4.6 131 days ago

Commit ID v2 — History Rewrite Spec

Problems Fixed

1 — Key-swap attack surface

compute_commit_id hashes only parent_ids + snapshot_id + message + committed_at. signer_public_key, author, and repo_id are not included. An attacker with write access to .muse/commits/ can replace the signing identity without changing the commit ID. run_verify reports it clean.

2 — author not bound to commit ID

Two commits with identical code, message, and timestamp but different authors get the same commit ID. For unsigned commits there is no signature to catch this.

3 — repo_id not bound to commit ID

A commit from one repo could be replayed into another with the same ID.

4 — Bare base64 signature prefix

161 historical commits store Ed25519 signatures as bare base64url (AAAA…) instead of the canonical ed25519:AAAA… prefix. All verify cryptographically. run_verify currently reports all 161 as signature failures.

5 — format_version schema versioning field

format_version exists as evolutionary scaffolding. After this rewrite there is one schema, forever. The field is removed entirely — including the format_version >= 7 guard in run_verify.

6 — branch field semantics are misleading

CommitRecord.branch stores the branch at time of creation, but a commit is reachable from multiple branches after a merge. Renamed to created_on_branch.

7 — Legacy object store paths (no algo directory)

Early Muse repos stored blobs at .muse/objects/<prefix>/<rest> with no algorithm subdirectory. The canonical layout is .muse/objects/sha256/<prefix>/<rest>. After migration, zero files should exist at any legacy path — every blob must be at the algo-prefixed path.


Solution — muse code migrate

A proper CLI subcommand, not a one-time script. Ships in the Muse binary so any user — not just gabriel — can run it when ready. Dry-run by default.

muse code migrate --dry-run   # inspect, make zero writes (default safe mode)
muse code migrate             # execute the full migration

The command does two things in one atomic sweep:

Part A — Commit DAG replay

  1. Recompute every commit ID with the v2 formula.
  2. Rename branchcreated_on_branch in every commit record.
  3. Drop format_version from every commit record.
  4. Normalise bare base64 signatures to ed25519:….
  5. Cascade new parent IDs forward so the chain is internally consistent.
  6. Update all branch heads to the new tip IDs.
  7. Delete the old commit files.

Part B — Object path migration

Blob content does not change — blob IDs are sha256:<hash-of-file-bytes> and file bytes are untouched. But any blob stored at a legacy path (no sha256/ algo directory) must be moved to the canonical path:

OLD: .muse/objects/<prefix>/<rest>
NEW: .muse/objects/sha256/<prefix>/<rest>

Steps:

  1. Walk .muse/objects/ looking for blobs not under an sha256/ subdirectory.
  2. For each: ensure the canonical path exists (create shard dir, move file).
  3. Delete the original at the legacy path.
  4. After all blobs are migrated, delete any now-empty legacy shard directories.

After migration: zero files exist at any legacy object path.


New compute_commit_id Formula

v1 (current):

SHA-256(parents \x00 snapshot_id \x00 message \x00 committed_at)

v2 (after rewrite):

SHA-256(repo_id \x00 parents \x00 snapshot_id \x00 message \x00 committed_at \x00 author \x00 signer_public_key)

Rules:

  • parents is the sorted, joined list of new parent commit IDs (after cascading).
  • author is "" for commits with no author.
  • signer_public_key is the canonical prefixed form (ed25519:<base64url>) or "".
  • repo_id is from repo.json.
  • All fields are null-byte separated. Field order is fixed.
  • No v1/v2 mode switch. After the rewrite, all commits use v2.

Schema Changes to CommitRecord

Field Change
format_version Removed
branch Renamed to created_on_branch

Remove all commit.format_version references from production code and tests. Remove the format_version >= 7 guard in run_verify — always verify signatures when commit.signature is non-empty.


Files to Touch

File Change
muse/core/snapshot.py Update compute_commit_id signature and body
muse/core/store.py Remove format_version; rename branchcreated_on_branch
muse/core/verify.py Remove format_version >= 7 guard
muse/cli/commands/commit.py Pass repo_id, author, signer_public_key to compute_commit_id
muse/cli/commands/code_migrate.py Newmuse code migrate command
muse/core/migrate.py New — DAG replay + object path migration logic
All callers of compute_commit_id Update call sites
All .branch on CommitRecord Update to .created_on_branch
All .format_version references Delete
Test files Update fixtures, assertions, field names

Find all callers before starting:

muse code grep "compute_commit_id" --json
muse code grep "format_version" --json
muse content-grep "\.branch\b" --json

muse code migrate Algorithm

Step 0 — Preflight

Abort if .muse/MERGE_STATE or .muse/rebase-merge/ exists.

Step 1 — Safety bundle

muse bundle create /tmp/<repo-name>-pre-v2.bundle --json

Step 2 — Load repo_id

repo_id = .muse/repo.json → "repo_id"

Step 3 — Collect all commits

BFS from all branch tips via parent_commit_id + parent2_commit_id.

Step 4 — Topological sort (Kahn's algorithm)

Parents before children. Seed with root commits (in-degree 0).

Step 5 — Rewrite each commit (Part A)

# a. Resolve new parent IDs via id_map
# b. Normalise bare base64 sig → "ed25519:…" (abort if crypto fails)
# c. Compute v2 commit ID
# d. Write new CommitRecord (no format_version; created_on_branch)
# e. id_map[old_id] = new_id

Step 6 — Update branch heads

Remap every ref file via id_map.

Step 7 — Delete old commit files

Unlink every commit_path(root, old_id) where old_id != new_id.

Step 8 — Migrate legacy object paths (Part B)

# a. Walk .muse/objects/ — find any blobs NOT under sha256/
# b. For each legacy blob:
#    - compute canonical dest = .muse/objects/sha256/<prefix>/<rest>
#    - skip if dest already exists (content-identical, already migrated)
#    - create shard dir if needed
#    - os.replace(legacy_path, canonical_path)
# c. Remove any now-empty legacy shard directories

Step 9 — Verify

muse verify --json  # expect all_ok: true, failures: []

Dry-Run Mode (--dry-run)

Prints:

  • old → new commit ID mapping
  • sig prefix fixes applied
  • new branch heads
  • legacy object paths that would be moved
  • summary counts: commits rewritten, blobs migrated, legacy dirs removed

Makes zero writes. Always run first.


Edge Cases

Case Handling
Root commit parent_ids=[]; all other fields still hashed
Merge commit Both parents resolved through id_map
Unsigned commit author="" and signer_public_key="" still included
Bare base64 sig fails crypto Abort immediately
Detached HEAD Remap .muse/HEAD if it contains a commit ID
Tags Nuke — drop any existing tag files
MERGE_STATE present Abort — finish merge first
Shelf entries Unaffected — no commit ID references
Blob already at canonical path Skip move (idempotent)
Legacy shard dir not empty after move Leave dir — only remove if empty

Execution Order (Workspace)

  1. muse-zsh (pilot — dry-run first, then apply on /tmp copy, verify, then in-place)
  2. muse
  3. musehub
  4. agentception
  5. maestro
  6. contracts
  7. Stori

Phases

# Work Status
1 TDD CommitRecord: remove format_version, rename branchcreated_on_branch ✅ Done
2 TDD compute_commit_id v2: add repo_id, author, signer_public_key ✅ Done
3 Update all callers; remove all format_version refs; .branch.created_on_branch; update docstrings ✅ Done
4 Update run_verify: remove format_version >= 7 guard ✅ Done
5 TDD muse code migrate: DAG replay + object path migration, dry-run and execute modes ✅ Done
6 Implement muse/core/migrate.py and muse/cli/commands/code_migrate.py ✅ Done
7 Dry-run on muse-zsh /tmp copy; inspect output ✅ Done
8 Apply to muse-zsh /tmp copy; run muse verify ✅ Done
9 Apply in-place to muse-zsh; push to hub ✅ Done
10 Apply to each remaining workspace repo in order; verify + push each ✅ Done
11 Delete this doc; commit ⬜ Next
File History 1 commit
sha256:b636f72dcba9e190afb980bece906fa5b717fbde014b76ef023df8cb96e01eb9 docs: expand cache plan with all seven testing tiers and do… Sonnet 4.6 131 days ago