Cache Reorganization Plan

Move all msgpack cache files from .muse/*.msgpack into .muse/cache/ and establish shared write utilities to eliminate duplicated idioms.

Motivation

The .muse/ root holds six cache files alongside authoritative VCS state (HEAD, refs, commits, config). Caches are recomputable and should not live next to irreplaceable data. Grouping them under .muse/cache/ makes the directory readable at a glance and enables future features (per-cache size limits, eviction policies, GC targeting).

---

Cache inventory

New path	Module	Size (typical)	Write pattern
cache/test_history.msgpack	muse/core/test_history.py	7 KB	mktemp + replace
cache/stat.msgpack	muse/core/stat_cache.py	150 KB	mkstemp + fsync + replace
cache/symbols.msgpack	muse/core/symbol_cache.py	35 MB	fixed .tmp + replace
cache/callgraph.msgpack	muse/core/callgraph_cache.py	11 MB	fixed .tmp + replace
cache/implicit_edges.msgpack	muse/core/implicit_edge_cache.py	154 KB	fixed .tmp + replace
cache/invariants.msgpack	muse/plugins/code/_invariants.py	19 MB	fixed .tmp + replace

All caches are recomputable — data loss on upgrade or corruption is safe. Old files at the pre-migration locations become orphans on first use after upgrade and can be removed by muse gc or manually.

---

Write pattern taxonomy

Pattern A — mkstemp + fsync (StatCache) Safe for concurrent writers. mkstemp gives each writer a unique temp path so two parallel saves cannot collide. fsync ensures the bytes hit the platters before rename. Used only by StatCache.

Pattern B — fixed .tmp suffix (SymbolCache, CallGraphCache, ImplicitEdgeCache, _InvariantFileCache) Two writers produce the same .tmp name; one silently overwrites the other. Atomic for a single writer, but a race condition under parallel saves. Target: consolidated to Pattern A in Phase 5.

Pattern C — mktemp + replace (TestHistory) Unique temp path (no collision), no fsync. Acceptable for append-only history where a torn write is detected on next load.

---

Seven testing tiers

Every cache module must have tests in all seven tiers. The tier definitions below apply uniformly across Phases 1–6.

Tier 1 — Unit

Isolated, no filesystem I/O. Mocked or in-memory only.

Covers:

• get() miss → None
• put() → dirty flag set
• get() hit after put()
• prune() removes stale keys, sets dirty; no-op when all live
• size property
• empty() → _cache_dir is None, no entries
• save() on empty() is a no-op (no file created anywhere)

Tier 2 — Integration

Real filesystem via tmp_path. No subprocess, no CLI.

Covers:

• load() on a missing file → empty cache
• save() creates the file at the correct path under .muse/cache/
• save() + load() round-trip: data identical after reload
• save() is a no-op when not dirty; file mtime unchanged on second save
• _dirty = False after a successful save
• Fixture helper creates .muse/cache/ subdir before each test

Tier 3 — End-to-end

Full CLI invocation against a real repository built in tmp_path.

Covers:

• Cold run populates the cache file on disk
• Warm run reads from cache (observable via I/O patching or timing)
• CLI output is identical between cold and warm runs
• Cache file survives a process restart (saved and reloaded correctly)
• muse gc removes orphaned old-location files (.muse/*.msgpack)

Tier 4 — Stress

Large inputs, high iteration counts. Not a performance gate — correctness under load.

Covers:

• 500+ distinct entries written and read back without corruption
• prune() on a 500-entry cache with 250 live keys: exact survivors
• 1 000 put() calls with repeated keys: size == unique key count
• Sequential save → load → save → load cycle: entries stable across 10 rounds

Tier 5 — Data integrity

Adversarial on-disk state: corruption, version mismatch, partial writes, cross-version upgrades.

Covers:

• Corrupt msgpack bytes → load() returns empty, never raises
• Wrong version field → load() returns empty
• Missing entries key → load() returns empty
• Invalid individual entry skipped; valid entries in same file survive
• Non-string key in entries dict → entry skipped, rest survive
• File truncated mid-write (simulated) → load() returns empty
• Atomic write: no .tmp file left behind after a successful save()
• Old-location orphan (cache_file.msgpack in .muse/) is ignored, not loaded

Tier 6 — Performance

Timing benchmarks. Assert speedup ratios, not absolute wall times, so CI hardware variance doesn't cause flakes.

Covers:

• Warm cache ≥ 5× faster than cold for 30-file repo (same assertion style as existing callgraph and implicit-edge tests)
• Second save() call (not dirty) completes in < 1 ms (no I/O)
• load() of a 500-entry cache completes in < 200 ms

Tier 7 — Security

Untrusted cache content and path-traversal attempts.

Covers:

• Relative .. path in a content-hash key → entry ignored on load
• Symlink at the cache file path → save() does not follow it (writes to temp then replaces, so symlink is overwritten not followed)
• Cache file owned by another user (mode 000) → load() returns empty, never raises; save() fails gracefully with a log warning, not a crash
• Msgpack payload with deeply nested dicts (bomb) → load() returns empty within a bounded time (no unbounded recursion)
• Field value of wrong type (e.g. int where str expected) → entry rejected, does not propagate into cache

---

Docstring standard

Every cache class and every public/private method that touches persistence must carry a docstring. Minimum structure:

class FooCache:
    """One-line summary.

    Longer description covering the cache key, value type, storage location,
    write pattern (Pattern A/B/C), and lifecycle.

    Attributes
    ----------
    _cache_dir : pathlib.Path | None
        Absolute path to ``.muse/cache/``.  ``None`` for in-memory-only
        instances (``empty()``).  ``save()`` is a no-op when ``None``.
    _dirty : bool
        Set to ``True`` by ``put()`` and ``prune()`` when entries change.
        Reset to ``False`` by a successful ``save()``.
    """

    @classmethod
    def load(cls, muse_dir: pathlib.Path) -> "FooCache":
        """Load from ``muse_dir/cache/<filename>``.

        Returns an empty cache on any failure — missing file, corrupt bytes,
        version mismatch — and never raises.  Invalid individual entries are
        skipped so a partially corrupt file does not poison the whole cache.

        Parameters
        ----------
        muse_dir:
            Path to the ``.muse/`` directory of the repository root.
        """

    def save(self) -> None:
        """Atomically persist the cache to disk if it has changed.

        Uses a temp-file-then-rename pattern so a crash mid-write never
        leaves a corrupt cache file.  Silently skips when ``_dirty`` is
        ``False`` or ``_cache_dir`` is ``None``.
        """

---

Phases

Phase 1 — Infrastructure + test_history ✅

• Added cache_dir(root) helper to paths.py → .muse/cache/
• Updated test_history_path() → cache_dir(root) / "test_history.msgpack"
• Added "cache" to _INIT_SUBDIRS (init.py) and _MUSE_SWEEP_DIRS (repo.py)

Test gaps to backfill (tiers 3–7):

• Tier 3: muse gc removes old .muse/test_history_cache.msgpack orphan
• Tier 5: corrupt test_history.msgpack → graceful empty load
• Tier 7: path-traversal key in history file → entry rejected

---

Phase 2 — stat_cache ✅

• stat_cache_path(root) → cache_dir(root) / "stat.msgpack"
• StatCache stores _cache_dir; load() and save() use it
• Temp files at .muse/cache/.stat_cache_*.tmp (Pattern A, already correct)
• All tiers covered by test_core_stat_cache.py and test_query_stat_cache.py

Test gaps to backfill:

• Tier 3 E2E: muse code symbols cold → warm CLI timing gap
• Tier 7: symlink at cache/stat.msgpack → save() overwrites symlink, not target

---

Phase 3 — symbol_cache, callgraph_cache, implicit_edge_cache ✅

• symbols.msgpack, callgraph.msgpack, implicit_edges.msgpack in cache/
• All three cache classes store _cache_dir
• Fixture helpers create .muse/cache/ subdir; 163 tests pass

Test gaps to backfill (all three modules):

• Tier 4 stress: 500-entry put/prune correctness
• Tier 7 security: .. in content-hash key rejected on load; mode-000 file → graceful load failure

---

Phase 4 — invariants cache

TDD order: write all seven test tiers first, then make them pass.

File: tests/test_invariant_file_cache.py

Production changes:

• Add invariants_cache_path(root) to paths.py → cache_dir(root) / "invariants.msgpack"
• Rename _FILE_CACHE_NAME = "invariants.msgpack"
• _InvariantFileCache.__init__: rename muse_dir → cache_dir; store self._cache_dir
• load(repo_root): derive cache_dir = muse_dir / "cache"; return cls(cache_dir, ...)
• save(): use self._cache_dir; call self._cache_dir.mkdir(parents=True, exist_ok=True)
• Add full docstring to _InvariantFileCache, load(), save(), get(), put(), prune()

Test coverage required (all seven tiers):

Tier 1 — Unit

• get() miss → None
• put() → dirty
• get() hit
• prune() removes stale, sets dirty; no-op when all live
• size property
• empty() → _cache_dir is None
• save() on empty() → no file created

Tier 2 — Integration

• load() missing file → empty
• save() creates cache/invariants.msgpack
• Round-trip: _FileData fields survive msgpack serialization intact
• save() no-op when not dirty
• _dirty = False after save
• Fixture helper creates .muse/cache/ before each test

Tier 3 — End-to-end

• Cold muse code invariants run creates cache/invariants.msgpack
• Warm run: ast.parse not called (patch and assert call count == 0)
• CLI output identical cold vs warm

Tier 4 — Stress

• 500 distinct content hashes put and read back correctly
• prune() on 500-entry cache with 250 live keys: exact survivors
• 1 000 repeated put() calls with same key: size == 1

Tier 5 — Data integrity

• Corrupt bytes → empty, no raise
• Wrong version → empty
• Missing entries → empty
• Invalid entry skipped; valid entry in same file survives
• Non-string content hash → entry skipped
• No .tmp leftover after successful save
• Old-location code_invariants_cache.msgpack in .muse/ is ignored

Tier 6 — Performance

• Warm muse code invariants ≥ 5× faster than cold for 30-file repo
• Not-dirty save() < 1 ms
• load() of 500-entry cache < 200 ms

Tier 7 — Security

• .. in content-hash key → entry rejected on load
• Mode-000 cache file → load() returns empty, never raises
• Deeply nested msgpack payload → load() returns empty within bounded time
• Wrong-type field value → entry rejected

---

Phase 5 — Fix concurrent write race (Pattern B → A)

Replace fixed .tmp suffix writes with mkstemp in:

• symbol_cache.py, callgraph_cache.py, implicit_edge_cache.py, _invariants.py

Add "cache" to _MUSE_SWEEP_DIRS prefix list so orphaned .tmp files under .muse/cache/ are swept on startup.

Test additions (Tier 7 security, Tier 5 integrity):

• Concurrent save simulation: two writers, assert final file is valid msgpack (not a half-written interleaving)
• Orphaned .tmp in .muse/cache/ is cleaned by muse gc

---

Phase 6 — Shared MsgpackCache base class

Extract common load() / _dirty / save() pattern into muse/core/cache_base.py.

MsgpackCache ABC:

• _cache_path(root) -> Path — abstract
• _serialize(entries) -> bytes — abstract
• _deserialize(doc) -> entries — abstract
• load(root) -> Self — classmethod, shared impl using Pattern A (mkstemp)
• save() — shared impl

All Pattern-B caches inherit from it. StatCache keeps its custom fsync and dimension-hash logic.

Test additions:

• All seven tiers applied to MsgpackCache itself via a minimal concrete subclass (_TestCache) defined inside the test file
• Existing per-cache tier tests remain and continue to pass (no removal)

---

Migration for existing repos

Old cache files at .muse/*.msgpack become orphans on first run after upgrade. They are silently ignored — caches rebuild at the new location on next use. muse gc will sweep the old locations as a housekeeping step (non-blocking, added in Phase 4).