# Cache Reorganization Plan

Move all msgpack cache files from `.muse/*.msgpack` into `.muse/cache/` and
establish shared write utilities to eliminate duplicated idioms.

## Motivation

The `.muse/` root holds six cache files alongside authoritative VCS state
(HEAD, refs, commits, config). Caches are recomputable and should not live
next to irreplaceable data. Grouping them under `.muse/cache/` makes the
directory readable at a glance and enables future features (per-cache size
limits, eviction policies, GC targeting).

---

## Cache inventory

| New path | Module | Size (typical) | Write pattern |
|----------|--------|----------------|---------------|
| `cache/test_history.msgpack` | `muse/core/test_history.py` | 7 KB | mktemp + replace |
| `cache/stat.msgpack` | `muse/core/stat_cache.py` | 150 KB | mkstemp + fsync + replace |
| `cache/symbols.msgpack` | `muse/core/symbol_cache.py` | 35 MB | fixed .tmp + replace |
| `cache/callgraph.msgpack` | `muse/core/callgraph_cache.py` | 11 MB | fixed .tmp + replace |
| `cache/implicit_edges.msgpack` | `muse/core/implicit_edge_cache.py` | 154 KB | fixed .tmp + replace |
| `cache/invariants.msgpack` | `muse/plugins/code/_invariants.py` | 19 MB | fixed .tmp + replace |

All caches are **recomputable** — data loss on upgrade or corruption is safe.
Old files at the pre-migration locations become orphans on first use after
upgrade and can be removed by `muse gc` or manually.

---

## Write pattern taxonomy

**Pattern A — mkstemp + fsync** (`StatCache`)
Safe for concurrent writers. `mkstemp` gives each writer a unique temp path
so two parallel saves cannot collide. `fsync` ensures the bytes hit the
platters before `rename`. Used only by `StatCache`.

**Pattern B — fixed `.tmp` suffix** (`SymbolCache`, `CallGraphCache`,
`ImplicitEdgeCache`, `_InvariantFileCache`)
Two writers produce the same `.tmp` name; one silently overwrites the other.
Atomic for a single writer, but a race condition under parallel saves.
Target: consolidated to Pattern A in Phase 5.

**Pattern C — mktemp + replace** (`TestHistory`)
Unique temp path (no collision), no `fsync`. Acceptable for append-only
history where a torn write is detected on next load.

---

## Seven testing tiers

Every cache module must have tests in all seven tiers. The tier definitions
below apply uniformly across Phases 1–6.

### Tier 1 — Unit
Isolated, no filesystem I/O. Mocked or in-memory only.

Covers:
- `get()` miss → `None`
- `put()` → dirty flag set
- `get()` hit after `put()`
- `prune()` removes stale keys, sets dirty; no-op when all live
- `size` property
- `empty()` → `_cache_dir is None`, no entries
- `save()` on `empty()` is a no-op (no file created anywhere)

### Tier 2 — Integration
Real filesystem via `tmp_path`. No subprocess, no CLI.

Covers:
- `load()` on a missing file → empty cache
- `save()` creates the file at the correct path under `.muse/cache/`
- `save()` + `load()` round-trip: data identical after reload
- `save()` is a no-op when not dirty; file mtime unchanged on second save
- `_dirty = False` after a successful save
- Fixture helper creates `.muse/cache/` subdir before each test

### Tier 3 — End-to-end
Full CLI invocation against a real repository built in `tmp_path`.

Covers:
- Cold run populates the cache file on disk
- Warm run reads from cache (observable via I/O patching or timing)
- CLI output is identical between cold and warm runs
- Cache file survives a process restart (saved and reloaded correctly)
- `muse gc` removes orphaned old-location files (`.muse/*.msgpack`)

### Tier 4 — Stress
Large inputs, high iteration counts. Not a performance gate — correctness
under load.

Covers:
- 500+ distinct entries written and read back without corruption
- `prune()` on a 500-entry cache with 250 live keys: exact survivors
- 1 000 `put()` calls with repeated keys: `size` == unique key count
- Sequential save → load → save → load cycle: entries stable across 10 rounds

### Tier 5 — Data integrity
Adversarial on-disk state: corruption, version mismatch, partial writes,
cross-version upgrades.

Covers:
- Corrupt msgpack bytes → `load()` returns empty, never raises
- Wrong `version` field → `load()` returns empty
- Missing `entries` key → `load()` returns empty
- Invalid individual entry skipped; valid entries in same file survive
- Non-string key in `entries` dict → entry skipped, rest survive
- File truncated mid-write (simulated) → `load()` returns empty
- Atomic write: no `.tmp` file left behind after a successful `save()`
- Old-location orphan (`cache_file.msgpack` in `.muse/`) is ignored, not loaded

### Tier 6 — Performance
Timing benchmarks. Assert speedup ratios, not absolute wall times, so CI
hardware variance doesn't cause flakes.

Covers:
- Warm cache ≥ 5× faster than cold for 30-file repo (same assertion style
  as existing callgraph and implicit-edge tests)
- Second `save()` call (not dirty) completes in < 1 ms (no I/O)
- `load()` of a 500-entry cache completes in < 200 ms

### Tier 7 — Security
Untrusted cache content and path-traversal attempts.

Covers:
- Relative `..` path in a content-hash key → entry ignored on load
- Symlink at the cache file path → `save()` does not follow it (writes to
  temp then replaces, so symlink is overwritten not followed)
- Cache file owned by another user (mode 000) → `load()` returns empty,
  never raises; `save()` fails gracefully with a log warning, not a crash
- Msgpack payload with deeply nested dicts (bomb) → `load()` returns empty
  within a bounded time (no unbounded recursion)
- Field value of wrong type (e.g. `int` where `str` expected) → entry
  rejected, does not propagate into cache

---

## Docstring standard

Every cache class and every public/private method that touches persistence
must carry a docstring. Minimum structure:

```python
class FooCache:
    """One-line summary.

    Longer description covering the cache key, value type, storage location,
    write pattern (Pattern A/B/C), and lifecycle.

    Attributes
    ----------
    _cache_dir : pathlib.Path | None
        Absolute path to ``.muse/cache/``.  ``None`` for in-memory-only
        instances (``empty()``).  ``save()`` is a no-op when ``None``.
    _dirty : bool
        Set to ``True`` by ``put()`` and ``prune()`` when entries change.
        Reset to ``False`` by a successful ``save()``.
    """

    @classmethod
    def load(cls, muse_dir: pathlib.Path) -> "FooCache":
        """Load from ``muse_dir/cache/<filename>``.

        Returns an empty cache on any failure — missing file, corrupt bytes,
        version mismatch — and never raises.  Invalid individual entries are
        skipped so a partially corrupt file does not poison the whole cache.

        Parameters
        ----------
        muse_dir:
            Path to the ``.muse/`` directory of the repository root.
        """

    def save(self) -> None:
        """Atomically persist the cache to disk if it has changed.

        Uses a temp-file-then-rename pattern so a crash mid-write never
        leaves a corrupt cache file.  Silently skips when ``_dirty`` is
        ``False`` or ``_cache_dir`` is ``None``.
        """
```

---

## Phases

### Phase 1 — Infrastructure + test_history ✅
- Added `cache_dir(root)` helper to `paths.py` → `.muse/cache/`
- Updated `test_history_path()` → `cache_dir(root) / "test_history.msgpack"`
- Added `"cache"` to `_INIT_SUBDIRS` (init.py) and `_MUSE_SWEEP_DIRS` (repo.py)

**Test gaps to backfill (tiers 3–7):**
- Tier 3: `muse gc` removes old `.muse/test_history_cache.msgpack` orphan
- Tier 5: corrupt `test_history.msgpack` → graceful empty load
- Tier 7: path-traversal key in history file → entry rejected

---

### Phase 2 — stat_cache ✅
- `stat_cache_path(root)` → `cache_dir(root) / "stat.msgpack"`
- `StatCache` stores `_cache_dir`; `load()` and `save()` use it
- Temp files at `.muse/cache/.stat_cache_*.tmp` (Pattern A, already correct)
- All tiers covered by `test_core_stat_cache.py` and `test_query_stat_cache.py`

**Test gaps to backfill:**
- Tier 3 E2E: `muse code symbols` cold → warm CLI timing gap
- Tier 7: symlink at `cache/stat.msgpack` → `save()` overwrites symlink, not target

---

### Phase 3 — symbol_cache, callgraph_cache, implicit_edge_cache ✅
- `symbols.msgpack`, `callgraph.msgpack`, `implicit_edges.msgpack` in `cache/`
- All three cache classes store `_cache_dir`
- Fixture helpers create `.muse/cache/` subdir; 163 tests pass

**Test gaps to backfill (all three modules):**
- Tier 4 stress: 500-entry put/prune correctness
- Tier 7 security: `..` in content-hash key rejected on load; mode-000 file
  → graceful load failure

---

### Phase 4 — invariants cache
**TDD order: write all seven test tiers first, then make them pass.**

File: `tests/test_invariant_file_cache.py`

Production changes:
- Add `invariants_cache_path(root)` to `paths.py` → `cache_dir(root) / "invariants.msgpack"`
- Rename `_FILE_CACHE_NAME = "invariants.msgpack"`
- `_InvariantFileCache.__init__`: rename `muse_dir` → `cache_dir`; store `self._cache_dir`
- `load(repo_root)`: derive `cache_dir = muse_dir / "cache"`; return `cls(cache_dir, ...)`
- `save()`: use `self._cache_dir`; call `self._cache_dir.mkdir(parents=True, exist_ok=True)`
- Add full docstring to `_InvariantFileCache`, `load()`, `save()`, `get()`, `put()`, `prune()`

Test coverage required (all seven tiers):

**Tier 1 — Unit**
- `get()` miss → `None`
- `put()` → dirty
- `get()` hit
- `prune()` removes stale, sets dirty; no-op when all live
- `size` property
- `empty()` → `_cache_dir is None`
- `save()` on `empty()` → no file created

**Tier 2 — Integration**
- `load()` missing file → empty
- `save()` creates `cache/invariants.msgpack`
- Round-trip: `_FileData` fields survive msgpack serialization intact
- `save()` no-op when not dirty
- `_dirty = False` after save
- Fixture helper creates `.muse/cache/` before each test

**Tier 3 — End-to-end**
- Cold `muse code invariants` run creates `cache/invariants.msgpack`
- Warm run: `ast.parse` not called (patch and assert call count == 0)
- CLI output identical cold vs warm

**Tier 4 — Stress**
- 500 distinct content hashes put and read back correctly
- `prune()` on 500-entry cache with 250 live keys: exact survivors
- 1 000 repeated `put()` calls with same key: `size == 1`

**Tier 5 — Data integrity**
- Corrupt bytes → empty, no raise
- Wrong version → empty
- Missing `entries` → empty
- Invalid entry skipped; valid entry in same file survives
- Non-string content hash → entry skipped
- No `.tmp` leftover after successful save
- Old-location `code_invariants_cache.msgpack` in `.muse/` is ignored

**Tier 6 — Performance**
- Warm `muse code invariants` ≥ 5× faster than cold for 30-file repo
- Not-dirty `save()` < 1 ms
- `load()` of 500-entry cache < 200 ms

**Tier 7 — Security**
- `..` in content-hash key → entry rejected on load
- Mode-000 cache file → `load()` returns empty, never raises
- Deeply nested msgpack payload → `load()` returns empty within bounded time
- Wrong-type field value → entry rejected

---

### Phase 5 — Fix concurrent write race (Pattern B → A)
Replace fixed `.tmp` suffix writes with `mkstemp` in:
- `symbol_cache.py`, `callgraph_cache.py`, `implicit_edge_cache.py`, `_invariants.py`

Add `"cache"` to `_MUSE_SWEEP_DIRS` prefix list so orphaned `.tmp` files
under `.muse/cache/` are swept on startup.

**Test additions (Tier 7 security, Tier 5 integrity):**
- Concurrent save simulation: two writers, assert final file is valid msgpack
  (not a half-written interleaving)
- Orphaned `.tmp` in `.muse/cache/` is cleaned by `muse gc`

---

### Phase 6 — Shared MsgpackCache base class
Extract common `load()` / `_dirty` / `save()` pattern into
`muse/core/cache_base.py`.

`MsgpackCache` ABC:
- `_cache_path(root) -> Path` — abstract
- `_serialize(entries) -> bytes` — abstract
- `_deserialize(doc) -> entries` — abstract
- `load(root) -> Self` — classmethod, shared impl using Pattern A (mkstemp)
- `save()` — shared impl

All Pattern-B caches inherit from it. `StatCache` keeps its custom `fsync`
and dimension-hash logic.

**Test additions:**
- All seven tiers applied to `MsgpackCache` itself via a minimal concrete
  subclass (`_TestCache`) defined inside the test file
- Existing per-cache tier tests remain and continue to pass (no removal)

---

## Migration for existing repos

Old cache files at `.muse/*.msgpack` become orphans on first run after upgrade.
They are silently ignored — caches rebuild at the new location on next use.
`muse gc` will sweep the old locations as a housekeeping step (non-blocking,
added in Phase 4).