gabriel / muse public
feat minor migrate feat/migrate-hardening #1 / 1
gabriel · 153 days ago · Apr 9, 2026 · Diff

feat(migrate): stream large repos, fix asserts, add NDJSON progress, full test coverage

Streaming: - _batch_commit_log: replace subprocess.run+capture_output with Popen+chunked streaming — raw git log output never fully buffered (critical for 1M+ commit repos) - _batch_diff_tree: replace communicate() with line-by-line stdout streaming — eliminates ~2GB buffer for Linux kernel-scale histories

Safety: - _CatFile: replace bare assert for stdin/stdout None checks with RuntimeError so failures surface cleanly instead of crashing silently under python -O

Agent UX: - --json now emits NDJSON progress events (branch_start, progress every 100 commits, branch_done, done) instead of a single summary at the end

Tests (38 new, zero before): - Unit: _should_exclude (12 cases), _manifest_directories (6 cases) - Integration: single-branch multi-commit, rename, delete, exclusion, dry-run, --all branches, --branch selection, error paths, --json NDJSON shape - Regression: _CatFile raises RuntimeError not AssertionError on pipe failure

snapshot-diff: - 5 new edge-case tests: bad --format, missing ref_b, --raw+zero changes, --json shorthand, no-args error

sha256:f7cc60c52e8b31d7e52f08f20d7b566e2e6e9d0dc16b9efa5a6bf147fea10276 sha
sha256:e5c2aac797c69e19a958bc54c5ed60c2fa29e2ec6b2bc81000b3d0023c0de051 parent
+68 ~9 symbols
sha256:dcf80ad7cc11f8144f4d31a6eb747dc00bf32c5ccf76ff52278d80f25e5c2a71 snapshot
+68
symbols added
~9
symbols modified
0
dead code introduced
Semantic Changes 77 symbols
~ tests/test_migrate.py .py 62 symbols added
+ TestCatFileRegression class class TestCatFileRegression L456–473
+ test_catfile_raises_on_pipe_failure method method test_catfile_raises_on_pipe_failure L457–473
+ TestManifestDirectories class class TestManifestDirectories L155–191
+ test_deep_nesting method method test_deep_nesting L188–191
+ test_dirs_are_sorted method method test_dirs_are_sorted L171–178
+ test_empty_manifest_gives_empty_list method method test_empty_manifest_gives_empty_list L156–158
+ test_nested_file_gives_parent_dirs method method test_nested_file_gives_parent_dirs L165–169
+ test_root_level_files_give_no_dirs method method test_root_level_files_give_no_dirs L160–163
+ test_shared_parents_deduplicated method method test_shared_parents_deduplicated L180–186
+ TestMigrateDryRun class class TestMigrateDryRun L309–331
+ test_dry_run_json_marks_dry_run_true method method test_dry_run_json_marks_dry_run_true L321–331
+ test_dry_run_writes_nothing method method test_dry_run_writes_nothing L310–319
+ TestMigrateErrors class class TestMigrateErrors L431–448
+ test_no_init_flag_skips_muse_init method method test_no_init_flag_skips_muse_init L440–448
+ test_not_a_git_repo_exits_1 method method test_not_a_git_repo_exits_1 L432–438
+ TestMigrateJson class class TestMigrateJson L334–400
+ test_json_all_lines_are_valid_json method method test_json_all_lines_are_valid_json L379–389
+ test_json_dry_run_done_event_has_dry_run_true method method test_json_dry_run_done_event_has_dry_run_true L391–400
+ test_json_emits_branch_start_and_done method method test_json_emits_branch_start_and_done L351–361
+ test_json_emits_done_event method method test_json_emits_done_event L335–349
+ test_json_emits_progress_events method method test_json_emits_progress_events L363–377
+ TestMigrateMultiBranch class class TestMigrateMultiBranch L403–428
+ test_all_flag_migrates_multiple_branches method method test_all_flag_migrates_multiple_branches L404–415
+ test_branch_flag_selects_specific_branch method method test_branch_flag_selects_specific_branch L417–428
+ TestMigrateSingleBranch class class TestMigrateSingleBranch L215–306
+ test_commit_count_matches method method test_commit_count_matches L226–235
+ test_deleted_file_not_in_final_snapshot method method test_deleted_file_not_in_final_snapshot L263–273
+ test_excluded_files_not_in_object_store method method test_excluded_files_not_in_object_store L287–296
+ test_extra_exclude_pattern method method test_extra_exclude_pattern L298–306
+ test_file_modification_tracked method method test_file_modification_tracked L250–261
+ test_object_content_round_trips method method test_object_content_round_trips L237–248
+ test_renamed_file_in_final_snapshot method method test_renamed_file_in_final_snapshot L275–285
+ test_single_commit_writes_muse_repo method method test_single_commit_writes_muse_repo L216–224
+ TestShouldExclude class class TestShouldExclude L97–147
+ test_default_ds_store_excluded method method test_default_ds_store_excluded L116–118
+ test_default_git_dir_excluded method method test_default_git_dir_excluded L100–102
+ test_default_muse_dir_excluded method method test_default_muse_dir_excluded L104–106
+ test_default_pyc_suffix_excluded method method test_default_pyc_suffix_excluded L112–114
+ test_default_pycache_excluded method method test_default_pycache_excluded L108–110
+ test_extra_prefix_excluded method method test_extra_prefix_excluded L124–126
+ test_extra_suffix_excluded method method test_extra_suffix_excluded L128–130
+ test_node_modules_excluded method method test_node_modules_excluded L132–134
+ test_normal_py_file_not_excluded method method test_normal_py_file_not_excluded L120–122
+ test_prefix_dir_itself_excluded method method test_prefix_dir_itself_excluded L144–147
+ test_root_level_normal_file_not_excluded method method test_root_level_normal_file_not_excluded L140–142
+ test_venv_excluded method method test_venv_excluded L136–138
+ _commit function function _commit L56–64
+ _git function function _git L36–44
+ _latest_manifest function function _latest_manifest L78–89
+ _make_git_repo function function _make_git_repo L47–53
+ _migrate function function _migrate L67–75
+ git_repo function function git_repo L200–204
+ CliRunner import import CliRunner L27–27
+ annotations import import annotations L17–17
+ hashlib import import hashlib L19–19
+ json import import json L20–20
+ pathlib import import pathlib L21–21
+ pytest import import pytest L24–24
+ read_snapshot import import read_snapshot L26–26
+ subprocess import import subprocess L22–22
+ muse_target function function muse_target L208–212
+ runner variable variable runner L29–29
~ muse/cli/commands/migrate.py .py 9 symbols modified
~ _CatFile
~ __init__
~ close
~ read
~ _batch_commit_log
~ _batch_diff_tree
~ _replay_branch
~ register
~ run
~ tests/test_plumbing_snapshot_diff.py .py 6 symbols added
+ TestSnapshotDiffEdgeCases class class TestSnapshotDiffEdgeCases L322–361
+ test_bad_format_value_exits_user_error method method test_bad_format_value_exits_user_error L325–331
+ test_json_shorthand_flag_accepted method method test_json_shorthand_flag_accepted L350–356
+ test_no_args_no_stdin_exits_user_error method method test_no_args_no_stdin_exits_user_error L358–361
+ test_raw_with_zero_changes_produces_no_diff_lines method method test_raw_with_zero_changes_produces_no_diff_lines L339–348
+ test_ref_a_provided_ref_b_missing_exits_user_error method method test_ref_a_provided_ref_b_missing_exits_user_error L333–337
← Older Oldest on feat/migrate-hardening
All commits
Newer → Latest on feat/migrate-hardening

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:f7cc60c52e8b31d7e52f08f20d7b566e2e6e9d0dc16b9efa5a6bf147fea10276 --body "your comment"