gabriel / muse public
perf BREAKING migrate perf/migrate-batch #1 / 1
gabriel · 158 days ago · Apr 9, 2026 · Diff

perf(migrate): 14x speedup via batched git processes

Before: O(N) subprocess calls — one git diff-tree + one git show per commit. After: O(1) subprocess calls for the entire branch:

- _batch_commit_log: single git log call reads all SHA + metadata at once using ASCII record/field separators (\x1e/\x1f) for safe multi-line parsing. Merge detection uses the %P parent field — no extra subprocess needed.

- _batch_diff_tree: single git diff-tree --stdin call streams all per-file deltas for the entire branch through one long-running process.

- _CatFile: existing git cat-file --batch process unchanged.

Result on cgcardona/muse (267 commits, 264 merge commits skipped): 26.0s → 9.6s → 1.79s (14.5x overall vs original)

sha256:841573ca87d89c75dce940ebe48d776cd296612080535171dfc655a857ee5858 sha
sha256:29ea74e20d91874d790917a6c1b15082e392fc38e4568b294afab6dfb4f210cc parent
+15 ~11 −11 symbols
sha256:9819243465ecf912544c6d74018ca75a70f497e291243ada87826d98b2211c87 snapshot
+15
symbols added
~11
symbols modified
−11
symbols removed
0
dead code introduced
Semantic Changes 37 symbols
~ muse/cli/commands/migrate.py .py 15 symbols added, 11 symbols removed, 11 symbols modified
− _META_SEP variable variable _META_SEP L131–131
− _extract_tree_to function function _extract_tree_to L214–250
− _git_commit_meta function function _git_commit_meta L173–186
− _git_commits_oldest_first function function _git_commits_oldest_first L146–170
− _git_parent_shas function function _git_parent_shas L189–192
− _is_merge_commit function function _is_merge_commit L195–196
− require_repo import import require_repo L67–67
− shutil import import shutil L58–58
− tarfile import import tarfile L61–61
− tempfile import import tempfile L62–62
− walk_workdir_with_dirs import import walk_workdir_with_dirs L68–68
+ Manifest variable variable Manifest L81–81
+ _CatFile class class _CatFile L147–187
+ __enter__ method method __enter__ L183–184
+ __exit__ method method __exit__ L186–187
+ __init__ method method __init__ L155–161
+ close method method close L178–181
+ read method method read L163–176
+ _Delta class class _Delta L135–139
+ _FIELD_SEP variable variable _FIELD_SEP L219–219
+ _RECORD_SEP variable variable _RECORD_SEP L220–220
+ _batch_commit_log function function _batch_commit_log L223–264
+ _batch_diff_tree function function _batch_diff_tree L267–326
+ _manifest_directories function function _manifest_directories L334–349
+ dataclasses class import dataclasses L56–56
+ hashlib import import hashlib L58–58
~ _DEFAULT_EXCLUDE_PREFIXES
~ _git
~ _git_is_repo
~ _git_local_branches
~ _replay_branch
~ _replay_commit
~ _should_exclude
~ compute_commit_id
~ compute_snapshot_id
~ register
~ run
← Older Oldest on perf/migrate-batch
All commits
Newer → Latest on perf/migrate-batch

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:841573ca87d89c75dce940ebe48d776cd296612080535171dfc655a857ee5858 --body "your comment"