Merged

perf: download skill sources in parallel

Looks clear5 minors

  1. WhatWhat, minor: All clones stay on disk until the install loop reaches themadd.ts:1488–1490
  2. ReviewReview, minor: Plain-download fallback writes a well-known lock entry without sourceUrladd.ts:1403–1408
  3. ReviewReview, minor: Well-known lock entry is built in two placesadd.ts:1366–1379
antfubotfeat/parallel-source-downloads→main+482 −102 · 9 filesGitHub
What

What

What does this change do?

Fetch all sources, then install

install, experimental_sync and update now clone or download every skill source first, 4 at a time by default. Then they install or check each one in the old order. Well-known lock entries now take this path too.

virpo ran /show-me-pr@prehistoriqClaude Code, Opus 5.5after mergeoutside the team

Each command now runs in two phases. Network: fetchSources (or fetchGlobalSource / an up-front cloneRepo in update) fetches every source through mapConcurrent. The limit comes from SKILLS_CLONE_CONCURRENCY and defaults to 4. Results keep input order, and a failed source becomes an error instead of a throw. Local: installFromSource now takes the fetched result. It filters, installs, writes skills-lock.json and removes the temp dir.

Well-known sources restored from the lock no longer go through runAdd. They are wrapped as WellKnownSourceSkill, installed with installWellKnownSkillForAgent, and locked through wellKnownLockEntry, which writes the same fields as before.

skills-lock.json / skills field
fetchSourcesnew
mapConcurrent (limit 4)new
installFromSource (one at a time)changed
fetchSourcenew
installToTargets
resolveWellKnownSkillsnew
resolveSkills (blob / clone / download)
addSkillToLocalLock
skills-lock.json / skills field SourceRequest list fetchSources. fetchSources to mapConcurrent (limit 4). mapConcurrent (limit 4) per request fetchSource. fetchSource well-known URL resolveWellKnownSkills. fetchSource otherwise / fallback resolveSkills (blob / clone / download). fetchSources FetchedSource list, input order installFromSource (one at a time). installFromSource (one at a time) to installToTargets. installToTargets to addSkillToLocalLock
install and experimental_sync: the network phase (green) now runs before the install loop. installFromSource no longer fetches anything itself.
  • minorAll clones stay on disk until the install loop reaches themsrc/add.tsL1488–1490

    The temp dir is now created in fetchSource and removed in installFromSource's finally. The concurrency limit caps clones in flight, not clones on disk. In update, the deletion confirm prompt can run while later clones are still on disk. The CLI has no SIGINT handler, so Ctrl+C there leaves them in tmp.

  • nitWell-known lock restores no longer send telemetrysrc/install.tsL63–82

    The old runAdd path called track() for well-known installs. installFromSource has no track() call. Git sources already behaved this way.

Files changed
9
Lines
+482 / -102
Default concurrency
4
Review

Review

Is it built right?

No material issues; 4 minor/nit

The parallel download change has no blocker or major issue. Four optional suggestions cover a well-known lock-entry edge case, duplicated code, and test coverage.

virpo ran /review-anvil@mrshuClaude Code, Opus 5.5after mergeoutside the team

3 minor1 nit

  • minorPlain-download fallback writes a well-known lock entry without sourceUrlsrc/add.tsL1403–1408

    If the well-known index returns no skills, fetchSource falls back to a plain download. installFromSource then writes projectLockEntry, which has no sourceUrl or wellKnownDigest for a 'well-known' source. Later, skills update stops treating that skill as well-known (src/update.ts:804). runAdd skips the project lock for plain downloads (src/add.ts:2377).

    Fix

    Skip the project lock write when a well-known source was installed through the plain-download fallback, as runAdd does for directDownload.

  • minorWell-known lock entry is built in two placessrc/add.tsL1366–1379

    wellKnownLockEntry builds the same entry that handleWellKnownSkills still builds inline (src/add.ts:1035-1043). If one copy changes, add and install can write different entries.

    Fix

    Have handleWellKnownSkills call wellKnownLockEntry.

  • minorNo multi-source failure test for update and synctests/parallel-source-fetch.test.tsL47–158

    The new tests only cover runInstallFromLock. Nothing tests how updateGlobalSkills (src/update.ts:617-629) and runSync (src/sync.ts:925-928) match each result to its source by index when one fetch fails.

    Fix

    Add an updateGlobalSkills test with two source groups where the first fetch fails, and check that the second group is still checked against its own tree or clone.

  • nitClone-failure branch repeats existing failure handlingsrc/update.tsL924–928

    The new tempDir === null branch repeats the message and failCount update that the catch block and the resolvedPaths === null check already do.

    Fix

    Send a failed clone through the existing failure path.

Set aside (not confirmed): skills install now installs nothing until every source finishes downloading (src/install.ts:69). Well-known content requests have no deadline, so a response that never finishes also blocks sources listed before it. On the base commit those earlier sources installed before the hang. The hang itself already existed. A probe was written but not run, because no isolated proof runner was configured.

3 of 3 passed · 1 skipped
  • pnpm type-check
  • Related tests (concurrency, parallel-source-fetch, install)11 tests passed
  • Reviewers (2 Codex + 1 Claude)All 3 returned valid output
  • Adversarial second checkOff by default for read-only review
Show me

Show me

What does it look like?

One spinner, 6.8s → 2.9s

With a lock file listing 3 sources, `experimental_install` now downloads them all under one spinner, then installs. Same 6 skills, exit 0, 6.8s → 2.9s (local fixture with 2s simulated clone latency).

virpo ran /run@anthropicsClaude Code, Opus 5.5after mergeoutside the team
install wall time
2.9s-3.9s vs 6.8s
update -p wall time
2.4s-4.3s vs 6.7s
skills installed
6 / 6
concurrency=1
6.7sback to sequential
node <base>/src/cli.ts experimental_install
│
●  Restoring 6 skills from skills-lock.json into .agents/skills/
│
◇  Repository cloned
│
◇  Found 2 skills
│
◆  Restored alpha-one, alpha-two
│
◇  Repository cloned
│
◇  Found 2 skills
│
◆  Restored beta-one, beta-two
│
◇  Repository cloned
│
◇  Found 2 skills
│
◆  Restored gamma-one, gamma-two
---
exit=0  wall=6.8s
installed: alpha-one alpha-two beta-one beta-two gamma-one gamma-two
exit 0
node <head>/src/cli.ts experimental_install
│
●  Restoring 6 skills from skills-lock.json into .agents/skills/
│
◇  Downloaded 3 sources
│
◆  Restored alpha-one, alpha-two
│
◆  Restored beta-one, beta-two
│
◆  Restored gamma-one, gamma-two
---
exit=0  wall=2.9s
installed: alpha-one alpha-two beta-one beta-two gamma-one gamma-two
exit 0

Fixture: three local bare git repos (alpha, beta, gamma, 2 skills each). Git config rewrites https://skills.example.test/*.git to file://, so there is no network access. A git wrapper on PATH sleeps 2s per clone to simulate network latency, and both versions ran through it. The skills-lock.json came from real skills add runs. Each version ran from a fresh project dir via node src/cli.ts. Spinner output is shown as it looks on screen when finished.

With SKILLS_CLONE_CONCURRENCY=1, head took 6.7s and the spinner showed Downloaded 1 of 3 → 2 of 3 → Downloaded 3 sources.

3 of 3 passed · 2 skipped
  • experimental_install: same skills restored, exit 0
  • update -p -y check phase prints 'Fetching 3 sources…'
  • SKILLS_CLONE_CONCURRENCY=1 restores sequential behaviour
  • update with a pending reinstallthe reinstall runs bin/cli.mjs, which needs a dist/ build; the base worktree had none
  • experimental_sync remote sources / well-known URL sourcesnot exercised; sync uses the same fetchSources path
Proof

Proof

Does it actually work?

Parallel fetch works, 40% faster

Ran install, experimental_sync and update against slowed local git/HTTP fixtures: up to 4 sources overlap, the cap env var works, and failures stay isolated with no temp dirs left behind.

virpo ran /verify@anthropicsClaude Code, Opus 5.5after mergeoutside the team
install, 6 sources
6.8sbase 12.0s
update -g, 5 sources
7.4sbase 12.3s
Touched + adjacent tests
226/226
Full suite
982/9874 env-only, same on base
10 of 10 passed · 1 with a warning · 1 skipped
  • Full vitest suite4 failures from Claude Code env vars (detect-agent) and missing git-lfs; same on base
  • GitHub Trees API branch of update -gNeeds github.com; not exercised
9 more passed
  • experimental_install: 5 git + 1 well-known sourcePeak 4 clones in flight; 5th waited for a free slot; all 6 restored in lock order
  • Well-known lock entry same as basecomputedHash and wellKnownDigest byte-identical
  • SKILLS_CLONE_CONCURRENCY = 1/2/2.9/0/-3/abc/100Peaks 1/2/2/4/4/4/5; invalid values use default 4
  • Broken git + dead well-known source mixed inEach failed alone, exit 1, others restored. Base ended the run at the dead well-known source
  • experimental_sync with 5 remote package.json entries4 overlapped, broken one isolated, via recorded
  • update -p -y after upstream changesOnly the 3 changed skills reinstalled
  • update -g -y, 1 unreachable source4 overlapped checks; only changed skills updated
  • Temp dirs after runs and Ctrl-CIsolated TMPDIR empty, including after SIGINT with 4 clones live
  • pnpm type-check / format:check
node src/cli.ts experimental_install   # 6 sources, each slowed by 1.5s
●  Restoring 6 skills from skills-lock.json into .agents/skills/
◇  Downloaded 6 sources
◆  Restored alpha
◆  Restored beta
◆  Restored delta
◆  Restored eps
◆  Restored zeta
◆  Restored gamma
WALL=6.83s
--- proxy log
  32.29 open  #2 inflight=1
  32.40 open  #3 inflight=2
  32.43 open  #4 inflight=3
  32.46 open  #5 inflight=4
  34.08 close #2 inflight=3
  34.16 open  #6 inflight=4
  ...
  36.20 close #6 inflight=0
exit 0

1 minor3 nits

  • minorPre-existing: AGENTS.md says bare skills install restores from the lockAGENTS.md

    It goes to add and fails with 'Missing required argument: source'; only experimental_install restores.

nits3 small things, none blocking
  • nitupdate prints the 'Checking skills from source' lines all at oncesrc/update.ts

    These lines now appear together after the fetch finishes; 'Fetching N sources…' is the only progress shown while waiting. install/sync also stopped printing per-source 'Found N skill(s)'.

  • nitFailure-isolation improvement isn't in the commit messagesrc/install.ts

    A failing well-known source no longer ends experimental_install (on the base it did).

  • nitSKILLS_CLONE_CONCURRENCY also caps well-known HTTP fetchessrc/concurrency.ts

    The env var name says clone, but well-known downloads share the same limit.

Not run yet

  • UndoUndoHow hard is it to take back?/pr merge-danger
  • SecuritySecurityDoes it open a door?/security-review/cso