Lab: swamp extension install/pull does not prune source files dropped between versions, causing catalog to re-index orphans

Summary

When an extension drops a source file between versions, swamp extension install (lockfile-driven) and swamp extension pull --force both fail to remove the orphan files from .swamp/pulled-extensions/<name>/. The catalog rebuild then re-indexes those orphans, which means even a full cache wipe (rm -rf .swamp/bundles && rm .swamp/_extension_catalog.db*) followed by swamp extension install does NOT actually repair the system — the next model method run still resolves to the orphaned older bundle entry.

This is upstream of the bug filed in swamp Lab #201. That issue describes the catalog retaining stale rows; this one explains why the catalog keeps coming back broken even after a wipe: the source-of-truth filesystem is dirty, and install treats source files as additive.

Reproduction

Pull @hivemq/harvester/[email protected] — ships two files: models/harvester/kubeconfig.ts and models/harvester/fetch_kubeconfig.ts.
Update to @hivemq/harvester/[email protected] (or pull --force) — the new version manifest only declares models/harvester/kubeconfig.ts.
Verify: ls .swamp/pulled-extensions/@hivemq/harvester/kubeconfig/models/harvester/ shows BOTH files. fetch_kubeconfig.ts is the orphan from the older version; install/pull never removed it.
Even a hard reset does not help. Sequence we tried (in order):
- rm -rf .swamp/bundles — clears bundle dir.
- rm -f .swamp/_extension_catalog.db* — clears the catalog.
- swamp extension install — recreates everything from the lockfile.
- Result: pulled-extensions STILL has both files, catalog STILL has two bundle_types rows for the same model type, and model method run still non-deterministically resolves to the older row's bundle path.
The only repair that worked was: rm -rf .swamp/pulled-extensions/@hivemq/harvester/kubeconfig + swamp extension pull --force. Once the orphan source file was physically gone from disk, the catalog rebuild produced exactly one row and resolution worked.

Why this matters

Combined with swamp Lab #201, the user-visible failure mode is:

An extension legitimately removes a file in a new version.
The user runs swamp extension update (or pull --force, or rm + pull, or wipes .swamp/bundles + extension install) — every "correct" recovery path.
All of them silently leave the orphan source file on disk.
The catalog re-indexes the orphan and still resolves to the old version's bundle.
The user / agent has no way to discover this without inspecting pulled-extensions manually.

In our session this consumed ~30 minutes and required filesystem surgery on a directory swamp owns. An agent that respects swamp's boundaries had no path to fix it via swamp commands.

Asks

swamp extension install should compute the set of source files declared by the lockfile-pinned version and remove anything else under pulled-extensions/<name>/ that does not appear in that set. Same for nested directories that become empty.
swamp extension pull --force should do the same — treat the manifest as authoritative and prune the destination directory to match.
swamp extension update likewise.
swamp doctor extensions should report orphan source files (files under pulled-extensions/<name>/ that aren't declared by the manifest) as a warning with a fix-it hint.

Environment

swamp 20260501.010449.0-sha.c24fb601
Repro extension: @hivemq/harvester/kubeconfig (versions 2026.04.24.32 → 2026.05.01.39)

swamp Lab #199 — warn on outdated extensions
swamp Lab #200 — surface native cache-refresh path
swamp Lab #201 — catalog retains stale rows; this issue is the underlying root cause for why the rebuild path described in #201 doesn't recover

swamp issue get should rate-limit unauthenticated users instead of blocking

swamp issue get should not require authentication

telemetry: emit child entries for follow-up action method invocations

Publish release-candidate / unstable extension versions

extension source rm leaves stale @local/. rows in catalog, blocking later pulls of registry types

extension push drops binaries: field from re-emitted archive manifest.yaml

macOS launchd autoupdate (club.swamp.autoupdate) silently fails — binary stays stale

Clicking @hivemq/honeycomb extension card on /extensions shows 'Something went wrong'

Docs: Update model-definitions.md and workflows.md for direct type execution

Docs: document binaries manifest field in extension-manifest.md

Add an official @swamp/ssh extension for general-purpose SSH (brownfield-friendly)

Accept and display binaries field from extension push metadata

Direct type execution: collapse model create + method run into one command

Per-method telemetry events for workflow runs

Workflow run liveness: orphaned 'running' records when originating CLI process dies mid-run

Provide a CLI-shape primer for AI agents to reduce rediscovery overhead

quality rubric: don't penalize extensions whose upstream constrains them to a single platform

Extension update rejects multiple .ts files extending the same target type within one local extension (regression)

swamp extension push deadlocks when invoked from inside a swamp workflow step

Collective-scoped auth keys + OIDC federation for CI publishing

forEach self.* in modelIdOrName not resolved in runtime execution path

Award leaderboard points for referrals and collective invites

Add agent harness detection and AiTool to telemetry

Workflow-level runtime expressions (env.*, vault.*) not resolved in driverConfig — docker driver receives literal ${{ ... }} strings

Implement W5: Per-fingerprint import URLs + subprocess test harness (extension catalog rearchitecture)

swamp config set crashes with YAML serialization error

datastore compact: VACUUM fails in compiled binary (SQLITE_LIMIT_ATTACHED=0)

Repo-level version gating: minSwampVersion high-water mark for team consistency

Docs: document self.* expressions in modelIdOrName during forEach

Docs: How-to guide for background autoupdating

Manifest version bumps silently ignored for existing local extension aggregates

materialiseExtensions misclassifies pulled rows when manifest name collides with a pulled extension

Local extension edits don't reliably trigger rebundle

Missing unique indexes on user.email and user.username allow duplicate users

Resolve self.* expressions in modelIdOrName during forEach expansion

discord-bot double-sends sign_up notifications

Discord bot sends duplicate signup notifications

Add 'swamp workflow list' as alias for 'swamp workflow search'

Add 'swamp auth status' as alias for 'swamp auth whoami'

Extension bundle cache does not invalidate on source edits

extension pull fails on @local/[email protected] phantom-claim collision when local repo has its own extension

Lab search by numeric issue ID returns no results

W3 sourceToRow writes empty source_mtime — should carry filesystem mtime through Source entity

Warm-start rebundleAndUpdateCatalog should respect terminal RowStates set by reconcile

Implement W4: KindAdapter + unified loader (extension catalog rearchitecture)

Docs: add vault read-secret command to reference manual

Extension layer garbage collection: prune catalog rows + evict orphaned bundles

Docs: document workflow concurrency limits in reference manual

Locally-sourced extension: source_mtime updates without regenerating stale bundle

Extension push: allow shipping executable host helpers (bin/mudroom blocker)

Vault expressions silently deliver __SWAMP_VSEC__ sentinels under the docker driver

First-class shell-shim support for extensions, with registry-level visibility

Local extension model bundles don't rebuild when source changes (no rebuild CLI; manual cache delete breaks the runner)

Configurable concurrency limits for workflow fan-out (forEach, parallel jobs/steps)

feat(security): redact sensitive method arg values from audit log

docs: document swamp datastore compact and GC WAL behaviour

UAT: swamp datastore compact reclaims WAL and catalog space

Plan v4 step 9 literal test untestable under current catalog PK semantics

Cross-process concurrency stress for W2 lifecycle services

UAT additions for W2 lifecycle services (Install/Remove/Upgrade)

Implement W3: ReconcileFromDisk + freshness-as-aggregate-query (extension catalog rearchitecture)

doctor extensions repair: clean catalog-only orphans

Extensions should be able to ship Claude Code skills

Pre-existing TOCTOU windows in YAML repo walkers (findAll directory level + findById)

`swamp datastore setup` migration is not resumable / leaves the repo in a partial state on failure

Built-in models must honor AbortSignal so --timeout works in practice

Reader-lock or lock-free read path for data list/get/search/query

data search: surface jobTag alongside workflowTag and stepTag (follow-up to #237)

global-arg silently strips unknown keys; should reject via strict Zod schema

redmine extension: adopt state machine patterns from @magistr for agent-driven workflows

performance degrades significantly with large SQLite catalog

summarise: timeout/hang on repos with many workflow runs

extension quality/fmt fail on pulled extensions (path mismatch)

vault: add read-secret CLI command for agent-driven secret retrieval

data query: stepName and jobName fields always empty in CEL results

Extension ecosystem: shared utility library, version sync tooling, and HTTP resilience patterns

Agentic CLI improvements: --json stdout isolation, array inputs, and --repo-dir consistency

data delete fails with "Directory not empty (os error 39)" when concurrent writes are active

Extract LockfileRepository (W2 prequel for swamp-club#231)

Extend DatastoreSyncService.markDirty() with optional relPath argument

Workflow-level runtime expressions (env., vault.) not resolved in driverConfig — docker driver receives literal ${{ ... }} strings