[rush-daemon] Invalidate warm operations whose declared outputs were deleted or changed

[Sean Larkin (TheLarkInn)]

Summary

A warm rushd graph decided "up to date" by comparing input state hashes only. After rm -rf <project>/lib, git clean -xdf or heft clean, the next rush-client build (even --to <project>) exited 0 with 0 operations and left the outputs missing. Native rush build with the build cache restores them.

Root cause

Build outputs are git-ignored, so they are not part of any operation state hash. The warm skip in PhasedOperationPlugin (shouldEnableOperation) compares a retained successful record's hashes. WorkspaceEngineComponentFactory only invalidates the operations that ProductionDaemonRequestResolver.getChangedOperations maps, and that function compares own-state hashes. Nothing noticed that outputs had disappeared.

Fix

• New OperationOutputFingerprints (rush-daemon):
  • After each iteration, for every operation whose record from that iteration became the retained Success or FromCache result, it records a cheap fingerprint of the operation's declared outputFolderNames: one stat per folder, capturing existence, inode and mtime.
  • Operations that were disabled in the iteration keep their earlier fingerprint. That way a deletion made during an unrelated build is not absorbed.
• ProductionDaemonRequestResolver's invalidation mapper now returns the hash-changed operations plus the retained operations whose output fingerprint is missing or changed.
  • Only those operations are invalidated. They are restored from the build cache, or re-executed.
  • Dependents stay untouched when their inputs are unchanged.
• With no build cache, Rush's legacy skip detection would report the invalidated operation as SKIPPED while its outputs are still gone. The tracker therefore removes that operation's package-deps_<id>.json legacy skip record, which forces execution. The legacy skip plugin deletes the same record before it executes an operation.
• README: documents the behavior and its limits.

What the fingerprint detects: deletion or recreation of an output folder, and adding, removing or renaming its direct children (this covers rm -rf lib, rm -rf lib/*, git clean, heft clean and new emitted files). It does not detect in-place edits of nested output files. Operations without outputFolderNames are not covered, which matches the build cache.

Tests

New tests in ProductionDaemonRequestResolver.test.ts, against the real native engine:

• No-cache fixture (legacy skip): full build, then a warm no-op build (not scheduled). Next, rm -rf a/lib followed by --only c, which is still not scheduled. Then --to b re-executes only a and restores a/lib; b does not run. A stray file added to c/lib re-runs only c. A final no-op build is not scheduled, and the graph identity is unchanged.
• Cache fixture: rm -rf a/lib → --to b reports exactly one non-skipped result, a (compile) FROM CACHE. No script executions, lib is restored, and the next build is not scheduled.

Linux validation (WSL Ubuntu-24.04)

• rush build --to @rushstack/rush-daemon && rush test --only @rushstack/rush-daemon passed. heft test reports 365 passed, 0 failed (366 total). The targeted ProductionDaemonRequestResolver suite has 31/31 passing.
• Before/after comparison: mkws-synth --projects 12 workspaces (build cache on), one lab invocation, script $LAB/fixes/output-invalidation-repro.sh. "Before" is the unfixed toolchain rush-client; "after" is apps/rush-cli-client/bin/rush-client from this branch.

step	before	after
warm no-op (x8)	0 ops, median ~2.0 s	0 ops, median ~2.2 s
rm -rf packages/p03/lib → build	rc=0, 0 ops, lib missing	rc=0, 1 restored from cache (p03 only), lib restored
build --to p03 again	0 ops, lib missing	0 ops (no false positive)
echo stray > p05/lib/stray.txt → build	0 ops	1 restored (p05 only)
next build	0 ops	0 ops

• No-op overhead: the host was heavily loaded (load average around 87, with several fix sessions installing in parallel), so end-to-end wall-clock numbers are dominated by noise. I measured the fingerprint cost directly instead. Stat-ing 1,176 candidate output folders (6 per project across all 196 rushstack projects) takes about 4.1 ms per request on the loaded host, which is roughly 3.5 µs per folder. The 12-project synthetic workspace costs microseconds.

Known unrelated CI failures

The Ubuntu jobs currently fail in PhasedRequestBatching.test and DaemonRequestWirePhased.test with The summary cannot be prepared until after close() is called (from PhasedRequestSummary.ts). main fails the same way since #6068; the fix is in #6095. Once #6095 merges, this branch will be updated from main and CI re-run.

Optional follow-ups

• Stronger fingerprints, for example the output file list and sizes the cache plugin computes when packing, to catch in-place nested edits. The cost is O(files) per request.
• Mapping project-watcher events under output folders directly to the owning operation.

Fixes #6058

This came out of the automated rushd Linux performance and behavior analysis ("Rushd Hive", board #103/#117; prototype by agent A01).

[Copilot]

Copilot review overview

🟡 Changes recommended

A cleanup failure can permanently discard fingerprint state before the operation is invalidated.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

• Delete fingerprint only after successful legacy skip file removal · New

What changed in this PR

Adds output-folder fingerprinting so warm rushd builds invalidate operations whose declared outputs changed or disappeared.

Changes:

• Tracks output-folder identity and modification time.
• Invalidates and reruns/restores affected operations.
• Adds integration tests, documentation, and change metadata.

File	Description
libraries/​rush-daemon/​src/​OperationOutputFingerprints.ts	Implements output fingerprint tracking and legacy-state cleanup.
libraries/​rush-daemon/​src/​ProductionDaemonRequestResolver.ts	Incorporates output changes into invalidation mapping.
libraries/​rush-daemon/​src/​test/​ProductionDaemonRequestResolver.test.ts	Tests cache and no-cache restoration behavior.
libraries/​rush-daemon/​README.md	Documents output invalidation behavior and limitations.
common/​changes/​@rushstack/​rush-daemon/​output-invalidation_2026-09-23.json	Records the patch change.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.