[rush-lib] Skip build cache writes when an operation's inputs changed during execution - #6086
Merged
Sean Larkin (TheLarkInn) merged 2 commits intoSep 24, 2026
Conversation
… during execution The cache key is derived from the iteration's inputs snapshot, but the outputs were written under that key without re-verifying the inputs. Record a stat signature of each cacheable operation's tracked input files right after the snapshot and refuse the cache write if it changed. Fixes #6073 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
New files, absolute additional-input paths, and downstream operations can still produce poisoned cache entries.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Adds a stat-based guard to prevent Rush from writing cache entries when operation inputs change during execution.
Changes:
- Records tracked-input stat signatures before execution.
- Rechecks signatures before normal and cobuild cache writes.
- Adds focused signature tests and a patch change entry.
| File | Description |
|---|---|
CacheableOperationPlugin.ts |
Integrates input-change detection into cache writes. |
InputFilesStatSignature.ts |
Implements stat-signature generation and comparison. |
InputFilesStatSignature.test.ts |
Tests signature behavior for file changes. |
fix-build-cache-poisoning-race_2026-09-24.json |
Records the Rush patch change. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…, and resolve absolute input paths Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Mo Jazayeri (mojaza)
approved these changes
Sep 24, 2026
Sean Larkin (TheLarkInn)
deleted the
thelarkinn-fix-build-cache-poisoning-race
branch
September 24, 2026 21:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Fixes a build cache poisoning race. If a source file is edited or created while a build is running, Rush could store the post-edit outputs under the cache key computed from the pre-edit inputs. Any later build with the original inputs (local, or through a shared/remote cache) then restores wrong outputs. Native and daemon (
rush-client/rushd) builds are both affected.Root cause
An operation's cache key comes from the iteration's inputs snapshot, taken when the iteration is scheduled.
CacheableOperationPluginwrites whatever is in the output folders after the operation finishes (OperationBuildCache.trySetCacheEntryAsync) and never re-verifies that the inputs still match the snapshot. Consumers' keys also embed the dependency's pre-edit state hash.Fix
beforeExecuteIterationAsyncruns right after the snapshot. For each enabled, cache-writable operation, it records:(size, mtimeNs, ino), fromgetTrackedFileHashesForOperation. AbsolutedependsOnAdditionalFilespaths are resolved correctly.git ls-files --others --exclude-standardfinds untracked, non-ignored files among them. Only the new entries are checked, and the operation's output folders are excluded. If Git is unavailable or fails, the new entries are assumed to be inputs.Input files changed while this operation was executing; not writing a build cache entry., skips the write, and setsisCacheWriteAllowed = false. The existing propagation then blocks consumers' writes too. Operation status is unchanged. The next snapshot sees the new content, so the next native, incremental or daemon iteration re-runs the operation.InputFilesStatSignature.ts.Overhead: at iteration start, one
statSyncper tracked input file plus onereaddirSyncper input folder, for each cache-writable operation. Operations that actually execute pay the same again before writing. Git is spawned only when new entries appear in an input folder, e.g.temp/on a clean build, and only for those paths. Nothing is re-hashed.Tests
InputFilesStatSignature.test.ts(10 tests):hasUntrackedGitFilesagainst a real temp Git repo: ignored files and excluded output folders return false, untracked files and folders return trueInputFilesStatSignature|OperationGraph|OperationExecution|PhasedScriptAction|BuildCache: 149 passed / 0 failed.rush build --to @microsoft/rush-lib, including lint, passes.Linux validation (WSL Ubuntu-24.04)
The repro is a variant of
cacherace.shon a fresh synthetic workspace (8 projects, 500 ms per operation). It either editsp06/src/index.jsor createsp06/src/zz-new.jsafter the snapshot but before p06 runs. It then reverts the source, deleteslib, and rebuilds natively from the shared cache.RUSH_DAEMON=1 rush-client), beforebuild --to p07): before, p07 was restored from a stale entry; after, p07 re-executes.rm -rf lib, 8 of 8 operations were restored from the build cache.Known limitations / optional follow-ups
beforeExecuteIterationAsyncaren't detected. Recording stat identity inside the snapshot itself would close that window.incrementalBuildIgnoredGlobs(but not.gitignore) conservatively skips the write.dependsOnAdditionalFilesglob in a folder that had no matches aren't detected.This change is independent of #6064, which also touches
CacheableOperationPlugin.This came out of the automated rushd Linux performance/behavior analysis ("Rushd Hive").
Fixes #6073