[rush-daemon] Read Linux process group state from procfs instead of ps

[Sean Larkin (TheLarkInn)]

Summary

On Linux, the daemon checks that a rushx or global command's process group has exited by running ps --sid <pgid> -o stat=. When procps ps is missing (node:*-slim, distroless, many CI images) or is busybox's ps (which rejects --sid), that check rejects. The rejection is then recorded as a workspace resource cleanup failure. After that, the session is retired, every later request replays the old Failed to release resources … spawn ps ENOENT error, and daemon stop leaves the process running with its listener bound.

This PR reads group membership and state directly from /proc and keeps ps only as a fallback for when procfs cannot be listed.

Root cause

• GlobalCommandExecutionContext#trackChildAsync SIGKILLs the child's group, then calls waitForLinuxProcessGroupExitAsync. If kill(-pgid, 0) still sees the group (the killed members have not been reaped yet), that function execs ps. The window is timing-dependent, which is why the bug is intermittent.
• ENOENT or an unsupported flag rejects the promise. recordWorkspaceRequestCleanupFailure then retires the session. From that point the host deliberately keeps ownership because it cannot prove the group was joined.

Fix

• LinuxProcessGroupExit.ts: a new procfs path lists /proc, reads each <pid>/stat, and parses after the last ) so command names with spaces or parentheses are handled. It collects field 3 (state) for entries whose field 6 (session) equals the group. Entries that exit between the listing and the read are ignored. The existing zombies-only and timeout semantics are unchanged.
• The procfs reader is injectable: waitForLinuxProcessGroupExitAsync(groupId, timeoutMs, procfs: ILinuxProcfsReader). It is internal and not exported from the package index.
• ps --sid still runs, but only when procfs cannot be listed. Its failure semantics are unchanged.

Based on A10's prototype (board #204).

Tests

• LinuxProcessGroupExit.test.ts: the existing ps tests now run with an injected "no procfs" reader. New procfs tests:
  • only zombies remain → the wait completes and ps never runs
  • a live member is waited for
  • the wait is bounded
  • a command name containing ) S 1 <sid> <sid> is not misparsed
  • a pid that exits between listing and reading is ignored
  • a ps that would fail with ENOENT is never consulted while procfs is readable
• NativeMutationCleanupFailure.test.ts injects a hidden procfs, so its injected ps inspection or timeout failure is still exercised end to end.

Linux validation (WSL Ubuntu 24.04, node 22.23.2)

• rush build --to @rushstack/rush-daemon --to @rushstack/rush-cli-client: success; the build includes lint.
• rush test --only @rushstack/rush-daemon: success (1m27s).
• Targeted run: LinuxProcessGroupExit 17/17 and NativeMutationCleanupFailure 4/4 pass.
• Deterministic before/after: waitForLinuxProcessGroupExitAsync called on a real, live detached group (sleep 1.5), with a PATH that has no ps:
  • before (toolchain @60007c9a8c): REJECTED after 6ms: spawn ps ENOENT. In the daemon, this is the error that poisons the session.
  • after: resolved after 1516ms, which is the correct wait for the live member.
• End-to-end repro: 4 rounds of rushx bg (a redirected backgrounded child), then rushx build, then daemon stop, with the daemon PATH lacking ps. It gave 0/4 failures both before and after, because the race window did not open under the current load. A10 observed the same thing (0/8 later versus 3/3 earlier; board Add three more events: preRushInstall, postRushInstall, preRushBuild. #204). The direct probe above covers the failing path deterministically.

Scope and follow-ups (intentionally not in this PR)

• Sticky, session-wide cleanup failure (WorkspaceRequestResources.ts) and daemon stop leaving the host alive after a failed join (RushDaemonHost#closeOnceAsync). Both are deliberate safety invariants today: a host that cannot prove an owned group was joined keeps its ownership rather than becoming reclaimable over live children, and existing tests (DaemonShutdownOwnership, NativeMutationCleanupFailure, GlobalCommandRequestRouter) assert this. This PR removes the false-positive trigger (a missing or incompatible ps), so the invariant now fires only when a group is genuinely still live after SIGKILL plus 5s, or cannot be probed (EPERM). Suggested follow-up:
  • fail only the originating request, with its error printed (today stderr is empty)
  • give the next request a distinct "workspace retired, restart the daemon" error instead of replaying the old one
  • add a bounded close path so daemon stop terminates, or surfaces a clear diagnostic, after a genuine join failure
• The rushx "owned group" policy SIGKILLs processes a script deliberately backgrounds (differs from native rushx; board Add a way to serialize the rush configuration #156/If conflicting pinned versions are defined in 'rush.json' and 'pinnedVersions.json', prefer the latter #159).

Fixes #6072

This came out of the automated rushd Linux analysis ("Rushd Hive").

[Copilot]

Copilot review overview

🟡 Changes recommended

Procfs read errors can hide live processes, and process-table reads use unbounded concurrency.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Propagate non-ENOENT errors when checking session members · New
• Bound filesystem scans and stop after finding a live member · New

What changed in this PR

Replaces Rush daemon’s Linux process-group inspection with procfs, retaining ps as a fallback.

Changes:

• Adds injectable procfs-based process-state inspection.
• Expands procfs and fallback test coverage.
• Adds a patch change entry.

File	Description
LinuxProcessGroupExit.ts	Implements procfs inspection and ps fallback.
LinuxProcessGroupExit.test.ts	Tests procfs parsing, races, waiting, and fallback.
NativeMutationCleanupFailure.test.ts	Preserves fallback failure-path coverage.
fix-rushd-procfs-process-group-checks_2026-09-24-03-00.json	Records the patch change.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.