Skip to content

fix(efficacy): a live judge, scoring or security run refuses a second (#1832) - #1834

Merged
braboj merged 5 commits into
mainfrom
fix/judge-run-lock
Sep 19, 2026
Merged

braboj merged 5 commits into
mainfrom
fix/judge-run-lock

Conversation

@braboj

@braboj braboj commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Closes #1832.

Two runs of one tool against one scoring area deleted each other's working
directories, and nothing refused the second. A run now claims the area first.

The claim

harness.claim_area(area, tool) creates <tool>.running in the scoring area
exclusively, holding the run's pid, argv and start time, and release_area
removes it on exit, failure included. When the marker already exists:

  • it names a process whose command line runs the same tool → refused,
    naming the pid and start time
  • it names a process that has ended, or a pid now held by something that is
    not the tool → taken over, and the takeover is printed
  • it holds no readable record → refused, naming the file to remove

The command line is read as well as the pid because Windows reuses pids
quickly; a pid check alone would leave a crashed run's marker looking live
behind whatever process got its number. A process listing that does not show
the run itself is refused rather than read as "nothing is live".

Scope: three tools, not one

The issue asked whether score.py, security.py or harness.py share the
shape. Read before scoping:

Tool Shares it? Why
judge.py yes same next label; build_bundle clears the bundle a live judge reads
score.py yes clears and re-clones hidden-suite at every start, so two runs collide even on different trials; clears each trial's tree and venv
security.py yes clears each trial's venv under security/
harness.py no refuses a workspace that already exists, so a second run of a trial stops before touching the first

Each of the three takes the claim before its first destructive step. The
claim is per tool, so a judge run and a security run on one root still run
side by side. Not covered: score.py --rescore clearing a tree a live judge
is reading, which is two different tools.

Checks

harness.py --self-test plants a live stand-in (a sleeping interpreter whose
command line names the tool), confirms the process listing shows it, and
checks: a live run refuses and keeps its marker; the same live pid under
another tool's marker is taken over; an ended run is taken over; a run removes
its own marker and no other; an unreadable marker refuses.

judge.py, score.py and security.py --self-test each run their own main
against a planted live claim and require the refusal, the marker unchanged,
and nothing built (no bundle, no suite clone, no security/). The judge's
runs dry, so a regressed claim calls no model.

Controls, each mutation asserted to land and reverted from a commit:

Mutation Check that went FAILED
judge claims nothing a live judge.py run refuses a second
score claims nothing a live score.py run refuses a second
security claims nothing a live security.py run refuses a second
liveness always false a live run refuses a second, and keeps its marker
liveness ignores the command line a live pid that is not the tool is taken over
release removes any marker a run removes its own marker, and no other

Two of these first surfaced as a crash rather than a FAILED line, so the
checks now report: a refused claim returns None, and an exception past the
claim counts as not refusing.

Self-tests: harness 87/87, judge 20/20, score 50/50, security 20/20. Smoke
33/33. Conformance 22 passed, 0 failed; its three readings are read:
release ordering (no other open PR), off-limits paths (none, all six files
under tests/efficacy/), and existing tests changed. On the last: every
existing self-test gains a check and none loses or weakens one; the 17 removed
lines are import lines, three self-test docstrings extended, and the two README
paragraphs that told a person never to start a second run, replaced by the
refusal.

Docs

tests/efficacy/README.md and tests/efficacy/control/README.md describe the
refusal and takeover in place of the "never start a second run" warning.

No CHANGELOG entry: the benchmark's tooling is not observable by a consuming
project.

🤖 Generated with Claude Code

braboj and others added 5 commits September 19, 2026 16:36
…#1832)

Each tool clears a trial's working directory before rebuilding it, so two
runs against one scoring area deleted what the other was reading. A run now
claims the area with a marker naming its pid; a marker whose process still
runs that tool refuses the run, and one whose process ended or whose pid was
reused is taken over.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The judge's refusal check runs dry, so a claim that stopped refusing builds
a bundle and calls no model.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@braboj braboj added bug Something isn't working P2 Medium — important but not blocking labels Sep 19, 2026
@braboj
braboj merged commit 0842173 into main Sep 19, 2026
1 check passed
@braboj
braboj deleted the fix/judge-run-lock branch September 19, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working P2 Medium — important but not blocking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refuse a judge run against a root that already has one live

1 participant