fix(efficacy): a live judge, scoring or security run refuses a second (#1832) - #1834
Merged
Merged
Conversation
…#1832) Each tool clears a trial's working directory before rebuilding it, so two runs against one scoring area deleted what the other was reading. A run now claims the area with a marker naming its pid; a marker whose process still runs that tool refuses the run, and one whose process ended or whose pid was reused is taken over. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The judge's refusal check runs dry, so a claim that stopped refusing builds a bundle and calls no model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1832.
Two runs of one tool against one scoring area deleted each other's working
directories, and nothing refused the second. A run now claims the area first.
The claim
harness.claim_area(area, tool)creates<tool>.runningin the scoring areaexclusively, holding the run's pid, argv and start time, and
release_arearemoves it on exit, failure included. When the marker already exists:
naming the pid and start time
not the tool → taken over, and the takeover is printed
The command line is read as well as the pid because Windows reuses pids
quickly; a pid check alone would leave a crashed run's marker looking live
behind whatever process got its number. A process listing that does not show
the run itself is refused rather than read as "nothing is live".
Scope: three tools, not one
The issue asked whether
score.py,security.pyorharness.pyshare theshape. Read before scoping:
judge.pybuild_bundleclears the bundle a live judge readsscore.pyhidden-suiteat every start, so two runs collide even on different trials; clears each trial's tree and venvsecurity.pysecurity/harness.pyEach of the three takes the claim before its first destructive step. The
claim is per tool, so a judge run and a security run on one root still run
side by side. Not covered:
score.py --rescoreclearing a tree a live judgeis reading, which is two different tools.
Checks
harness.py --self-testplants a live stand-in (a sleeping interpreter whosecommand line names the tool), confirms the process listing shows it, and
checks: a live run refuses and keeps its marker; the same live pid under
another tool's marker is taken over; an ended run is taken over; a run removes
its own marker and no other; an unreadable marker refuses.
judge.py,score.pyandsecurity.py --self-testeach run their ownmainagainst a planted live claim and require the refusal, the marker unchanged,
and nothing built (no bundle, no suite clone, no
security/). The judge'sruns dry, so a regressed claim calls no model.
Controls, each mutation asserted to land and reverted from a commit:
Two of these first surfaced as a crash rather than a FAILED line, so the
checks now report: a refused claim returns None, and an exception past the
claim counts as not refusing.
Self-tests: harness 87/87, judge 20/20, score 50/50, security 20/20. Smoke
33/33. Conformance 22 passed, 0 failed; its three readings are read:
release ordering (no other open PR), off-limits paths (none, all six files
under
tests/efficacy/), and existing tests changed. On the last: everyexisting self-test gains a check and none loses or weakens one; the 17 removed
lines are import lines, three self-test docstrings extended, and the two README
paragraphs that told a person never to start a second run, replaced by the
refusal.
Docs
tests/efficacy/README.mdandtests/efficacy/control/README.mddescribe therefusal and takeover in place of the "never start a second run" warning.
No CHANGELOG entry: the benchmark's tooling is not observable by a consuming
project.
🤖 Generated with Claude Code