fix(efficacy): the judge reads each tree against the spec it was built from - #1839
Merged
Merged
Conversation
…t from The judge copied whatever SPEC.md was current into every bundle, so once the specification is extended for round 3, a tree an earlier round built would be judged against features it was never asked for. The bundle now carries the tree's own copy, the one the harness gave its agent, checked against the specifications rounds ran (line endings normalised); a tree carrying an edited or no specification is refused. Each judging records the specification's sha256 and round. Part of #1767 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1767 — a precondition for extending
tests/efficacy/SPEC.mdfor round 3.Defect
build_bundlecopied whateverSPEC.mdwas current into every bundle. Once the spec is extended, every tree an earlier round built — including the control trees whose gpt-6-astra top-up is owed on 2026-09-25 — would be judged against sign-in, customers and erasure it was never asked for.Change
SPEC.md, the copy the harness gave its agent, with line endings normalised to LF.SPECSlists them by sha256 (todayebecfff1…, "rounds 1 and 2"), and the current file is accepted too. An edited or missing copy refuses the trial, beside the existing leak refusal (refusal()).bundle.speccarries the sha256 and round.Verified
C:\efficacyclassify as "rounds 1 and 2". One of them, round 1'sC2, carries a CRLF copy; stripping CR makes it byte-identical, which is why the digest is taken after normalising.No CHANGELOG entry: benchmark tooling, nothing a consuming project sees.
🤖 Generated with Claude Code