Skip to content

fix(efficacy): an empty final message carries the judge CLI's cause (#1768) - #1835

Merged
braboj merged 2 commits into
mainfrom
fix/judge-cli-error
Sep 19, 2026
Merged

braboj merged 2 commits into
mainfrom
fix/judge-cli-error

Conversation

@braboj

@braboj braboj commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Closes #1768.

A codex exec refused by the model creates its --output-last-message file
before the call and leaves it empty. judge_bundle then reported the final message was not JSON: and nothing more, and the cause, which the CLI printed
to standard error, was dropped.

Change

  • An empty or unreadable final message now records the final message was empty (or not JSON: <text>) followed by the CLI said: <tail>, the last
    400 characters of what the CLI printed.
  • cli_said reads that tail for both failure branches, the missing-file one
    included, which already carried it inline.
  • The claude backend already reports its standard error when it prints
    nothing; unchanged.

Check

The self test's stand-in CLI gains a refusal mode, set by
FAKE_CODEX_REFUSES: it creates the message file empty, writes the cause to
standard error and exits 1, as the real CLI did. The new check requires the
file to exist and be empty (so the failure is read from the no-JSON branch,
not the missing-file one), the error to say was empty, and the planted cause
to be in it.

Control: with the cause removed from the error (cli_said(outcome) replaced
by "", landing asserted, reverted from a commit), that check goes FAILED,
20/21.

judge.py --self-test 21/21. Smoke 33/33. Conformance 22 passed, 0 failed;
readings: no other open PR, no off-limits path, and the one existing test
changed is the stand-in CLI, which gains the refusal mode behind an
environment variable and behaves as before without it.

No CHANGELOG entry: the benchmark's tooling is not observable by a consuming
project.

🤖 Generated with Claude Code

braboj and others added 2 commits September 19, 2026 22:18
…1768)

A CLI refused by the model creates its final-message file and leaves it
empty, so the failure read 'not JSON:' and nothing more; the cause was only
in what the CLI printed. Both failure branches now carry the tail of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@braboj braboj added bug Something isn't working P3 Low — nice to have labels Sep 19, 2026
@braboj
braboj merged commit 0c29e9c into main Sep 19, 2026
1 check passed
@braboj
braboj deleted the fix/judge-cli-error branch September 19, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working P3 Low — nice to have

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report the judge CLI's error when its final message is empty

1 participant