Skip to content

ci: improve release job execution #46387

Description

@jamietanna

Via Claude Fable 5:

Details
Here's where the time actually goes in both runs, and what would move the needle.

## Push/release run (33200439623) — 16 min wall-clock

The critical path is: setup → build → wait for **test-macos (4/4)** (finishes 18:47:48) → coverage-threshold → test-success → **release (8m29s)** → deploy-docs. Two things dominate:

**1. The release job's docker build/push is ~5m25s, running fully uncached.** Inside the 7m14s `semantic-release` step, the multi-arch (amd64+arm64, slim+full) image build takes 172s and the push takes 153s. The reason it's slow: `cache-from` in `tools/docker/bake.hcl:62-65` has been commented out since PR #34334 (March 2025, "I'm pretty sure this causes the recent build errors" — a hunch, not a diagnosed failure), yet `cache-to` still writes `ghcr.io/renovatebot/docker-build-cache:renovate{,-full}` on every single release. So every release pays to *write* a registry cache that nothing ever *reads*. The Dockerfile is well-shaped for caching — pinned base digests, `install-tool node/pnpm` and `pnpm fetch` layers keyed only on the lockfile, cross-compilation on `$BUILDPLATFORM` — so re-enabling `cache-from` should collapse most of the build stage to cache hits between releases. Buildx/BuildKit have moved a long way since the issue that prompted disabling it; I'd re-enable it (perhaps only `docker-build-cache:*` refs, not the image refs) and watch a few releases. Estimated saving: 2–4 min per release, many times a day.

Related waste: the `build-docker` job (3m43s) builds the same two platforms earlier in the same run with `--load`, then throws the result away — no cache is shared with the release build. With `cache-from` re-enabled they'd at least share the registry cache; a step further would be having `build-docker` also `cache-to` a per-branch cache so the release build hits even fresher layers.

**2. macOS test sharding is badly imbalanced and gates everything.** Ubuntu's 16 shards finish in ~1–2 min, but macOS packs the same 16 shards into 4 runners by *count*, not by measured duration (`tools/test-shards.ts:135-142`, `scheduleItems`). Result in this run: group 3/4 finished in 2m19s while 4/4 took 5m54s — and 4/4 is what held back coverage-threshold, test-success, and therefore the release start. Balancing groups by recorded shard durations (or just hand-tuning the grouping) would pull the gate from ~6 min toward ~4 min; the code comment says up to 5 macOS runners are allowed but only 4 are used, so going to 5 helps too.

Smaller: `codecov` and `coverage-threshold` have `needs: [test]`, which includes the macOS jobs even though only ubuntu shards produce coverage (`coverage: os === 'ubuntu-latest'`). If macOS ran as a matrix the coverage jobs don't wait on, their ~35s would overlap macOS runtime instead of following it. And the release job rebuilds the mkdocs site (21s) that `build-docs` already built — reusable if the release-stamped version isn't needed there.

Realistic outcome: **~16 min → ~10–11 min** for push builds, mostly from the docker cache and macOS balancing.

## PR run (35739842987) — 3m30s

This is already lean — macOS/Windows, `build-docker`, and release are all skipped, and shard filtering exists for `lib/`-only changes. The critical path is: `setup` (40s, gates everything) → slowest test shard `test (13/16)` (1m59s) → `coverage-threshold` (29s) → `test-success`. What's left:

- **Ubuntu shard imbalance** — shards ranged 48s to 1m59s in this run. Since ubuntu runs one shard per runner, the fix is in the shard *definitions* (`tools/test/shards.ts`): split the heaviest shard(s) or rebalance by measured duration. Worth ~40s off the max.
- **The 40s `setup` job** is serial overhead before anything starts (matrix calculation + node_modules prefetch). Hard to eliminate entirely, but any trimming here is 1:1 on wall-clock for every PR run.
- **The ~30s coverage tail**`coverage-threshold` spends 11s on Node setup and 12s merging reports after the last shard. Merging `codecov` and `coverage-threshold` into one job saves runner minutes but not wall time; shrinking the merge step (or the per-job ~27s node_modules restore generally) is the only real lever, and it's marginal.

Realistic outcome: **~3m30s → ~2m45s**, almost entirely from shard rebalancing.

If you want to pursue these, the ordered list by value-for-effort is: (1) re-enable `cache-from` in bake.hcl, (2) duration-based macOS shard grouping, (3) split/rebalance the heaviest ubuntu shards. Happy to draft any of these as a PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    None yet

    Datasource

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions