fix(docker-fast): drop comment lines from build-args, and pin the SBOM generator #43

Merged

ryangr0 merged 1 commit from fix/build-args-comments-and-sbom-generator into main

2026-08-05 11:56:23 +00:00

ryangr0 commented

2026-08-05 11:47:01 +00:00

Owner

Two problems found reading infrastructure run 234's build log. Neither is the reason that build is slow — see the bottom — but both are real.

1. Comment lines were becoming build-args 🔴

docker/build-push-action stripped # lines out of its build-args input. My CLI rewrite in #42 did not. From run 234's actual command line:

--build-arg IMAGE_REVISION=
--build-arg '# Route base images through Harbor: the Dockerfiles default these to docker.io/ghcr.io'
--build-arg '# (so .github/GHCR builds work unchanged); the in-cluster build overrides them to the'
--build-arg '# Harbor pull-through proxy projects (+ webgrip for the techdocs-builder inter-image dep).'

Harmless to the build itself — an undeclared ARG is ignored — but --provenance=mode=max records the full build definition, so that prose has been written into the signed SLSA attestation of every image this composite builds since #42 merged.

That's the opposite of what ADR-0004 asked for. The point of adding provenance was that the signature should describe how the image was built; provenance containing a Dockerfile comment mangled into an argument name is provenance nobody will trust.

Only a leading # is dropped — MSG=hello # world is still a value.

2. The SBOM generator was an unpinned Docker Hub pull 🟠

--sbom=true makes BuildKit fetch docker.io/docker/buildkit-syft-scanner at the floating stable-1 tag. Run 234:

=> docker-image://docker.io/docker/buildkit-syft-scanner:stable-1    105.7s

Every build. Straight from Docker Hub. An unpinned external dependency in the hot path of the job that produces our provenance — and it survived the sweep that moved every other image in this composite behind the Harbor proxy, because it's BuildKit's implicit default rather than something written in the file.

Now --sbom=generator=<image>, defaulting to the Harbor proxy with tag and digest, Renovate-tracked — the same treatment buildkit-image and binfmt-image already get.

What's actually making ci-runner slow

Neither of the above. The dominant cost in run 234:


`FROM harbor.webgrip.dev/ghcr/actions/actions-runner` (321 MB pull + extract)	355s
`buildkit-syft-scanner` from docker.io	105.7s
`RUN apt-get update && apt-get install …`	44s

forgejo-buildkitd's content store is new and empty. The registry cache manifest imported fine (0.1s), but the build still had to materialise the 321 MB base image because it isn't in the shared daemon's store yet.

That should be a one-time cost per node — three nodes, so up to three slow builds, then warm. It's worth confirming on the next release rather than assuming, which is why this PR doesn't claim to fix it.

One pre-existing thing this made visible, not introduced here: IMAGE_REVISION= is empty, because on_release_published.yml sets it from github.sha, which doesn't resolve in that trigger context. build-push-action was receiving the same blank value, so the OCI revision label has been empty for a while. Worth a separate fix.

Two problems found reading infrastructure run 234's build log. **Neither is the reason that build is slow** — see the bottom — but both are real. ## 1. Comment lines were becoming build-args 🔴 `docker/build-push-action` stripped `#` lines out of its build-args input. My CLI rewrite in #42 did not. From run 234's actual command line: ``` --build-arg IMAGE_REVISION= --build-arg '# Route base images through Harbor: the Dockerfiles default these to docker.io/ghcr.io' --build-arg '# (so .github/GHCR builds work unchanged); the in-cluster build overrides them to the' --build-arg '# Harbor pull-through proxy projects (+ webgrip for the techdocs-builder inter-image dep).' ``` Harmless to the build itself — an undeclared ARG is ignored — but **`--provenance=mode=max` records the full build definition**, so that prose has been written into the signed SLSA attestation of every image this composite builds since #42 merged. That's the opposite of what ADR-0004 asked for. The point of adding provenance was that the signature should describe *how* the image was built; provenance containing a Dockerfile comment mangled into an argument name is provenance nobody will trust. Only a **leading** `#` is dropped — `MSG=hello # world` is still a value. ## 2. The SBOM generator was an unpinned Docker Hub pull 🟠 `--sbom=true` makes BuildKit fetch `docker.io/docker/buildkit-syft-scanner` at the **floating `stable-1` tag**. Run 234: ``` => docker-image://docker.io/docker/buildkit-syft-scanner:stable-1 105.7s ``` Every build. Straight from Docker Hub. An unpinned external dependency in the hot path of the job that produces our provenance — and it survived the sweep that moved every other image in this composite behind the Harbor proxy, because it's BuildKit's implicit default rather than something written in the file. Now `--sbom=generator=<image>`, defaulting to the Harbor proxy with tag **and** digest, Renovate-tracked — the same treatment `buildkit-image` and `binfmt-image` already get. ## What's actually making ci-runner slow Neither of the above. The dominant cost in run 234: | | | |---|---:| | `FROM harbor.webgrip.dev/ghcr/actions/actions-runner` (321 MB pull + extract) | **355s** | | `buildkit-syft-scanner` from docker.io | 105.7s | | `RUN apt-get update && apt-get install …` | 44s | **forgejo-buildkitd's content store is new and empty.** The registry cache manifest imported fine (0.1s), but the build still had to materialise the 321 MB base image because it isn't in the shared daemon's store yet. That should be a **one-time cost per node** — three nodes, so up to three slow builds, then warm. It's worth *confirming* on the next release rather than assuming, which is why this PR doesn't claim to fix it. One pre-existing thing this made visible, not introduced here: **`IMAGE_REVISION=` is empty**, because `on_release_published.yml` sets it from `github.sha`, which doesn't resolve in that trigger context. `build-push-action` was receiving the same blank value, so the OCI revision label has been empty for a while. Worth a separate fix.

ryangr0 added 1 commit

2026-08-05 11:47:01 +00:00

fix(docker-fast): drop comment lines from build-args, and pin the SBOM generator 0f0bff3993

Two problems found reading infrastructure run 234's build log.

1. COMMENT LINES WERE BECOMING BUILD-ARGS.

docker/build-push-action stripped `#` lines out of its build-args input. The CLI rewrite that
replaced it did not, so every comment in a caller's docker-build-args block was passed as its own
--build-arg. From run 234:

--build-arg '# Route base images through Harbor: the Dockerfiles default these to docker.io/...'
--build-arg '# (so .github/GHCR builds work unchanged); the in-cluster build overrides them ...'

Harmless to the build — an undeclared ARG is ignored — but --provenance=mode=max records the full
build definition, so that prose was being written into the signed SLSA attestation of every image
this composite builds. Provenance containing a Dockerfile comment mangled into an argument name is
provenance nobody will trust. The whole point of ADR-0004 was that the signature should describe
how the image was built.

Only leading `#` is dropped. A `#` inside a value (MSG=hello # world) is still a value.

2. THE SBOM GENERATOR WAS AN UNPINNED DOCKER HUB PULL.

`--sbom=true` makes BuildKit fetch docker.io/docker/buildkit-syft-scanner at the FLOATING
`stable-1` tag. Measured in run 234: 105.7s, on every build, straight from Docker Hub — an
unpinned external dependency in the hot path of the job that produces our provenance, and one that
survived the sweep that moved every other image in this composite to the Harbor proxy.

Now `--sbom=generator=<image>`, defaulting to the Harbor proxy with tag AND digest, tracked by
Renovate — the same treatment buildkit-image and binfmt-image already get.

Neither of these is the reason ci-runner's build is slow. That is the base image: 321MB pulled and
extracted for 355s because forgejo-buildkitd's content store is new and empty. That cost is
one-time per node and should not recur once warm; it is worth confirming on the next release
rather than assuming.

ryangr0 merged commit 885624fdc4 into main

2026-08-05 11:56:23 +00:00

ryangr0 referenced this pull request from a commit

2026-08-05 11:56:23 +00:00

Merge pull request 'fix(docker-fast): drop comment lines from build-args, and pin the SBOM generator' (#43) from fix/build-args-comments-and-sbom-generator into main