fix(cve-gate): give the scan real disk, and retry only the could-not-run case #87
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/cve-gate-scan-space"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Run 234 is the first time the CVE gate has ever actually executed.
The seccomp fix from #83 works. The container started, the profile validated, grype ran. It then died on something nobody had reached before.
1. The scan had 2 GB of RAM to work in
/tmpin that container is--tmpfs …size=2g— RAM — and grype was using it for two things that don't fit:GRYPE_DB_CACHE_DIR=/tmp/grype-db/tmp/stereoscope-*ci-runner is multi-GB on its own. Spending 2 GB of node RAM per gate run was also the wrong resource on a cluster that's memory-bound.
Both paths are env-overridable, so this needs no new cve-gate release:
TMPDIR— removed by--rmwith the container.Both on the dind daemon's disk. Rootfs stays
--read-only;/tmpstays a noexec tmpfs, now 256m since nothing large lands there.What's given up: docker can't set
noexecon those two volumes. Acceptable for a static binary that execs nothing from either path — but it's a real reduction from the previous posture, so it's stated in the action rather than glossed over.2. Retry — but only on exit 2
The same run also hit Harbor mid-scan:
Harbor's health endpoint reports all eight components healthy now, so it was overloaded by concurrent CI traffic, not broken. A registry hiccup shouldn't sink an otherwise-fine release.
The exit contract makes this safe to automate:
Only 2 is safe to repeat — it means no answer was produced, and a scan is idempotent. Retrying 1 would launder a real failure into a pass, which is the one thing this gate must never do. Verified the loop runs exactly once for 0 and 1, three times for 2.
Not in scope
IMAGE_REVISION=is still empty (on_release_published.ymlsets it fromgithub.sha, which doesn't resolve in that trigger). Pre-existing, separate fix.Run 234 is the first time the gate has ever actually executed. The seccomp profile validated (31/31) and grype started. It then died on something new: write /tmp/grype-db/...: no space left on device /tmp in that container is a 2g tmpfs — RAM — and grype was using it for two things that do not fit in 2g: its vulnerability database (the image defaults GRYPE_DB_CACHE_DIR to /tmp/grype-db) and stereoscope's extraction of the layers of the image being scanned. ci-runner is multi-GB by itself. Spending 2g of node RAM per gate run was also the wrong resource on a cluster that is memory-bound. Both paths are env-overridable, so this needs no new cve-gate release: - a NAMED volume for the DB, which persists between runs, so grype stops re-downloading a multi-GB database on every release. grype verifies the DB checksum on import, so a persistent cache is not a trust hole. - an ANONYMOUS volume for scan scratch via TMPDIR, removed by --rm with the container. Both live on the dind daemon's disk. The rootfs stays --read-only and /tmp stays a noexec tmpfs, now 256m because nothing large lands there any more. The one property given up is noexec on those two volumes, which docker cannot set here — acceptable for a static binary that execs nothing from either path, and stated rather than glossed. Also: retry, but ONLY on exit 2. The exit contract is 0 = met its budget, 1 = blew its budget, 2 = could not reach a verdict. Only 2 is safe to repeat, because it means no answer was produced and a scan is idempotent. Retrying 1 would launder a real failure into a pass, which is the one thing this gate must never do. That retry earns its place: the same run also hit "502 Bad Gateway" from Harbor and a mid-stream INTERNAL_ERROR while pulling the image to scan. Harbor's own health endpoint reports all eight components healthy minutes later, so it was overloaded by concurrent CI traffic rather than broken. A registry hiccup should not sink a release that is otherwise fine.