Post-release verification

Opt-in gate that REPORTS post-publish defects — missing or corrupted assets, unlanded publishes, failed install smoke-tests, glibc-ceiling violations

The verify_release: gate runs last in the release pipeline — after the release is created and every publisher has run — and reports post-publish defects. Because it runs after the irreversible publish, it never blocks or undoes anything: a failed check reports the problem and exits non-zero so CI flags it, but the release is already published.

It is distinct from per-publisher post_publish_poll (which waits on Chocolatey/WinGet moderation queues): verify_release is a broader, top-level gate covering the GitHub release's assets and the produced Linux packages.

What it checks

Four independently-toggleable checks:

Minimal config

verify_release:
  enabled: true

With just enabled: true, the asset check and the landing checks run (they need no extra config — anodizer already knows what it produced, can fetch the release's asset list, and the run's own publish report and artifact set carry every landing coordinate). The smoke-test and libc-ceiling checks stay off until you configure them.

Full config reference

verify_release:
  enabled: true            # default false — the whole gate is opt-in
  assert_assets: true      # default true — diff produced vs. uploaded assets + size/digest
  assert_landing: true     # default true — probe cargo/npm/pypi/blob/snapcraft/docker landings
  install_smoke:           # absent => smoke-test off
    deb: { image: "debian:12" }      # default debian:stable-slim
    rpm: { image: "fedora:40" }      # default fedora:latest
    apk: { image: "alpine:3.20" }    # default alpine:latest
  glibc_ceiling: "2.36"    # absent => libc check off

(a) asset existence + content

verify_release:
  enabled: true
  assert_assets: true

anodizer diffs the artifacts it produced (the same upload set the release stage uploads from) against the assets actually stored on the published GitHub release, then reports any produced artifact with no matching uploaded asset:

$ anodizer release
...
     Warning 1 produced artifact(s) missing from the published release for crate 'myapp': myapp_1.0.0_amd64.deb
       Error verify-release: post-publish verification found 1 issue(s); the release IS published — investigate:
  - 1 produced artifact(s) missing from the published release for crate 'myapp': myapp_1.0.0_amd64.deb

Extra assets on the release (orphans from a prior re-cut) are reported as an advisory, never a failure on their own.

Size + digest verification

Every expected asset that is present also gets a byte-level check: the stored size must equal the local artifact's size, and the stored sha256 digest (GitHub computes one server-side for every uploaded asset) must equal the local sha256 — the checksum stage's already-computed hash is reused when available. A clean pass emits one result line:

     • github: crate 'myapp' 22/22 assets present, sizes+digests match

A mismatch names the asset and both values:

- asset 'myapp_1.0.0_linux_amd64.tar.gz' of crate 'myapp' size mismatch: local 4194304 B vs
  published 1048576 B — the uploaded asset does not match the produced artifact

When the release serves no digest for an asset (older GitHub Enterprise), anodizer downloads the asset and hashes it — capped at 64 MiB per asset; beyond the cap the asset is verified by size only, with a verbose notice.

Config-derived signature / SBOM expectations

The produced set alone cannot catch a sign or SBOM stage that silently produced nothing — there is no registered artifact to miss. So the check additionally derives, from your resolved config plus the artifact set, the signature / certificate / SBOM asset names that should exist:

• each signs: entry contributes one signature (and one certificate, when certificate: is set) per artifact its artifacts:/ids: filters select, named by its signature: template;
• each sboms: entry contributes its rendered documents: names per matched artifact.

No new config is required — the expectations come from what anodizer already knows. A release missing them fails with the exact names:

- crate 'myapp': 2 signature/SBOM asset(s) required by the resolved signs/sboms
  config were never uploaded (the producing stage registered no such artifact):
  myapp_1.0.0_checksums.txt.sig, myapp_1.0.0_linux_amd64.tar.gz.cdx.json

Intentional skips create no expectations: a sign config whose if: evaluated falsy or whose artifacts: none disabled it (the run's own skip record is consulted first as the authoritative account of what this run decided), an SBOM config whose skip: evaluated truthy, or a whole stage skipped via --skip=sign / --skip=sbom. SBOM documents: containing glob patterns are not predictable from config and create no expectations either. Under a release.ids upload filter, expectations follow the SUBJECT's verdict — a signature or SBOM is expected exactly when the artifact it derives from is uploaded.

(b) publisher landing checks

verify_release:
  enabled: true
  assert_landing: true   # the default

Every publisher that succeeded this run is probed to confirm the publish actually published — using the coordinates the run's own publish report recorded, so no extra config is needed:

One result line per publisher:

     • cargo: anodizer-core@0.15.4 visible on crates.io index
     • npm: myapp@0.15.4 visible on registry.npmjs.org
     • pypi: 9/9 uploaded file(s) listed on pypi.org
     • blob: 22/22 uploaded object(s) visible in s3://my-bucket
     • snapcraft: myapp 1.0.0 live in the Snap Store channel map
     • docker: 4/4 pushed image(s) visible on ghcr.io

Every line names the host it asked, and a run that spread its uploads over several hosts names each once, sorted:

     • blob: 2/2 uploaded object(s) visible in s3://my-bucket, s3://my-mirror

A single object, package or image reads in the singular, with no counter:

     • blob: v1/app.tar.gz visible in s3://my-bucket

Anything that qualifies the count arrives in ONE trailing clause — how many needed a propagation wait, and how many could not be reached at all:

     • docker: 3/4 pushed image(s) visible on ghcr.io (1/4 needed a propagation wait, 1 unverifiable)

Registry propagation

A registry that has accepted a publish does not always serve it on the next request. Every probe above therefore keeps asking — backing off from 5 seconds to a 30-second cap — before it reports an absence. npm has been observed to take ten minutes to serve a package after npm publish returned.

The window belongs to the whole sweep, not to each target: it opens once, runs for 12 minutes, and every probe of that run shares it. A release publishing 43 targets to a registry that never serves them therefore costs one 12-minute window, not 43 of them. retry.max_elapsed caps the window too, so lowering that lowers this; a dry run probes once and never waits.

Waiting out propagation is the expected case, so it prints no warning. The publisher's own result line says how much of the publish arrived late:

     • npm: 9/9 published package(s) visible on registry.npmjs.org (6/9 needed a propagation wait)

Run with -v to see each individual re-ask.

A failure re-asking cannot resolve — a rejected credential, a bucket store that could not be built — ends that target immediately instead of holding the shared window open on a fixed answer.

Docker files no publish report, so its targets come from the artifacts: an image artifact records that its push returned, which is what also carries the pushed set through a --publish-only run rehydrated from the preserved artifacts.json. A snapshot, a dry run and a skip_push: true manifest push nothing and are never probed, and a run that deselects docker (--skip=docker, or a --publishers list without it) probes nothing either — even when the manifest it rehydrated carries the markers of the leg that did push.

Having no publish report, docker has no required: flag either, so its findings are routed by what they prove. An absent tag and a digest mismatch are definitive and fail the gate. A registry that could not be consulted is a recorded warning that never fails the release: the probe reads only the plain auths entries in config.json and never runs a credsStore / credHelpers helper binary, so a repository whose credential lives in a helper answers 401 to a push that succeeded. The probe also asks over TLS everywhere but loopback (localhost, 127.0.0.0/8, [::1]), so a plain-HTTP registry reached through docker's insecure-registries answers as unverifiable rather than as a failure.

verify_release:
  enabled: true
  assert_landing: true

crates:
  - name: myapp
    dockers_v2:
      - images: ["ghcr.io/owner/myapp"]
        tags: ["{{ .Version }}", "latest"]
    docker_manifests:
      - name_template: "ghcr.io/owner/myapp:{{ .Version }}"
        image_templates:
          - "ghcr.io/owner/myapp:{{ .Version }}-amd64"
          - "ghcr.io/owner/myapp:{{ .Version }}-arm64"
        skip_push: false      # `true` builds the list locally and is not probed

A publisher that was skipped, deselected, or failed is not probed — it published nothing this run. A probe that cannot run (index unreachable, store build failure) is reported as an issue once the window closes, never silently passed: an unverifiable landing is a finding.

- cargo: myapp@1.0.0 reported published but is not visible on the crates.io index
- pypi: myapp-1.0.0-py3-none-win_amd64.whl reported uploaded but is not listed on pypi.org
- blob: s3://my-bucket/v1.0.0/myapp.tar.gz reported uploaded but is missing from the bucket
- snapcraft: myapp 1.0.0 was HELD for Snap Store manual review and is not live in the store — consumers get nothing until review approves (https://dashboard.snapcraft.io/snaps/myapp/)
- docker: ghcr.io/owner/myapp:1.0.0 reported pushed but is not in ghcr.io
- docker: ghcr.io/owner/myapp:latest was pushed as sha256:9f2c… but ghcr.io serves sha256:04ab…

A docker landing the probe could not confirm is recorded as a warning instead:

! unverifiable docker landing not gating the release — docker: could not confirm ghcr.io/owner/myapp:1.0.0 on ghcr.io: registry returned 401 Unauthorized

(c) install smoke-test

verify_release:
  enabled: true
  install_smoke:
    deb: { image: "debian:12" }
    rpm: { image: "fedora:40" }
    apk: { image: "alpine:3.20" }

For each produced Linux package, anodizer runs the install + a version check in a fresh pinned container, with the container platform pinned to the package's build architecture:

docker run --rm --platform linux/arm64 \
  --mount type=bind,source=<pkg>,destination=/pkg/<pkg>,readonly debian:12 \
  sh -c "dpkg -i '/pkg/<pkg>' || (apt-get update && apt-get -y -f install) && 'myapp' --version"

Per-package install commands: .deb → dpkg -i (with an apt-get -f dependency fixup), .rpm → rpm -i --nodeps, .apk → apk add --allow-untrusted. Each image defaults to a sane base (debian:stable-slim, fedora:latest, alpine:latest); override only the ones you care about.

The --platform pin comes from the package's build target — an arm64 .deb is installed in an arm64 container, never the runner's native one. Running a non-native platform needs cross-arch emulation (qemu/binfmt) on the Docker host; anodizer probes for it once per platform and, when it is missing, fails that package's smoke-test loudly instead of reporting a misleading in-container arch error or silently skipping the coverage:

- crate 'myapp': install smoke-test failed for myapp_1.0.0_linux_arm64.deb on debian:stable-slim:
  cannot run linux/arm64 containers on this linux/amd64 host: cross-arch emulation (qemu/binfmt)
  is unavailable. Install it (e.g. `docker run --privileged --rm tonistiigi/binfmt --install all`)
  or run install_smoke on a linux/arm64 runner. The package was NOT smoke-tested.

When Docker is unavailable, the smoke-test is skipped with a notice — it does not hard-fail the gate, and asset-existence and libc-ceiling still run:

     • skipped install smoke-test — Docker unavailable (asset-existence and libc-ceiling still run)

(d) libc ceiling

verify_release:
  enabled: true
  glibc_ceiling: "2.36"

anodizer reads the required glibc symbol versions from each glibc-linked .deb's embedded ELF (.gnu.version_r) and fails if the maximum required version exceeds your floor — catching the classic "built on a too-new builder, won't run on the target distro" regression:

       Error verify-release: post-publish verification found 1 issue(s); the release IS published — investigate:
  - crate 'myapp': usr/bin/myapp requires glibc 2.38, exceeding the configured ceiling 2.36

The comparison is numeric, component-wise (so 2.36 > 2.4, not the wrong lexical ordering). musl binaries have no glibc requirement and are skipped — which is the whole point: a musl build would otherwise hide a glibc-floor regression, so a ceiling-checked release should ship glibc .debs to be meaningful.

glibc_ceiling absent → the libc check is off.

Failure semantics

• A detected defect is reported with the specific artifact / package / version, and the gate exits non-zero so CI fails the job.
• The wording is always explicit: the release IS published — the gate never implies the publish failed and never attempts to undo it.
• Each check is best-effort and independent: Docker-unavailable skips only the smoke-test; the asset, landing, and libc checks need neither Docker nor extra config.
• Check axes are gated on their own publisher's selection: a --publishers npm run still verifies the npm landing while skipping the GitHub asset check; only a run that selects none of github-release, cargo, npm, blob self-skips the whole gate.

Workspaces

In workspace modes (lockstep or per-crate), the gate verifies every published crate: asset-existence across each crate's produced set, install-smoke and libc-ceiling across each crate's Linux packages. No crate is siloed.

Skipping

anodizer release --skip=verify-release