Release and distribution verification

InvarLock ships six coordinated Python distributions:

Reference

Surface: Core and first-party add-in distributions, release checks, provenance, and installation verification

Stability: Coordinated version and ABI compatibility rules are public; the release workflow may evolve while preserving those checks

Use this page when: Building, publishing, installing, or independently checking an InvarLock release set

Each add-in wheel and source distribution includes the repository license text; the coordinated distribution gate checks its contents and wheel metadata.

All six use the same release version. Provider add-ins declare that exact core dependency and must also match runtime-provider ABI 1 when loaded.

Diagram

Release verification chain from a tagged commit through built artifacts, provenance, installation checks, and independently verified evidence.

What the release workflow checks

For a tagged release commit or an explicitly selected pre-tag candidate, the repository workflow:

1. binds a pre-tag validation to the workflow event commit and declared package version, or resolves the exact tag commit for a tag build or publication;
2. requires a tag build or publication event commit to equal the resolved tag commit;
3. runs the complete repository, coverage, documentation, contract, and workflow gates;
4. scans the release history range for secrets;
5. builds one wheel and source distribution for the core and every first-party add-in;
6. validates every archive against the exact checkout and runs the release preflight again from a clean detached checkout, including full replay of all seven retained public signed evidence packs and all four retained evaluator-qualification transactions through the isolated candidate-wheel CLI;
7. runs twine check on every distribution;
8. installs the core wheel alone in a clean environment outside the checkout, exercises its native quickstart, captured consumer, and offline judge evaluation/verification/report consumer, then installs the coordinated add-ins;
9. exercises the public CLI, all provider conformance commands, diagnostics, and entry-point discovery;
10. audits the installed dependency surface and generates an SBOM;
11. records all twelve archives in one SHA-256 ledger and attaches build-provenance attestations during the tag run;
12. after a complete TestPyPI or PyPI publication, verifies every hosted archive against that tag-run ledger, installs the hosted wheels together, and repeats the conformance smoke; and
13. after a verified production PyPI complete or finish run, publishes the documentation from that exact tag source to its immutable version path, latest, and stable in one serialized gh-pages commit.

These checks authenticate and exercise the package set. They do not qualify a specific model artifact, runtime image, accelerator, dataset, or evidence pack. Those belong to the evaluate/verify trust model.

Before tagging, dispatch the release workflow from the candidate branch with publish disabled, release_tag empty, and candidate_version set to the package version without a leading v. This runs the complete Linux build and release gates against the workflow event commit without creating an authoritative candidate or publishing anything. The validation job has read-only repository permissions. Attestation write access and the OIDC token used for provenance are granted only to the short tag-only job that downloads the already validated archive set, rechecks its ledger, and creates the provenance statement. Publication preparation likewise runs without an identity token. Each environment-gated publish job downloads only its two previously validated archives, rechecks the immutable tag, and invokes the pinned trusted-publishing action; it does not check out or execute candidate Python code.

For a tag build or publication, the workflow resolves and checks out the release tag's exact commit before it builds. A tag push validates and builds the authoritative candidate but does not publish it. Publication is an explicit manual action against an existing tag. The manual workflow must be dispatched with that tag as its workflow ref and the successful tag-run ID as candidate_run_id, so the event commit, resolved tag commit, workflow-run identity, and downloaded artifact agree. Manual publication does not rebuild the archives. The tagged distribution and provenance artifacts are retained for 14 days, so publication must complete within that interval.

Build provenance uses GitHub's artifact-attestation mechanism. An attestation is useful only when the release verifier validates its subject digest, signer/workflow identity, and trusted transparency or provenance policy.

Local preflight before a tag

Run the release checks from a clean checkout whose HEAD is the candidate commit. The coordinated install smoke builds the core and all five optional packages, installs their pinned base dependency closure and all six wheels in a disposable virtual environment, runs pip check, and exercises provider discovery and conformance without the checkout or user site on sys.path. The smoke selects the maintained Python 3.12 or 3.13 lock for the invoking interpreter and fails closed when no matching lock exists. The read-only preflight then inspects the core wheel and source distribution against a separately generated sha256sum-format manifest, installs the candidate wheel in isolation, exercises its runtime surface, replays all eleven retained release-evidence packs through verify and report, and reruns the public-evidence audit. Separate closed reference sets must exactly cover the seven public-evidence directories and four evaluator-qualification transaction directories, so adding or removing a carrier requires an explicit release-compatibility update. The evaluator replay must preserve the exact declared outcome: one policy pass and three integrity-valid policy rejections.

The designated Qwen3.8 reference can be exercised directly against the current source-tree verifier with make release-reference-journey. Use make release-public-evidence-compatibility for the seven public packs, make release-evaluator-qualification-compatibility for the four evaluator transactions, or make release-retained-evidence-compatibility for both closed sets. The release preflight remains authoritative for the isolated candidate-wheel path.

Run the local release gates with:

make addins-install-smoke

HASH_MANIFEST="$(mktemp)"
trap 'rm -f -- "$HASH_MANIFEST"' EXIT
(cd dist && shasum -a 256 invarlock-*.whl invarlock-*.tar.gz) \
  > "$HASH_MANIFEST"
RELEASE_SHA="$(git rev-parse HEAD)"

make release-preflight RELEASE_PREFLIGHT_ARGS="\
  --release-sha ${RELEASE_SHA} \
  --expected-version X.Y.Z \
  --dist-dir dist \
  --hash-manifest "$HASH_MANIFEST" \
  --json"

Use sha256sum instead of shasum -a 256 where that is the platform tool. Keep the temporary manifest outside the checkout: preflight rejects untracked files as well as modified files. The manifest must contain only the two core artifacts by base name. X.Y.Z is the candidate version without a leading v; preflight rejects a dirty checkout, a different HEAD, unexpected artifacts, metadata/content mismatch, or a hash change. Each retained replay supplies a fresh verifier key and an external, digest-pinned policy and trust profile. It requires the expected artifact, runtime, schedule, request, and evidence-signer anchors, a freshly signed receipt, and byte-identical repeated HTML reports. The machine-readable result records every pack-manifest digest plus its fresh receipt and deterministic report digests. It does not rewrite checked-in evidence or receipts, rerun model or evaluator inference or conversion, or generalize any retained transaction to other artifacts or policies. A passing result is release-candidate compatibility evidence, not recipient authorization to deploy. Run the non-publishing branch workflow after these local checks to exercise the same release surface on the hosted Linux runner before creating a release tag.

Local preflight validates the core wheel/sdist pair and all five add-in pairs against the checkout, including versions, source contents and licenses. Its JSON result includes first_party_addins. The external hash manifest and isolated execution/replay consumer cover the core pair and core wheel respectively; make dist-check also validates the archives against the checkout. Archive validation alone does not exercise every add-in. Keep make addins-install-smoke and the release workflow's coordinated six-package installation checks alongside preflight.

The captured core-wheel consumer checks the exact three-command root before add-ins, all three synthetic starters, signed trust-v2 handoffs and receipt-v3 authentication, unsigned rejection, adverse policy gates, repeated destinations, unchanged evidence bytes, and report-v2 requested_outputs/written_outputs. The judge core-wheel consumer imports the committed bounded fixture, publishes signed evidence, computes independent recipient pins before publication, and replays verification and all three report formats. It preserves the fixture's insufficient_evidence decision and checks wrong signer, plan, subject, and unsigned evidence rejection. It runs before any add-in install and requires Inspect and OpenAI SDK modules to be absent. Provider SDK collection is qualified separately by make inspect-judge-sdk-test; offline replay needs only core.

The three-scorer consumer additionally exercises exact-match, normalized-NLL and judge selection through captured v2 requests and public SDK capture/import helpers. Its copied fixture inventory includes collection.json; tests exercise that exact inventory outside the checkout. The NLL contract fixtures are synthetic. The separate Harness likelihood reference retains real CPU measurements and an installed signed journey for one pinned same-model profile. Its file hashes, executed capture source, regenerated anchors, receipt and model identity in reports are replayed by the example tests.

Retained native release consumers request HTML and Markdown and validate report v2 with kind: runtime; ordinary native default/HTML-only callers keep v1 JSON. Validation preserves historical native receipts, upstream fixture bytes, and recorded K2 qualification status. Optional runtime tests are not substitutes for observed inference.

For released-wheel examples, follow Matching wheels and examples: select the tag archive from importlib.metadata.version("invarlock"). Local wheels require their exact build checkout's examples, with no mutable-branch fallback when a matching release archive is unavailable.

Test index and production publication

TestPyPI is an optional publication and installation smoke, not the source of the production candidate. To use it, dispatch the release workflow from the release tag, select testpypi, and provide the successful tag workflow run's numeric ID as candidate_run_id. The workflow authenticates that the supplied run is a successful tag-push execution of the release workflow at the exact tag and commit. It then downloads that run's immutable distribution artifact, checks the closed twelve-file set and its ledger, publishes through the six project-scoped TestPyPI identities, verifies all hosted archives, installs the six hosted wheels together, and reruns the CLI, diagnostics, provider-conformance, and entry-point smoke.

Production uses the same tagged candidate directly. Once local preflight, tag-to-commit checks, release notes, security review, provenance, and the tag workflow are complete, dispatch the workflow again from the release tag, select pypi, and provide the same tag workflow run ID as candidate_run_id. Use the default complete publication phase for an ordinary release. The production jobs consume the exact archives built by the tag run; they do not rebuild them or depend on TestPyPI state. A stale, incomplete, or filename-colliding TestPyPI project therefore cannot silently select or alter a production candidate. Before upload, each publication job downloads and checks any already hosted files for its exact version against the candidate ledger. Absent files may be uploaded and ledger-identical files may be skipped, making an interrupted multi-project publication safe to resume. Any conflicting filename, metadata digest, or downloaded bytes fail before upload, and the post-publication verifier cannot turn a mixed or partially replaced release into a successful run.

PyPI permits only three pending trusted publishers at once. When a coordinated release needs to create several new add-in projects, first check which projects already have established publishers. The production bootstrap phase publishes the core plus diagnostics, Inspect judge, GGUF, and vision-text distributions. It requires at most three of those project publishers to be pending; provision existing projects first when needed. After those pending publishers become ordinary project publishers, register the TensorRT-LLM publisher and dispatch the finish phase with the same release tag and candidate_run_id. The finish run first confirms that all ten bootstrap archives are hosted byte-for-byte from that ledger, publishes only TensorRT-LLM, then verifies all twelve hosted archives and installs all six wheels together. The two special phases are rejected for TestPyPI; later releases use complete and publish all six projects in one dispatch.

Configure a protected v* tag ruleset that blocks updates and deletion. Protect each project-scoped PyPI environment with an appropriate release authorization and deployment policy. These repository controls provide the authorization layer around the workflow's commit, tag-run artifact, ledger, and trusted-publisher identity.

After production publication, the workflow downloads all twelve archives from PyPI, compares their hashes with the tag-run ledger, installs the six hosted wheels together in a clean environment, and repeats the conformance smoke. Only after that smoke succeeds does the production workflow invoke the reusable documentation publisher. The publisher removes the leading v from the validated release tag, builds from the caller's exact tag commit, and updates the versioned path plus latest while making stable redirect to the immutable version. TestPyPI and bootstrap publication cannot update Pages, and one global documentation concurrency group prevents release and branch publishers from racing their gh-pages pushes. Reconcile the published filenames, version, source tag, provenance subjects, and release assets before announcing completion.

Installation and provenance checks

Before installing a release in a controlled environment:

1. select one exact version for core and any first-party add-ins;
2. obtain hashes from a trusted release record or package index response;
3. download artifacts without installing them;
4. verify every downloaded SHA-256 digest;
5. install with hash enforcement where the package-management workflow supports it; and
6. run invarlock --version and the conformance command for each installed runtime add-in.

Example discovery checks after installation:

invarlock --version
invarlock-gguf-conformance
invarlock-tensorrt-llm-conformance

Each conformance command must report ok: true, its expected provider name, and the ABI accepted by the installed core. A conformance pass verifies the install surface and lightweight provider contract, not a native runtime model run. Before qualification fan-out, produce and strictly verify one signed canary through the exact digest-pinned runtime image. Retain its evidence, signed receipt, original verifier-owned trust profile and referenced verifier private key for the maintained readiness and evidence targets. Reuse requires matching image, providers, task, acceptance binding and CPU/CUDA device class; see canary compatibility. A canary does not establish model-specific load, memory, backend, or execution success.

An example hash-enforced download/install flow is:

python -m pip download --only-binary=:all: --dest wheelhouse \
  'invarlock==X.Y.Z' \
  'invarlock-diagnostics==X.Y.Z' \
  'invarlock-runtime-hf-vision-text==X.Y.Z' \
  'invarlock-runtime-gguf==X.Y.Z' \
  'invarlock-runtime-tensorrt-llm==X.Y.Z' \
  'invarlock-inspect-judge==X.Y.Z'

# Populate requirements.lock with the independently verified hashes, then:
python -m pip install --require-hashes -r requirements.lock

X.Y.Z and the lockfile are placeholders. Never generate the lockfile from unverified local downloads and then treat the same downloads as independently verified.

Published artifact checklist

PyPI's index responses can supply hosted distribution digests, but the trust decision still belongs to the installer's package and provenance policy.

Offline review bundle

The repository can package already collected distributions, per-artifact Sigstore sidecars, the GitHub provenance bundle, and the CycloneDX SBOM for an offline release verifier:

scripts/release/make_offline_bundle.sh \
  --version X.Y.Z \
  --tag vX.Y.Z \
  --repo OWNER/REPO \
  --certificate-identity "https://github.com/OWNER/REPO/.github/workflows/SIGNING_WORKFLOW.yml@refs/tags/vX.Y.Z" \
  --dist-dir release-material/dist \
  --sbom release-material/sbom.json \
  --provenance-dir release-material/provenance \
  --output-dir release-material/offline

This script assembles existing material; it does not fetch or manufacture provenance. Supply the independently approved certificate identity of the actual signing workflow; a repository OIDC subject is not its certificate identity. The assembler inventories nested directories, including dist/addins. Every distribution must have its own adjacent Sigstore sidecar; unlisted files and symbolic links are rejected. When the release SHA256SUMS ledger is present, its entries must match every distribution path and digest; the manifest retains the ledger as a supporting file. Inspect the generated release_manifest.json, verify each file digest, then follow the bundle's README.txt identity and issuer checks. The current GitHub workflow uploads a build-provenance bundle but does not automatically create this offline archive, so maintainers must deliberately collect compatible sidecars and run the assembler.

Stop, recover, and supersede

If any pre-publication gate fails, stop publication, preserve the failing artifacts and logs privately, fix the source or workflow, and cut a fresh candidate. Never replace an artifact while retaining its filename, tag, or version.

If a defect is discovered after publication:

1. stop recommending and promoting the affected coordinated version;
2. assess whether all six distributions or only selected files require an index yank, and record the user-visible reason;
3. preserve the tag, provenance, hashes, and incident record needed to explain existing installations;
4. fix forward under a new version rather than overwriting published files;
5. rerun the complete local, provenance, production and any selected TestPyPI checks; and
6. reconcile documentation and public-evidence links to the replacement.

Yanking is a discovery warning, not remote uninstallation or revocation of bytes already downloaded. Security-sensitive compromise also requires the key, runtime, or dependency response described in the security documentation.

Keep the signature domains separate

Three signature or provenance domains can appear in a deployment:

• package build provenance associates distributions with a release build;
• the evidence signature authenticates one evidence manifest; and
• the verifier signature authenticates one receipt and its independent anchors.

None substitutes for another. Package provenance does not approve evaluation results, an evidence signer does not choose verifier trust anchors, and a verification receipt does not attest how a Python wheel was built.

Version compatibility

Do not infer compatibility from package names alone. Confirm all of:

• coordinated first-party package versions;
• the add-in's declared core version range;
• exact runtime-provider ABI equality;
• conformance-command success; and
• for native providers, compatibility of the authenticated artifact, pinned runtime image, runner/backend, device, and compute capability.

Evidence format versions are independent of package versions. A package update may preserve an existing format exactly; a breaking artifact interpretation requires a new format identifier and explicit reader support.

Reproducibility boundaries

The workflow proves that its uploaded distributions came from one selected release commit under the recorded build workflow. It does not claim bit-for-bit reproducibility across arbitrary builders. Operators needing that stronger property must independently rebuild the same source, control the full build environment, and compare each wheel and source-distribution digest.

Likewise, installing a verified wheel does not validate a runtime image. OCI image digests, provider/backend identities, model artifacts, schedules, and evidence signatures remain separate dependency chains checked by evaluation and verification.

Related documentation

• Runtime providers defines provider ABI compatibility and first-party add-in conformance.
• Public contracts separates evidence-format versioning from package releases.
• Architecture distinguishes distribution provenance from evidence creation and independent technical verification.
• Environment variables lists the runtime-image and native resource inputs checked after installation.