A Passing Release Test Can Contain Policy Rejections

A release preflight can pass only when expected policy rejections stay rejected. In the retained example, the candidate CLI replays eleven saved reference cases. A retained compatibility reference is a signed transaction plus its independently pinned inputs and expected verdict.

Eight references must remain policy passes. Three must remain integrity-valid policy rejections. Release compatibility means reproducing those exact outcomes and their verification checks; it does not mean making every inner policy pass.

This is a release-only check, not part of an ordinary evaluate, verify, or report command. Preflight creates an isolated environment, installs the candidate core wheel once, and uses that installed CLI for all eleven references.

One rejection, two results

For an expected rejection, verification returns command status 7, integrity_ok: true, policy_verdict: "fail", and inner ok: false. Once preflight confirms that complete expected result, the surrounding compatibility record returns outer ok: true.

The tagged reference records the expectation directly:

"expected": {
  "policy_verdict": "fail",
  "verification_scope": "paired_comparison"
}

The runner derives status 7 from the expected fail verdict. The two ok values answer different questions:

• Inner ok: false: this transaction does not pass its comparison policy.
• Outer ok: true: the candidate reproduced the transaction's expected behavior.

Three results answer three different questions

Scroll horizontally to see every column.

Integrity covers signatures, digests, cross-bindings, retained records, and semantic replay. It asks whether the verifier could trust and reconstruct the evidence.

The technical policy verdict is computed only after replay succeeds. A rejection is therefore not corrupt evidence or a broken verifier. It is an integrity-valid transaction whose result falls outside its declared comparison policy.

Release compatibility asks a third question. It compares the candidate wheel's behavior with the outcomes fixed by the tagged reference set.

The mix is a compatibility fixture

The v0.15.0 compatibility snapshot contains two closed inventories:

• Seven retained public evidence references, all expected to pass policy.
• Four retained evaluator transactions, one expected to pass and three expected to fail policy.

Together they produce eight expected passes and three expected rejections.

The release note inventories this 7+4 set. This article focuses on why the three rejections are required compatibility outcomes rather than failed fixtures.

Diagram

The compatibility runner replays seven public evidence references and four evaluator references, preserving each expected pass or rejection before returning one aggregate result.

Preserve every expected outcome

1 · Pin the references

Seven public packs and four evaluator transactions: eight expected passes, three expected rejections.

2 · Replay each one

An expected rejection must return status 7, valid integrity, policy fail, and inner ok: false. A matching replay returns outer ok: true.

3 · Check the complete set

All eleven must match. A changed verdict, invalid receipt, missing reference, or unequal report bytes blocks the candidate.

The aggregate passes when the expected behavior is preserved. The detailed diagram includes the full per-reference checks.

Each reference pins the pack manifest, comparison policy, schedule, request, and evidence signer. It also pins the baseline, subject, and runtime identities. The runner requires those configurations to cover the retained public-pack and evaluator-transaction directories exactly. A missing, extra, or duplicated reference blocks the candidate.

The three rejections act as negative controls for policy semantics. They show whether a candidate still recognizes a valid transaction that crosses its declared boundary.

A corpus containing only expected passes could detect a candidate becoming too restrictive. It would be weaker at detecting the opposite change: a candidate becoming unexpectedly permissive.

A flip in either direction blocks the candidate

Each retained reference configuration fixes the expected policy verdict. The runner derives the required command status from it: 0 for pass and 7 for fail.

A flip from pass to rejection, or from rejection to pass, blocks the candidate. With the evidence, policy, and trust anchors held fixed, either change is semantic drift.

Integrity drift also blocks the candidate. Every reference must remain structurally valid, match its authenticated pins, replay completely, and produce no verification warnings or errors.

One preflight install must stand on its own

This proves that the packaged wheel can verify the retained cases without help from the source checkout.

• Preflight installs the candidate core wheel once in an isolated environment and confirms that imports resolve from it.
• The installed CLI runs root, evaluate, verify, and report smoke checks before processing all eleven references.
• Each replay uses independent policy and trust inputs, signs a fresh receipt, and renders two byte-identical HTML reports.

The private verifier key is then removed. The checked-in evidence and historical receipts remain unchanged.

This tests the candidate wheel's ability to read, verify, and render retained transactions with stable semantics. It is more than a package-import smoke, but it is still a compatibility replay.

What the preflight does not recompute

The eleven-reference replay does not rerun model inference, evaluator execution, or artifact conversion. It starts from retained evidence records and tests the candidate verifier and reporter against them.

As a result, it can expose drift in evidence interpretation, anchor checking, policy replay, receipt construction, exit behavior, and deterministic report rendering. It cannot expose a change that occurs only while producing new model outputs.

Limitations

• The check covers the eleven references in the retained compatibility set; behavior absent from that closed set needs its own fixture.
• The isolated candidate-wheel check covers the core distribution surface. Coordinated add-in archives, hosted packages, provenance, and installation conformance are exercised by separate release gates.
• Deterministic rendering is checked by two byte-identical reports produced during each replay; this is not a claim of arbitrary cross-platform reproducibility.

Sources

Website documentation explains the maintained workflow. The eleven-reference inventory and compatibility behavior described here use InvarLock v0.15.0; the tagged implementation and tests preserve that evidence basis.

• Release verification: local preflight before a tag
• Reports and receipts: verification result
• InvarLock 0.15.0: Replayable Deployment Evidence and Bounded Decisions
• Eleven-reference inventory and replay implementation
• Per-reference result validation
• Candidate-wheel preflight integration
• Tagged release-compatibility tests