A Passing Release Test Can Contain Policy Rejections

A software release check can pass even when some model comparisons fail their policies. In fact, it must preserve those expected failures. In this example, the candidate CLI rechecks eleven saved cases. Each compatibility reference includes signed evidence, independently fixed checking inputs, and the expected verdict.

Eight cases must still pass policy. Three must still produce valid evidence of a policy rejection. If the new verifier starts passing one of those three, the software has become too permissive. A test set containing only expected passes could miss that change.

Release compatibility therefore means preserving the expected outcomes and their verification checks.

This is a release-only check, not part of an ordinary evaluate, verify, or report command. Preflight creates an isolated environment, installs the candidate core wheel once, and uses that installed CLI for all eleven references.

One rejection, two results

For an expected rejection, verification returns command status 7, integrity_ok: true, policy_verdict: "fail", and inner ok: false. Once preflight confirms that complete expected result, the surrounding compatibility record returns outer ok: true.

The tagged reference records the expectation directly:

"expected": {
  "policy_verdict": "fail",
  "verification_scope": "paired_comparison"
}

The runner derives status 7 from the expected fail verdict. The two ok values answer different questions:

• Inner ok: false: this transaction does not pass its comparison policy.
• Outer ok: true: the candidate reproduced the transaction's expected behavior.

Three results answer three different questions

Integrity covers signatures, digests, cross-bindings, retained records, and semantic replay. It asks whether the verifier could trust and reconstruct the evidence.

The technical policy verdict is computed only after replay succeeds. A rejection is therefore not corrupt evidence or a broken verifier. It is an integrity-valid transaction whose result falls outside its declared comparison policy.

Release compatibility asks a third question. It compares the candidate wheel's behavior with the outcomes fixed by the tagged reference set.

The mix is a compatibility fixture

The v0.15.0 compatibility snapshot contains two closed inventories:

• Seven retained public evidence references, all expected to pass policy.
• Four retained evaluator transactions, one expected to pass and three expected to fail policy.

Together they produce eight expected passes and three expected rejections.

The release note inventories this 7+4 set. This article focuses on why the three rejections are required compatibility outcomes rather than failed fixtures.

Diagram

The compatibility runner replays seven public evidence references and four evaluator references, preserving each expected pass or rejection before returning one aggregate result.

Preserve every expected outcome

1 · Pin the references

Seven public packs and four evaluator transactions: eight expected passes, three expected rejections.

2 · Replay each one

An expected rejection must return status 7, valid integrity, policy fail, and inner ok: false. A matching replay returns outer ok: true.

3 · Check the complete set

All eleven must match. A changed verdict, invalid receipt, missing reference, or unequal report bytes blocks the candidate.

The aggregate passes when the expected behavior is preserved. The detailed diagram includes the full per-reference checks.

Each reference fixes the evidence, policy, cases, request, signer and model/runtime identities. The runner checks the complete set: a missing, extra or duplicated reference blocks the candidate. The tagged reference configurations retain the exact checking inputs.

A flip in either direction blocks the candidate

Each retained reference configuration fixes the expected policy verdict. The runner derives the required command status from it: 0 for pass and 7 for fail.

A flip from pass to rejection, or from rejection to pass, blocks the candidate. With the evidence, policy, and trust anchors held fixed, either change is semantic drift.

Integrity drift also blocks the candidate. Every reference must remain structurally valid, match its authenticated pins, replay completely, and produce no verification warnings or errors.

One preflight install must stand on its own

This proves that the packaged wheel can verify the retained cases without help from the source checkout.

• Preflight installs the candidate core wheel once in an isolated environment and confirms that imports resolve from it.
• The installed CLI runs root, evaluate, verify, and report smoke checks before processing all eleven references.
• Each replay uses independent policy and trust inputs, signs a fresh receipt, and renders two byte-identical HTML reports.

The private verifier key is then removed. The checked-in evidence and historical receipts remain unchanged.

The result shows whether the installed package can verify these historical comparisons and reproduce their reports.

What the preflight does not recompute

The eleven-reference replay does not rerun model inference, evaluator execution, or artifact conversion. It starts from retained evidence records and tests the candidate verifier and reporter against them.

As a result, it can expose drift in evidence interpretation, anchor checking, policy replay, receipt construction, exit behavior, and deterministic report rendering. It cannot expose a change that occurs only while producing new model outputs.

Limitations

• The check covers the eleven references in the retained compatibility set; behavior absent from that closed set needs its own fixture.
• The isolated candidate-wheel check covers the core distribution surface. Coordinated add-in archives, hosted packages, provenance, and installation conformance are exercised by separate release gates.
• Deterministic rendering is checked by two byte-identical reports produced during each replay; this is not a claim of arbitrary cross-platform reproducibility.