Verification failure lab

This lab turns policy and trust failures into observable command behavior. It uses the checked-in offline example, so it exercises real signing, bundle integrity, policy replay, receipt signing, and report authentication without downloading a model.

User guide

Outcome: Reproduce an authentic report-local policy rejection plus wrong-anchor, tamper, extra-file, and receipt-authorization failures.

Audience: Verifier operators, CI authors, and maintainers validating fail-closed behavior.

Prerequisites: A source checkout, the core package installed, a shell with mktemp, and permission to create a disposable working directory.

Diagram

A valid control is compared with a real policy regression, changed authenticated bytes, and wrong external anchors to distinguish signed rejection, integrity rejection, and authority rejection.

Create a disposable valid transaction

WORK_DIR="$(mktemp -d)"
cp -R examples "$WORK_DIR/examples"
cd "$WORK_DIR/examples"

python generate_keys.py --output-dir .keys
invarlock evaluate request.yaml --signing-key .keys/evidence-signer.pem

EVIDENCE_SIGNER_FINGERPRINT="$(tr -d '\n' < .keys/evidence-signer.fingerprint)"
BASELINE_ARTIFACT="$(python -c 'import json; print(json.load(open("trusted-inputs/input-digests.json"))["baseline_artifact"])')"
SUBJECT_ARTIFACT="$(python -c 'import json; print(json.load(open("trusted-inputs/input-digests.json"))["subject_artifact"])')"
SCHEDULE="$(python -c 'import json; print(json.load(open("trusted-inputs/input-digests.json"))["canonical_schedule"])')"
BASELINE_RUNTIME='sha256:1111111111111111111111111111111111111111111111111111111111111111'
SUBJECT_RUNTIME='sha256:2222222222222222222222222222222222222222222222222222222222222222'

invarlock verify artifacts/evidence \
  --policy policy/acceptance.json \
  --expected-baseline-artifact "$BASELINE_ARTIFACT" \
  --expected-subject-artifact "$SUBJECT_ARTIFACT" \
  --expected-schedule "$SCHEDULE" \
  --expected-baseline-runtime "$BASELINE_RUNTIME" \
  --expected-subject-runtime "$SUBJECT_RUNTIME" \
  --expected-signer "$EVIDENCE_SIGNER_FINGERPRINT" \
  --receipt accepted.receipt.json \
  --verifier-signing-key .keys/verifier.pem \
  --verifier-identity failure-lab-verifier \
  --json

The control verification exits 0, reports ok: true, and writes a signed acceptance receipt outside artifacts/evidence. Keep this original directory unchanged; each failure starts from a separate copy.

Authentic policy rejection

The second checked-in request uses the same schedule, policy, identities, and runtimes. Its subject answers one of 50 records incorrectly: baseline accuracy is 100%, candidate accuracy is 98%, and the point change is −2 percentage points. The paired interval lower bound is approximately −10.4954 percentage points, below the approved minimum of −10. The minimum count of 50 and maximum interval width of 20 percentage points both pass; the conservative change bound fails. Produce its immutable evidence, then verify it under the same anchors:

invarlock evaluate rejected-request.yaml --signing-key .keys/evidence-signer.pem

if invarlock verify artifacts/rejected-evidence \
  --policy policy/acceptance.json \
  --expected-baseline-artifact "$BASELINE_ARTIFACT" \
  --expected-subject-artifact "$SUBJECT_ARTIFACT" \
  --expected-schedule "$SCHEDULE" \
  --expected-baseline-runtime "$BASELINE_RUNTIME" \
  --expected-subject-runtime "$SUBJECT_RUNTIME" \
  --expected-signer "$EVIDENCE_SIGNER_FINGERPRINT" \
  --receipt policy-rejected.receipt.json \
  --verifier-signing-key .keys/verifier.pem \
  --verifier-identity failure-lab-verifier \
  --json; then
  echo 'unexpected policy acceptance' >&2
  exit 1
fi

Expected result: the evidence remains authentic and integrity-valid, the replayed policy verdict is fail, verification exits 7, and policy-rejected.receipt.json is a verifier-signed rejection. This is a valid measurement outcome rather than an infrastructure or cryptographic failure. Evaluation itself exits 0 because evidence publication succeeded. Its terminal summary shows the recorded policy failure separately from publication; recipient verification has not yet occurred at that point.

Wrong evidence signer anchor

WRONG_SIGNER='sha256:0000000000000000000000000000000000000000000000000000000000000000'

if invarlock verify artifacts/evidence \
  --policy policy/acceptance.json \
  --expected-baseline-artifact "$BASELINE_ARTIFACT" \
  --expected-subject-artifact "$SUBJECT_ARTIFACT" \
  --expected-schedule "$SCHEDULE" \
  --expected-baseline-runtime "$BASELINE_RUNTIME" \
  --expected-subject-runtime "$SUBJECT_RUNTIME" \
  --expected-signer "$WRONG_SIGNER" \
  --receipt wrong-signer.receipt.json \
  --verifier-signing-key .keys/verifier.pem \
  --verifier-identity failure-lab-verifier \
  --json; then
  echo 'unexpected acceptance' >&2
  exit 1
fi

Expected result: nonzero verification, an explicit signer mismatch, and a signed rejection receipt when the transaction reached receipt issuance. The receipt's existence is not acceptance; its signed verdict is rejection.

Changed authenticated bytes

cp -R artifacts/evidence tampered-evidence
chmod u+w tampered-evidence/reports/evaluation.report.json
printf '\n' >> tampered-evidence/reports/evaluation.report.json

if invarlock report tampered-evidence; then
  echo 'unexpected report success' >&2
  exit 1
fi

Expected result: reporting rejects before rendering because the report digest no longer matches the closed signed inventory. Do not repair the manifest or checksums to follow the changed report; that would create a different unsigned artifact.

Extra file in the closed bundle

cp -R artifacts/evidence extra-file-evidence
chmod u+w extra-file-evidence
touch extra-file-evidence/unexpected.txt

if invarlock report extra-file-evidence; then
  echo 'unexpected report success' >&2
  exit 1
fi

Expected result: closed-inventory rejection. Receipts and HTML belong beside, not inside, the evidence directory.

Wrong runtime anchor

WRONG_RUNTIME='sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff'

if invarlock verify artifacts/evidence \
  --policy policy/acceptance.json \
  --expected-baseline-artifact "$BASELINE_ARTIFACT" \
  --expected-subject-artifact "$SUBJECT_ARTIFACT" \
  --expected-schedule "$SCHEDULE" \
  --expected-baseline-runtime "$WRONG_RUNTIME" \
  --expected-subject-runtime "$SUBJECT_RUNTIME" \
  --expected-signer "$EVIDENCE_SIGNER_FINGERPRINT" \
  --receipt wrong-runtime.receipt.json \
  --verifier-signing-key .keys/verifier.pem \
  --verifier-identity failure-lab-verifier \
  --json; then
  echo 'unexpected acceptance' >&2
  exit 1
fi

Expected result: integrity can remain true while external runtime acceptance fails. This distinction is why the verifier receives runtime anchors instead of copying them from the pack.

Unauthorized receipt signer

The accepted receipt embeds a verifier public key, but that key does not self-authorize. Verify the receipt through the Python facade with a deliberately wrong expected verifier fingerprint:

import json
from pathlib import Path

from invarlock.engine import verify_signed_verification_receipt

anchors = json.loads(Path("trusted-inputs/input-digests.json").read_text())
result = verify_signed_verification_receipt(
    Path("accepted.receipt.json"),
    Path("artifacts/evidence"),
    policy_path=Path("policy/acceptance.json"),
    expected_artifact_digests={
        "baseline": anchors["baseline_artifact"],
        "subject": anchors["subject_artifact"],
    },
    expected_schedule_digest=anchors["canonical_schedule"],
    expected_runtime_digests={
        "baseline": "sha256:" + "1" * 64,
        "subject": "sha256:" + "2" * 64,
    },
    expected_pack_signer_fingerprint=Path(
        ".keys/evidence-signer.fingerprint"
    ).read_text(encoding="utf-8").strip(),
    expected_verifier_identity="failure-lab-verifier",
    expected_verifier_fingerprint="sha256:" + "0" * 64,
)
assert not result.ok
assert "receipt verifier key does not match caller expectation" in result.errors

Expected result: the receipt signature may be cryptographically valid while authorization fails. Obtain the real expected verifier fingerprint through a separate trust registry before relying on a receipt.

Interpret the outcomes

Delete the disposable WORK_DIR after review. Never run tamper exercises on a published pack or overwrite a signed receipt that has been distributed.

Continue with Evidence and verification for the full trust model and Troubleshooting for production recovery.