Getting started

InvarLock evaluates one pinned baseline and one pinned subject on the same deterministically prepared schedule, applies a release-regression policy to the selected paired interval, publishes signed evidence, and lets a separate verifier replay the result.

For a first useful result without a model or container engine, run the five-minute signed-evidence check. It installs the core wheel, verifies retained evidence against independent anchors, issues a new verifier-signed receipt, and renders HTML on a regular CPU. Continue here when you are ready to produce evidence from your own model artifacts.

invarlock evaluate request.yaml
invarlock verify evidence/ ... --receipt verification.receipt.json
invarlock report evidence/

User guide

Outcome: Prepare and run a real paired Hugging Face comparison through the OCI-backed evaluate -> verify -> report path.

Audience: First-time operators, release decision owners, and engineers integrating a model build with an evidence gate.

Prerequisites: Python 3.12 or newer; Docker or Podman; a digest-addressed local InvarLock runtime image; two local SafeTensors snapshots; a pinned JSONL evaluation source; and separate Ed25519 evidence-signer and verifier keys.

To check results from an existing evaluator without preparing an OCI runtime, start with Captured results. Native requests support exact_match, normalized_nll_per_utf8_byte, and judge; this tutorial uses normalized NLL. For rubric-based grading of native answers, use the judge request workflow. The same commands and SDK support multiple metrics and slices with explicit captured assurance. This page walks through native execution and its independent receipt. For periodic or incident-triggered hosted evaluation, follow Requalify a hosted service; the capture harness supplies fresh executions and the core verifies their retained evidence.

Matching wheels and examples

For a released wheel, install invarlock and obtain its version with importlib.metadata.version("invarlock"). The root README and quickstart download the GitHub tag archive v${INVARLOCK_VERSION} and extract invarlock-${INVARLOCK_VERSION}/examples/quickstart and invarlock-${INVARLOCK_VERSION}/examples/acceptance-handoff/golden. Stop if that matching archive is unavailable; never substitute main or another version.

For a locally built wheel, use examples from the exact checkout used for the build. A package version does not identify a local commit or uncommitted source. Copy examples/quickstart/run.py and examples/acceptance-handoff/golden to an empty directory outside the checkout, clear PYTHONPATH, and run with the interpreter that installed the wheel. make quickstart-wheel-smoke automates the isolated local-build checks, including captured evaluation/recipient commands before optional add-ins are installed. Never regenerate retained evidence to make a recipe pass.

Diagram

The getting-started journey closes a request, runs the pair, publishes signed evidence, verifies it independently, renders a report, and retains the evidence and receipt without rewriting prior bytes.

What each command owns

Run mode is the primary path. Import mode is available when another controlled execution already produced the complete provider sidecars; the final section uses the repository fixture to exercise that secondary path without model inference.

1. Install the host CLI

python -m pip install "invarlock==0.16.0"
invarlock --version
invarlock evaluate --help

The core includes the HF provider. Model execution dependencies are supplied by the runtime image, including the source-derived hardened Accelerate wheel. For host-side model preparation from a source checkout, use the repository runtime setup.

Build or obtain the runtime image from the same release you intend to run. It must be available locally under a digest-bearing reference or exact image ID. The host launcher uses --pull=never, so evaluation never substitutes an image download for caller-controlled preparation.

For a source checkout, choose the image that matches the worker device:

First create and authenticate the exact Git source archive using the runtime-image procedure. Pass its commit, path, and digest to the build target shown below.

Scroll horizontally to see every column.

The CUDA device flag only exposes a GPU to a CUDA-capable image. It does not add CUDA libraries to the CPU image.

For a complete journey that compares a pinned Qwen3.5-0.8B checkpoint with an explicit behavioral derivative and derives all verifier inputs independently, run the checked-in Hugging Face example. The remaining sections explain the same transaction field by field for a real release workspace.

Record the resulting sha256:... identity. A local image ID authenticates local bytes but is not a fetchable reference for another operator; use a pinned registry-manifest reference when the evidence must be reproduced elsewhere.

2. Prepare the request workspace

Use a new directory for one comparison:

release-check/├── request.yaml├── artifacts/│   ├── baseline/                 # local HF SafeTensors snapshot│   └── subject/                  # candidate snapshot├── inputs/│   └── release-regression.jsonl└── policy/    └── acceptance.json

The evidence destination declared in request.yaml is created beneath this directory by the host transaction. Keep the evidence-signing private key outside the tree. It remains host-only and is never mounted into a model worker.

Pin the evaluation source

Each selected JSONL line must be a JSON object. For example:

{"case_id":"release/001","prompt":"Return the capital of France.","expected":"Paris"}
{"case_id":"release/002","prompt":"Complete: 2 + 2 =","expected":"4"}

Hash the exact file bytes:

python -c 'import hashlib, pathlib; print(hashlib.sha256(pathlib.Path("release-check/inputs/release-regression.jsonl").read_bytes()).hexdigest())'

Do not sort, serialize again, or edit the file after recording the digest. Run mode preserves source order, maps the declared top-level fields, and constructs canonical invarlock/runtime-behavioral-schedule-v1 bytes. If id_field is omitted, IDs are deterministically generated as record/00000000, record/00000001, and so on. Stable source IDs are preferable for review.

Pin both HF snapshots

Derive checkpoint_tree_sha256 and tokenizer_metadata_sha256 from each exact local snapshot with the public helpers shown in Runtime providers. The tree digest rejects links, special files, and changes during traversal. The tokenizer digest binds the tokenizer contract used to interpret target text.

Normalized NLL remains comparable across authenticated tokenizer contracts because the target loss is normalized by expected-output UTF-8 bytes. When the two tokenizer digests match and each pair also has the same positive target token count, the report can add a verifier-derived token-weighted perplexity ratio as a likelihood interpretation.

Write a one-metric policy

This example permits a normalized expected-continuation NLL ratio up to 1.05:

{"resolved_policy":{"metrics":{"normalized_nll_per_utf8_byte":{"ratio_max":1.05}}}}

The verdict uses the interval upper bound, not only the point ratio. See Schedule and policy before selecting a production threshold.

3. Write the run request

Replace every illustrative digest with the value derived from the exact input. Digest fields inside HF provider settings use the exact form returned by their public helper; runtime-image identities always use the sha256: prefix.

format_version: invarlock/evaluation-request-v1
comparison:
  baseline:
    artifact:
      path: artifacts/baseline
      model_id: acme/baseline
      locator: hf://acme/baseline@0123456789abcdef0123456789abcdef01234567
    runtime:
      provider: hf_transformers
      settings:
        batch_size: 1
        checkpoint_tree_sha256: "1111111111111111111111111111111111111111111111111111111111111111"
        context_length: 2048
        immutable_revision: 0123456789abcdef0123456789abcdef01234567
        max_output_tokens: 64
        offline: true
        seed: 7
        timeout_seconds: 300
        tokenizer_metadata_sha256: "3333333333333333333333333333333333333333333333333333333333333333"
  subject:
    artifact:
      path: artifacts/subject
      model_id: acme/subject
      locator: hf://acme/subject@fedcba9876543210fedcba9876543210fedcba98
    runtime:
      provider: hf_transformers
      settings:
        batch_size: 1
        checkpoint_tree_sha256: "2222222222222222222222222222222222222222222222222222222222222222"
        context_length: 2048
        immutable_revision: fedcba9876543210fedcba9876543210fedcba98
        max_output_tokens: 64
        offline: true
        seed: 7
        timeout_seconds: 300
        tokenizer_metadata_sha256: "3333333333333333333333333333333333333333333333333333333333333333"
  dataset:
    path: inputs/release-regression.jsonl
    sha256: "4444444444444444444444444444444444444444444444444444444444444444"
    format: jsonl
    name: release-regression
    split: validation
    input_field: prompt
    expected_output_field: expected
    id_field: case_id
  policy: policy/acceptance.json
  task: text_causal
  metric: normalized_nll_per_utf8_byte
execution:
  mode: run
output:
  evidence: evidence/release-001

The request contains comparison intent, not host authorization. It cannot select an OCI engine, grant network access, name arbitrary executables, or embed private-key bytes.

4. Evaluate through OCI

From the directory containing release-check/, run:

invarlock evaluate release-check/request.yaml \
  --signing-key evidence-signer.pem \
  --baseline-runtime-image registry.example/invarlock-runtime-cuda@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa \
  --baseline-runtime-image-digest sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa \
  --subject-runtime-image registry.example/invarlock-runtime-cuda@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd \
  --subject-runtime-image-digest sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd \
  --container-engine docker \
  --baseline-runtime-device cuda:0 \
  --subject-runtime-device cuda:1

evaluate performs preflight before it starts either worker. Append --preflight --json when you want to stop at that boundary and inspect the result. It validates the request, referenced inputs, schedule, policy, runtime capabilities, signing key, output destination, OCI engine, and both local pinned images without starting a container or creating output. Rerun the same command without --preflight only when you deliberately want to continue into the evaluation.

For repeated run requests, put the image, device and resource settings in an explicit runtime profile, then use:

invarlock evaluate release-check/request.yaml \
  --signing-key evidence-signer.pem \
  --runtime-profile runtime.json --preflight

Create runtime.json using that reference and your actual pinned images before running this command. Preflight text output shows the effective settings and where they came from. Explicit CLI overrides take precedence over profile values. The profile does not contain signing keys or acceptance policy and is not accepted for import requests. Remove --preflight to execute the same resolved setup.

Use --container-engine podman when appropriate. Device values are cpu, cuda, or cuda:<index>. Shared --runtime-image, --runtime-image-digest, --runtime-device, and --runtime-entrypoint options provide defaults for both sides. Per-side options override those defaults. Workers that share a generic or identical CUDA device run sequentially; explicitly different CUDA indexes can run in parallel.

When invoked from the host, evaluate authenticates the source and prepares the canonical schedule, then launches a constrained worker for each side. Each worker receives only its artifact and support resources read-only, an isolated writable output directory, a read-only container root, and the selected CPU/CUDA resources. Networking is disabled and capabilities are dropped. The host validates both outputs, atomically publishes release-check/evidence/release-001, and signs the bundle with its host-only key.

A successful publication exits 0, prints Evidence created, and shows the recorded policy result and evidence directory. A policy failure can still be published successfully; use independent verify to gate acceptance. The bundle contains the prepared canonical schedule, both provider observations, paired records, the comparison report, runtime bindings, checksums, manifest, and evidence signature.

5. Verify with independent anchors

The verifier must obtain these values through its own trusted channel:

• the exact policy file;
• expected baseline and subject artifact-identity digests;
• the expected canonical schedule digest;
• expected baseline and subject runtime-image digests;
• expected evidence-signer fingerprint;
• expected normalized-request digest when either side uses llama_cpp;
• its verifier identity and private key.

invarlock verify release-check/evidence/release-001/ \
  --policy release-check/policy/acceptance.json \
  --expected-baseline-artifact sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc \
  --expected-subject-artifact sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee \
  --expected-schedule sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
  --expected-baseline-runtime sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa \
  --expected-subject-runtime sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd \
  --expected-signer sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb \
  --receipt release-check/verification.receipt.json \
  --verifier-signing-key verifier.pem \
  --verifier-identity release-verifier \
  --json

Acceptance requires exit status 0, ok: true, integrity_ok: true, a passing policy verdict, the expected signer authenticity, and a valid signed receipt. An integrity-valid policy rejection may still produce a signed rejection receipt and then exit nonzero.

6. Render the report

invarlock report release-check/evidence/release-001/
invarlock report release-check/evidence/release-001/ \
  --html release-check/evidence.html \
  --explain

The report leads with Policy satisfied or Policy not met and identifies the baseline and subject. It shows both side means, the point comparison, the selected paired interval, and every configured metric, count, precision and side-accuracy check. HTML details expand to show exact values and bindings. report checks the embedded evidence signature and bundle integrity before rendering, but it does not use the independent verifier anchors or replace the signed receipt. Successful rendering exits 0 even when the recorded policy failed. Use --json for a machine-readable rendering status and HTML path; do not parse the terminal layout to decide acceptance.

Interpret the result

Native exact-match and normalized-NLL invarlock/comparison-report-v3 reports record:

• a point estimate over all authenticated records;
• a paired Newcombe 95% interval plus regression/improvement counts and an exact McNemar probability for exact match, or the fixed 2,048-replicate paired_percentile_bootstrap_sha256_v1 interval for normalized NLL;
• the selected policy limit; and
• optional record-count and interval-width qualification when the policy binds those coupled requirements;
• optional baseline and subject accuracy qualification for exact match when the policy binds minimum_side_accuracy; and
• a verdict controlled by the conservative interval bound and any configured sample and side-accuracy qualification.

For exact match, the lower bound must meet or exceed the percentage-point floor. For byte-normalized NLL, the upper bound must be at or below the ratio ceiling. Its interval resamples paired schedule positions, so each baseline observation stays coupled to its subject observation.

Preflight can establish that the authenticated schedule meets a configured minimum count, but interval width remains pending until execution. Strict verification also preserves v2 reports without side-accuracy qualification and v1 evidence with its original exact-match method.

Normalized NLL is teacher-forced expected-continuation likelihood regression, not a general model-quality measure. A displayed token-weighted perplexity ratio is derived interpretation only; it has no policy, interval, or verdict authority.

This describes sensitivity to paired resampling of the fixed schedule. It is not a population confidence interval, statistical-power calculation, safety claim, or proof that the schedule represents production traffic.

Secondary offline import smoke

The repository's examples/ directory contains complete synthetic provider sidecars. It makes no network calls and performs no model inference, but exercises the same evidence, signature, interval, verifier, receipt, and rendering contracts:

cd examples
python generate_keys.py --output-dir .keys
invarlock evaluate request.yaml --signing-key .keys/evidence-signer.pem

EVIDENCE_SIGNER_FINGERPRINT="$(tr -d '\n' < .keys/evidence-signer.fingerprint)"
BASELINE_ARTIFACT_DIGEST="$(python -c 'import json; print(json.load(open("trusted-inputs/input-digests.json"))["baseline_artifact"])')"
SUBJECT_ARTIFACT_DIGEST="$(python -c 'import json; print(json.load(open("trusted-inputs/input-digests.json"))["subject_artifact"])')"
SCHEDULE_DIGEST="$(python -c 'import json; print(json.load(open("trusted-inputs/input-digests.json"))["canonical_schedule"])')"
invarlock verify artifacts/evidence/ \
  --policy policy/acceptance.json \
  --expected-baseline-artifact "$BASELINE_ARTIFACT_DIGEST" \
  --expected-subject-artifact "$SUBJECT_ARTIFACT_DIGEST" \
  --expected-schedule "$SCHEDULE_DIGEST" \
  --expected-baseline-runtime sha256:1111111111111111111111111111111111111111111111111111111111111111 \
  --expected-subject-runtime sha256:2222222222222222222222222222222222222222222222222222222222222222 \
  --expected-signer "$EVIDENCE_SIGNER_FINGERPRINT" \
  --receipt verification.receipt.json \
  --verifier-signing-key .keys/verifier.pem \
  --verifier-identity local-example-verifier

invarlock report artifacts/evidence/ --html evidence.html --explain

Use this fixture to test key handling and artifact transport, not as evidence that a runtime-backed model comparison works.

Apply the transaction to a release change

The model-change workflow guide maps fine-tuned, pruned, quantized, GGUF, TensorRT-LLM, multimodal, harness, endpoint, and evidence handoff inputs to the same transaction. The runnable examples execute the maintained Hugging Face, PEFT, TorchAO, GGUF/llama.cpp, LM Evaluation Harness, and TensorRT-LLM journeys. A separate evidence-handoff fixture exercises acceptance, policy rejection, and tamper rejection without a model runtime. Artifact creation remains in the system that owns it.

Recovery and next steps

Evidence, receipt, and HTML destinations are no-clobber. Do not repair a published bundle in place. Retain a failed attempt when review policy requires it, fix the source boundary, and use a new output name.

Continue with:

• Evaluation request for every request field;
• Schedule and policy for source preparation, thresholds, and interval semantics;
• Runtime providers for artifact identity and provider qualification;
• Evidence and verification for transfer and independent review; and
• Troubleshooting for fail-closed recovery.