Public evidence

See what the evidence actually says.

See how a reviewer can check a model comparison and understand why it passed or failed the chosen requirements. Explore three recorded examples. Each has its own report, source evidence and replay instructions.

These are separate technical comparisons, not customer results or a model ranking. This page displays retained results; it does not run verification in your browser. The separate CPU tutorial replays existing evidence and does not run model inference or test your candidate.

Explore the comparisons ↓Try the separate CPU tutorial

Inspect the artifacts

Start with the change you need to review.

Explore a release decisionRecorded results

Can we use the smaller model within our allowed quality loss?

Ministral 3 8B · BF16 → Q5_K_M GGUF

400 paired MMLU-Pro records

Policy satisfied

The lower bound clears the −2 percentage-point requirement. The record-count, interval-width and minimum-accuracy checks also pass.

Exact-match accuracy

Baseline
44.75%
Candidate
45.25%

+0.50 percentage points

95% interval: −1.12 to +2.12 percentage points.

Open this comparison’s report

This is a materialized GGUF comparison on a fixed public benchmark. It does not establish performance on customer traffic or preservation of every model capability.

At least 400 pairs, a lower effect bound of −2 percentage points, an interval no wider than 10 percentage points, and at least 20% accuracy on both sides. These are this example’s requirements, not recommended defaults.

The source pack retains artifact and runtime identities, the paired schedule, the policy and a signed verification receipt. Rendering the report does not issue a new receipt.

Sources and report renderer pinned to v0.16.0. The original capture identities and signed outcomes are preserved.

Inspect this comparison’s source packReplay this comparison

A policy result and a release decision are different.

The Ministral comparison and held-out extraction study pass their declared policies. The Mistral likelihood comparison retains a verified rejection. Successful replay does not turn a rejection into a pass. Missing records, mismatched identities or invalid signatures can instead prevent a verified result.

Each example uses its own metric and requirements. Read the units, uncertainty and policy together. The examples are not a ranking, and their thresholds are not recommended defaults for your workload. Opening an HTML report does not independently verify its source pack.

A pass applies to the recorded comparison and chosen policy. It does not prove representative sampling, honest execution, broad model quality, safety or compliance, and it does not authorize deployment. Your reviewer still decides whether the data, scoring and requirements are appropriate for the intended use. Read the trust model and decision semantics.

Replay the result you selected

Use Python 3.12 or newer and the released v0.16.0 core, with the pinned source files linked below. Obtain and approve the reference pins independently of any submitted evidence. These examples demonstrate reference replay, not your organization’s acceptance. No model or provider calls are needed.

python -m venv .venv
. .venv/bin/activate
python -m pip install "invarlock==0.16.0"
git clone --depth 1 --branch v0.16.0 https://github.com/invarlock/invarlock.git invarlock-references
cd invarlock-references
git rev-parse HEAD
# Expected: bc83b9fa11bb8f75188c6f1e126a190cfd376c69

Start in an empty directory with Git installed. The setup downloads the reference checkout; replay runs locally. Keep the reference checkout unchanged and follow the selected guide below.

Quantize a model: Ministral 3 8B

400 paired MMLU-Pro records. Expected recorded result: pass.

Replay the Ministral comparison with its approved reference anchors

After the setup above, run this from the reference checkout. The helper uses the separately pinned anchor and policy files, creates a temporary recipient key, verifies the retained pack and writes a new receipt and reports to the sibling ministral-recipient-output directory. Expect ok: true and policy_verdict: pass. Use a new output directory for each run.

python - <<'PYTHON'
import json, runpy, shutil
from pathlib import Path

root = Path.cwd()
helper = runpy.run_path(str(root / "scripts/release/release_reference_journey.py"))
result = helper["run_release_reference_journey"](
    repo_root=root,
    command=(shutil.which("invarlock"),),
    workspace=root.parent / "ministral-recipient-output",
    allow_checkout_source=False,
    reference_config=Path("scripts/release/reference_evidence/ministral3-8b-bf16-to-q5-k-m-gguf-anchors.json"),
)
print(json.dumps(result, indent=2))
PYTHON
Open the Ministral 3 8B replay guide

Use evaluator records: Mistral 7B

400 paired reference continuations · LM Evaluation Harness. Expected recorded result: regression.

Open the Mistral 7B replay guide

For Ministral, use the linked pack with the native verification command and independently approved policy, artifact, runtime, schedule, request and signer expectations. For Mistral, follow the reference’s recipient reconstruction steps; verification exits 7 for the expected rejection, then reporting succeeds. The extraction helper authenticates the held-out archive and replays both its QA and extraction studies. Its expanded-size allowance is 216,183,325 bytes. The frozen-answer companion supplies model and prompt context separately from those signed results. Its task excerpts retain their source attribution and CC BY-SA 4.0 terms.

Try a separate five-minute CPU tutorial

This CPU quickstart uses a separate 50-record demonstration fixture with a recorded pass. It verifies signed evidence and creates a new receipt; it does not replay any of the comparisons above. Selecting a rejection there does not change this fixture or its expected result.

Use Python 3.12 or newer, curl and tar in a POSIX shell, from a new empty directory. Internet access is needed to install the pinned package and download the tagged files. The replay itself needs no model, GPU, container engine or networked service.

Tagged evidence replay commands

python -m venv .venv
. .venv/bin/activate
python -m pip install "invarlock==0.16.0"
curl -fsSLO https://github.com/invarlock/invarlock/archive/refs/tags/v0.16.0.tar.gz
tar -xzf v0.16.0.tar.gz --strip-components=3 \
  invarlock-0.16.0/examples/quickstart \
  invarlock-0.16.0/examples/acceptance-handoff/golden
python run.py --fixture golden

Expect Decision: pass and three outputs in invarlock-quickstart-output/: verification.result.json, verification.receipt.json and evidence.html. The script creates a one-use demo verifier key and removes it after receipt issuance. Bundled example anchors demonstrate the mechanics; your recipient must obtain production anchors independently.

Open invarlock-quickstart-output/evidence.html in your browser to inspect the report. Find it inside the directory where you ran the commands.

Read the tagged quickstart source and instructions

Use your own evaluations

The demo teaches verification. To compare your own model change, start with representative cases, meaningful scoring and agreed acceptance requirements. The released workflow supports two input paths; check which fits before preparing the full request.

Existing evaluation records

Compare complete captured records using exact match, normalized likelihood or bounded judging. Exact match needs answers and references; likelihood needs measured reference log probabilities; judging needs frozen answers, a rubric and retained judge calls. Aggregate scores cannot supply missing facts. Verification replays the supplied evidence without attesting its original execution.

Choose a captured-results workflow

Run a new model comparison

Evaluate a baseline and candidate through a supported runtime and metric in your environment. This requires the model artifacts, evaluation data and configured runtime images, plus the compute to generate and score outputs.

Check runtime and metric support

Try captured records without a model

With the v0.16.0 core installed, run these commands in an empty directory. They compare synthetic extraction records and write an unsigned local report. This demonstrates the workflow; it does not measure a real model or issue an independently authenticated receipt.

invarlock evaluate --init release-check --example extraction
invarlock evaluate release-check/request.yaml --unsigned
invarlock report release-check/artifacts/evidence --html report.html

To send evidence to a reviewer, follow the signed captured handoff. For rubric-based scoring, use the bounded judge workflow. Its retained calls can be replayed offline; rating correctness and general model quality require separate evidence.

Prepare a request and check the setup

The getting-started guide covers the versioned request, policies, signing keys and exact input identities. For native execution, append --preflight --json to the complete evaluation command to check inputs and local runtime images before starting workers. Preflight does not run an evaluation or establish model quality.

Follow the setup and preflight guide →

Give the reviewer an independent check

Your reviewer supplies approved inputs, policy and signer expectations through a channel independent of the submitted evidence. Native comparisons require artifact and runtime identities; captured comparisons require complete-run and request pins; judging requires its independently approved plan and measurements. They verify the evidence and retain their signed receipt separately. This records the verification outcome; release approval remains their decision.

See how recipient verification works

Apply this to a recurring release review

Bring your current evaluation process, an upcoming model change and the reviewer who needs to assess it. The assisted pilot connects one workflow across two real releases and measures the effort to prepare, verify and review the evidence.

Scope a release-evidence pilot