See the report your team can review.

Choose a change like yours. Read its findings, inspect the available records, then follow the checks for that example.

These are separate technical comparisons, not customer results or a model ranking. Reading a report displays retained results; it does not run verification in your browser.

Choose an example

Each example has its own tests, requirements and evidence boundaries.

Replace a coding model selected

Public study extracts

Which coding tasks improved, and which regressed?

GPT-6 Sol → GPT-6.1 Sol

200 complete pairs · 208 attempted tasks

15 gains and 7 losses in the complete pairs.

Scope to keep in view

Public counts can be reproduced. Those checks do not authenticate the full signed archive or its private source records. Resource findings use shared successes only.

What you can inspect

A study report, per-task public extracts, retained summaries and a checking script.

Read report Check the public counts Explore this example

Preparing your own comparison? Open the setup guide.

Need a guided handoff? Discuss your pilot.

A completed coding-model study

GPT-6 Sol → GPT-6.1 Sol, tested on the same Python coding tasks at High reasoning effort. The report identifies individual gains and regressions, examines repeat attempts and compares recorded time and token use.

Public study · 200 paired tasks

What changes when you switch coding models?

GPT-6 Sol → GPT-6.1 Sol · Python tasks · High reasoning effort

Outcomes for 200 complete task pairs:

Each square is one task. Select a group to trace it.

200 complete task pairs. The overall accuracy difference remained uncertain. 8 of 208 attempted tasks lacked complete paired results.

Open the study report →
Public evidence and study scope

The public checker reproduces published counts. The full signed archive is not public. This study used specialized tooling; applying it to your tasks requires its own scope.

Resources · 136 tasks both models passed

Fewer tokens
~62%
Less recorded time
~27%

Adjusted for task type, on shared successes only. Token reductions are not measured billing savings.

Read the findings →Check the public counts →

Three comparisons with signed evidence

Reports, signed records and replay instructions for three comparisons, including one that failed its requirement.

Model-change report previewRecorded results

Proposed change

Can we use the smaller model within our allowed quality loss?

Ministral 3 8B · BF16 → Q5_K_M GGUF

What was tested: 400 paired MMLU-Pro records

Meets the agreed regression threshold

The measured accuracy is slightly higher, but the interval includes zero. This supports the allowed-loss check, not a claim of improved accuracy.

Results and regressions

Exact-match accuracy

Baseline
44.75%
Candidate
45.25%

+0.50 percentage points

95% interval: −1.12 to +2.12 percentage points.

Agreed checks for this example: Allowed loss: 2 percentage points. Lower effect bound: −1.12 points. The other recorded checks also pass.

Open this comparison’s report

This is a materialized GGUF comparison on a fixed public benchmark. It does not establish performance on customer traffic or preservation of every model capability.

Review files, verification and technical details
What the reviewer receives
The HTML report, paired answer records, model and runtime identifiers, agreed checks, signed evidence and a verification receipt.
What they can check independently
Check that the evidence matches the approved model, runtime and test identities; authenticate the signature; and recalculate scores, uncertainty and the result. These checks do not establish performance on production traffic.

The lower bound clears the −2 percentage-point requirement. The record-count, interval-width and minimum-accuracy checks also pass.

At least 400 pairs, a lower effect bound of −2 percentage points, an interval no wider than 10 percentage points, and at least 20% accuracy on both sides. These are this example’s requirements, not recommended defaults.

The source pack retains artifact and runtime identities, the paired schedule, the policy and a signed verification receipt. Rendering the report does not issue a new receipt.

Report renderer: v0.17.0. Replay checked with v0.17.0. The original capture identities and signed outcomes are preserved.

Inspect this comparison’s source packReplay this comparison
About these reports, versions and metrics

These three reports use the v0.17.0 renderer. Their original measurements, signed evidence and verification receipts are unchanged. Replay with v0.17.0 reproduces the recorded results, including rejections, without new model calls. Any new verification receipts are written separately from the original evidence.

Each example uses its own metric and requirements. Read the units, uncertainty and acceptance criteria together. The examples are not a ranking, and their thresholds are not recommended defaults for your workload. Opening an HTML report does not independently verify its source pack.

Four parts of a model-change review

Recorded result
The report records the change, measurements, uncertainty and result against chosen requirements. The evidence package retains detailed results and setup.
Authored explanation
The narrative explains findings, failed cases and limitations. It is separate from the recorded result and technical verification receipt.
Verification result
Verification checks signatures, approved inputs and supported calculations. The signed receipt records the outcome. Missing records, unexpected inputs or invalid signatures can prevent verification.
Reviewer’s decision
A recorded pass is not release approval. The reviewer assesses whether the data, tests and requirements fit the intended use, then records acceptance separately from the report and verification receipt.

Limits. A pass applies only to the recorded comparison and chosen policy. It does not prove representative sampling, honest execution, broad model quality, safety or compliance. Read the decision semantics.

See the report and receipt formats →
Separate trust inputs → VerificationThe reviewer obtains approved reference records, requirements and signer details independently of the submitted package. Read the trust model.

The coding study’s public checker reproduces published counts; its complete signed archive is private. The other comparisons provide public reference files and replay instructions below. Opening a report alone performs neither check.

Independently check the result you selected

To repeat these checks, use Python 3.12 or newer and the v0.17.0 core, with the pinned source files linked below. Obtain and approve the reference pins independently of any submitted evidence. These examples demonstrate reference replay, not your organization’s acceptance. No model or provider calls are needed.

Show the reference setup commands

Native comparisons require artifact and runtime identities; captured comparisons require complete-run and request pins; judging requires its independently approved plan and measurements.

python -m venv .venv
. .venv/bin/activate
python -m pip install "invarlock==0.17.0"
git clone --depth 1 --branch v0.17.0 https://github.com/invarlock/invarlock.git invarlock-references
cd invarlock-references
git rev-parse HEAD
# Expected: 1fbe9a45d353fb92bf35e9b2c467ef2d54bff8c9

Start in an empty directory with Git installed. The setup downloads the reference checkout; replay runs locally. Keep the reference checkout unchanged and follow the selected guide below.

Quantize a model: Ministral 3 8B

400 paired MMLU-Pro records. Expected recorded result: pass.

Replay the Ministral comparison with its approved reference anchors

After the setup above, run this from the reference checkout. The helper uses the separately pinned anchor and policy files, creates a temporary recipient key, verifies the retained pack and writes a new receipt and reports to the sibling ministral-recipient-output directory. Expect ok: true and policy_verdict: pass. Use a new output directory for each run.

python - <<'PYTHON'
import json, runpy, shutil
from pathlib import Path

root = Path.cwd()
helper = runpy.run_path(str(root / "scripts/release/release_reference_journey.py"))
result = helper["run_release_reference_journey"](
    repo_root=root,
    command=(shutil.which("invarlock"),),
    workspace=root.parent / "ministral-recipient-output",
    allow_checkout_source=False,
    reference_config=Path("scripts/release/reference_evidence/ministral3-8b-bf16-to-q5-k-m-gguf-anchors.json"),
)
print(json.dumps(result, indent=2))
PYTHON
Open the Ministral 3 8B replay guide

Use evaluator records: Mistral 7B

400 paired reference continuations · LM Evaluation Harness. Expected recorded result: regression.

Open the Mistral 7B replay guide
Notes for each replay

For Ministral, use the linked pack with the native verification command and independently approved policy, artifact, runtime, schedule, request and signer expectations. For Mistral, follow the reference’s recipient reconstruction steps; verification exits 7 for the expected rejection, then reporting succeeds. The extraction helper authenticates the held-out archive and replays both its QA and extraction studies. Its expanded-size allowance is 216,183,325 bytes. The frozen-answer companion supplies model and prompt context separately from those signed results. Its task excerpts retain their source attribution and CC BY-SA 4.0 terms.

Explore other retained handoffs

These additional references have separate tasks and policies. Both were replayed with the v0.17.0 core without models, containers or provider calls. Successful replay preserves their rejections; it does not authorize a deployment.

  • ModelKit delivery: Mistral 7B BF16 to Q4. A 128-case comparison retains a policy rejection. Offline replay checks signed evidence and the historical handoff; it does not repeat delivery of model files or model execution.
  • Evaluator callbacks and controlled HTTP capture. Inspect, Harness, Promptfoo and Langfuse retain 32 exact-match and likelihood comparisons, all with policy rejections. The HTTP example uses a locally controlled task service, not the hosted Langfuse product or an arbitrary cloud endpoint.
Replay these additional references

After the replay setup above, run these commands from the reference checkout. They use the installed core package and retained records. Expect successful integrity checks and the original policy rejections.

python -I examples/integrations/modelkit-handoff/references/mistral-7b/replay.py
python -I examples/integrations/evaluator-live/references/priority-workflows/replay.py
Separate tutorial · CPU only

Try a separate five-minute CPU tutorial

Verify a separate 50-record demonstration fixture and create a new receipt. Expected result: Decision: pass. It does not replay any of the comparisons above, run model inference or test your candidate.

The tagged guide has the commands, prerequisites and output files. It uses example trust anchors; obtain production anchors independently.

Open the tagged quickstart source and instructions →

Use your own evaluations

To compare your own model change, start with representative cases, meaningful scoring and agreed acceptance requirements.

Existing evaluation records

Compare complete captured records using exact match, normalized likelihood or bounded judging. Verification replays the supplied evidence without attesting its original execution.

Choose a captured-results workflow

Run a new model comparison

Evaluate a baseline and candidate through a supported runtime and metric in your environment.

Check runtime and metric support
Try captured records without a model, then prepare a request

Try captured records without a model

With the v0.17.1 core installed, run these commands in an empty directory. They compare synthetic extraction records and write an unsigned local report. This demonstrates the workflow; it does not measure a real model or issue an independently authenticated receipt.

invarlock evaluate --init release-check --example extraction
invarlock evaluate release-check/request.yaml --unsigned
invarlock report release-check/artifacts/evidence --html report.html

To send evidence to a reviewer, follow the signed captured handoff. For rubric-based scoring, use the bounded judge workflow. Its retained calls can be replayed offline; rating correctness and general model quality require separate evidence.

Prepare a request and check the setup

The getting-started guide covers the versioned request, policies, signing keys and exact input identities. For native execution, append --preflight --json to the complete evaluation command to check inputs and local runtime images before starting workers. Preflight does not run an evaluation or establish model quality.

Follow the setup and preflight guide →

Complete your own reviewer handoff

Bring your current evaluation process, one current or recently completed change and the reviewer who needs to assess it. Agree a fixed quote for one or two changes after sample review.

Discuss your pilot