Evaluation lifecycle and failure boundaries

The evaluate, verify, and report transactions have deliberately separate read and write boundaries. The stages below describe native exact-match/NLL and deterministic-extension run/import evidence. Native judge retains its runtime capture but uses the judge collection, publication and recipient contracts, including a private resumable workspace. Captured deterministic, likelihood and recorded-score comparisons use the same transactions with paired runs, a captured directory pack, and independent run/request pins in place of native provider, schedule, and runtime bindings.

Reference

Surface: Transaction stages, write boundaries, failure classification, and outputs

Stability: Behavioral contract for public evaluate, verify, and report transactions

Use this page when: Diagnosing where a transaction stopped, designing retries, or determining which output may exist after failure

Diagram

Transaction lifecycle states advance from draft to closed request, paired run, signed evidence, verified receipt, and release decision with fail-closed exits.

Only the publication transition creates the evidence directory. A failure at an earlier transition leaves no partial public destination.

1. Load the request

The request loader:

• accepts one JSON-compatible YAML object no larger than 1 MiB;
• rejects aliases, tags, duplicate keys, merge keys, include-like directives, non-canonical scalars, excessive depth, and excessive node count;
• anchors every file reference beneath the real request directory;
• rejects absolute, URL, traversal, backslash, symbolic-link, and unsafe path components; and
• checks provider availability and metric capability.

The transaction repeats no-follow reads later and detects inputs that change while being read. Request validation is therefore not the only filesystem check.

2. Execute or import

Run mode

Run mode requires an artifact path for each side and a digest-pinned local JSONL dataset object. InvarLock preserves source order, resolves or derives record IDs, authenticates each input, and prepares the canonical schedule. The caller supplies runtime resources separately from the request. The host CLI launches one independently pinned Docker or Podman worker for each side. Each worker receives only its read-only job, canonical schedule, artifact, and closed support resources, plus one isolated writable output directory. It emits a report, runtime manifest, artifact identity, provider receipt, scoring observation, and canonical runtime configuration. The host authenticates those six files before deriving or signing the comparison.

If either side cannot produce complete strict evidence, the overall evidence destination is not published.

For each side, provider execution follows one lifecycle:

1. validate the closed runtime spec;
2. authenticate the artifact and caller-owned resources;
3. open a session inside the strict runtime context;
4. score the canonical schedule in order;
5. obtain a provider receipt bound to the completed observation;
6. write and authenticate the side report and runtime manifest; and
7. close the session on success or failure.

Two CPU workers may run in parallel. Two workers assigned to different explicit CUDA indexes may also run in parallel. Generic CUDA selection, the same explicit CUDA index, and a CPU/CUDA pair run sequentially to avoid GPU contention. Pairing depends on the authenticated schedule and record IDs, not on worker concurrency.

Import mode

Import mode requires, for both sides, an artifact identity, provider receipt, scoring observation, runtime-side report, runtime manifest, and runtime config. It also requires the canonical schedule and paired-record file.

The engine does not trust imported aggregates. It reproduces artifact identity from the request/provider, authenticates receipt and runtime bindings, derives paired records from observations and the schedule, and requires the imported paired file to be canonical and exactly equal to that derivation.

Run and import therefore converge on the same bundle and verifier semantics.

3. Derive the comparison

The engine requires both observations to have the same ordered record IDs and input digests as the schedule. It derives built-in record scores or replays the authorized scorer, then derives means, the comparison value, paired interval, optional count/interval-width and side-accuracy qualification, and policy verdict from the relevant lower or upper bound and any configured sample controls. New transactions write invarlock/comparison-report-v3; verification preserves the report-version-specific meaning of signed v1 and v2 evidence. The comparison ID is deterministically derived from normalized intent, both artifact identities, schedule, policy, runtime digests, and paired records.

4. Publish once

Publication validates the complete candidate in a private staging directory, writes canonical payloads and checksums, signs the manifest, and atomically renames the directory into place. The destination and a sibling publication lock are no-clobber. Only a completed staged tree is renamed into place. If a publication check observes a changed path, the operation fails; completed or competing output, or private staging, may remain for inspection. Cleanup does not recursively remove a replacement at an old staging name.

Published permissions are read-only, and all later outputs remain external. See Evidence artifacts for the exact inventory.

The atomic rename protects visibility, while checksums, the signed manifest, and later snapshot verification protect integrity. Read-only permissions are defense in depth; they are not accepted as cryptographic proof.

5. Verify independently

Verification requires independent baseline and subject artifact digests, canonical schedule digest, policy, both runtime digests, evidence signer, verifier identity, and verifier-key inputs. It replays the entire comparison rather than trusting stored scores or verdicts.

A completed verification writes a no-clobber, read-only signed receipt outside the pack. This includes rejection receipts: integrity or policy failure remains a nonzero command result, but the verifier can still authenticate what it rejected and under which anchors.

Structural failure that prevents the verification transaction from reaching a completed result may prevent receipt creation. Automation must check both exit status and expected receipt presence.

Receipt and report files are written in private staging, synchronized, and published without replacing an existing destination. A synchronization or path check failure after publication can leave a completed or competing file even though the command failed. Inspect and authenticate any retained evidence, and choose a new destination for a retry; file presence alone does not establish successful verification.

Verification snapshots the submitted pack before replay so subsequent reads refer to the same regular-file inventory. It validates syntax and signatures before depending on semantic content, then reconstructs identities, pairings, scores, interval, report, and policy decision under the independent anchors.

6. Render the authenticated report

Reporting authenticates signed evidence bundle integrity and renders its closed canonical report. It can write new HTML, Markdown, and JUnit files outside the pack. Independent verification and its signed receipt remain the acceptance record. Output destinations are checked together before writing. A later write failure can leave earlier completed outputs; the v2 rendering result lists them in written_outputs.

Failure classification

Scroll horizontally to see every column.

Retries should use a new output or receipt path after correcting the underlying input. InvarLock never treats overwriting an earlier result as a retry.

Transaction outputs

For forensic retention, preserve the immutable pack and any external signed receipt together. Do not modify either to add annotations; store operator notes as separate records that identify their target by manifest or receipt digest.

Related documentation

• Architecture defines the component and trust boundaries behind each lifecycle stage.
• Command-line interface maps transaction outcomes to public exit and write behavior.
• Evidence artifacts describes the directory published at the evaluation boundary.
• Reports and receipts interprets success and rejection outputs.