Reports and receipts

InvarLock separates provider facts, a canonical comparison report, independent verification output, a signed verification receipt, and console or HTML summaries. They have different trust meanings.

Reference

Surface: Runtime-side reports, canonical comparison reports, verification results, signed receipts, and console or HTML summaries

Stability: Versioned machine contracts are stable by format; console and HTML output are presentation surfaces

Use this page when: Parsing a decision, validating a receipt, or distinguishing signed evidence from independently accepted evidence

Report formats depend on the selected scorer and evidence family. The native runtime sections below describe exact match, normalized NLL and deterministic extensions. Captured comparisons use the multi-metric report described in evaluation records. Native and captured judge requests produce judge analysis and verification results described in judge measurements; they do not use the native invarlock/comparison-report-v3 shape. All are rendered by invarlock report.

Report appearance

HTML reports use a paper-style review memo: comparison identities and decision context, a scoped result, findings to review, limitations and the next decision, followed by detailed results and supporting evidence. A restrained light palette, readable tables and numbered results work on screen and paper. Status words accompany color so the result remains readable in monochrome.

The paper lockup retains InvarLock's bracket-and-signal geometry with a single ink stroke and live-text name. It omits the app icon's tile, translucent overlays and small center dot. This is a report-specific rendition, not a replacement for the website or application identity. The report uses no remote fonts or images.

This structure draws on the intended-use and evaluation context in Model Cards for Model Reporting and SageMaker Model Cards. Promptfoo's exports also separate stakeholder reports from full evaluation data. InvarLock applies these patterns to a specific baseline-to-subject change, with recorded requirements, supporting file locations and separate recipient verification.

Reading a report for a release review

The evidence-only report starts with Decision context, including the compared identities. When no authored review is provided, intended use, business context, decision owner and recommendation are explicitly marked as not supplied. The renderer does not infer them from technical provenance or a passing result.

A client handover supplies that context as separately attributed commentary. The shared presentation view can include an AuthoredReview: a label, author, context, recommendation, decision, rationale and next action. It appears in both HTML and Markdown without changing the technical result, evidence or receipt. Use invarlock report --review-context context.json --package handoff to supply the versioned context and generate a decision brief, full review, technical appendix and recipient guide. See client review packages. This workflow does not change signed evidence or act as an approval system.

The completed worked examples choose all those fields using clearly labeled fictional organizations and owners around real retained comparisons. Their primary reviews contain completed decisions; missing context remains appropriate for evidence-only views and missing-data test cases.

The result heading names the recorded policy result; its explanation states what led to it before the chart. A passing description identifies the satisfied interval requirements instead of repeating the heading. A failed bound can mean the observed change exceeded the allowance or that uncertainty prevented establishing the required result. Too few records, a wide interval or an unmet absolute score requirement can also prevent a pass. Judge reports distinguish adverse results from insufficient evidence; advisory results do not authorize a required gate. A multi-metric opening summarizes all scopes and explains an adverse or unresolved result without ranking unlike metrics by their numeric values. Combined evidence reports name both required component outcomes. Advisory judge reports identify their advisory role in the title and at the start of the explanation. Browser-tab titles include the recorded outcome.

Findings to review highlights failed or unresolved metric/scope results and links to their numbered checks. If more than five need review, the report says how many remain and retains every result below. It does not rank unrelated metrics by numerical severity. Scope and limitations and Next decision remain visible before the detailed statistics.

The opening also gives a Review action appropriate to the recorded result. This is generated guidance, not an owner's recommendation. The Change context preview quotes a retained comparison note and identifies any additional notes below; missing context is not counted as an observed change. Outcome counts refer to metric/scope results, not independent cases. An unavailable outcome uses a neutral treatment, and absent limitations or check details are explicitly labeled. When an authored review is supplied, its Owner recommendation, Owner decision, Review rationale and Decision follow-up replace the generic business-context placeholders and next-decision guidance. Their attribution stays visible in print.

Each metric's Supporting record gives a path relative to the evidence root and a JSON pointer after #; an empty pointer identifies the whole JSON object. Captured metric pointers retain original array positions even when the display puts failures first. Combined reports prefix the component's actual member path. These references locate the recorded result; they do not authenticate it.

What was checked states the operations performed by reporting. Some workflows authenticate retained records without replaying the comparison; see rendering and recipient acceptance. A captured report that records a pass while an available displayed requirement fails is rejected as inconsistent; this check does not replay scoring.

Then check What was compared: confirm the baseline, subject, task and recorded configuration change. The report distinguishes model-artifact identity from a hosted service's reported identity, and native execution from imported records. Use the retained fields for complete configuration comparisons.

Then read Results and requirements. Exact-match values are accuracies; NLL values are nats per UTF-8 byte and a subject-to-baseline ratio; judge values are normalized rubric scores, not percentages of correct answers. Judge case counts, repeated ratings and independent units are distinct. The interval and configured checks determine the policy result, not the point estimate alone.

Scroll horizontally to see every column.

A policy can correctly pass a comparison whose absolute quality is unsuitable for an application. Confirm that the declared requirements cover that workload, including an absolute quality minimum when needed. Record the rationale for those requirements with the release decision.

Use Evidence details for retained identities and exact values. Judge reports can show selected retained cases with report --case-id; native and captured reports do not provide a universal per-case debugging interface. Their source records remain in the evidence bundle. A useful example pairs its report with instructions for finding those records and replaying the comparison.

Metric and scope headings use readable labels for identifier-style names, such as grounded_qa-judge-quality becoming Grounded QA judge quality. The original identifiers for these formatted labels remain in Recorded metric and scope identifiers. Display formatting does not change grouping, verification inputs or existing JUnit identifiers.

Deliver a complete review package

An original technical report and its evidence remain unchanged when shared. A client review package also includes the interpretation, intended use, change rationale, known limitations and the owner's recommendation or an explicit statement that these were not supplied. Label authored commentary separately from signed evidence, and keep the independent verification receipt and any organizational approval as distinct records. A rendered report alone does not contain these additional decisions.

Keep the explanation in the downloaded and printed package, not only on the surrounding web page. This renderer's HTML and Markdown preserve the review context; print styles expose collapsed explanations and all metric tabs. Open raw evidence details before printing to include them in the printout. Original evidence files, hashes and signatures are unaffected by rendering. Generating a new presentation does not replace a historical canonical download.

Print layouts place the result and review action together, then let assurance checks flow below. Running page numbers help keep longer copies in order. Identity labels stay with their values, and keyboard focus outlines are omitted from paper. These print features are exercised in Chromium on A4 and US Letter; other print engines may handle pagination differently.

The accessibility checks draw on W3C guidance for reflow, contrast, text spacing and keyboard tabs. They exercise narrow layouts, explicit status words, local navigation and text equivalents for interval graphics. They do not establish full accessibility conformance or substitute for assistive-technology testing with users.

Where the interval comes from

How this interval was calculated explains the retained method and sample structure beside the chart. The marker is the observed comparison; the bar spans the recorded interval. A lower-bound requirement tests the left endpoint against the minimum, while an upper-bound requirement tests the right endpoint against the maximum. Other configured checks, such as completeness, sample count and precision, still apply.

• Exact match: Baseline and subject answer the same cases. The Newcombe hybrid score method uses paired successes and failures for a nominal 95% confidence interval on their accuracy difference. Native reports also show the four paired outcome counts. A change of 2 percentage points means, for example, 70% to 72% accuracy; it is not a 2% relative change. The confidence level is not an answer-correctness rate or the fraction of model runs that exceed the policy limit. The observed change can be within the allowance while its interval fails the bound requirement; this does not establish a loss larger than that allowance.
• NLL and other resampled scores: The declared percentile method resamples paired records together within the fixed schedule. Native reports retain 2,048 resamples; captured reports show their recorded count. The interval describes the central recorded proportion of the resampling distribution for the mean ratio or score change. It is not a population confidence interval.
• Judge scores: Repeated ratings are averaged within cases, then cases within declared independent units; units have equal weight. Two-sided Hoeffding bounds use the bounded rating scale and independent-unit count. The report shows the actual alpha error budget and its allocation across the declared comparison family. More repetitions do not add independent units. Changes use normalized rubric score points, not accuracy percentage points.

These methods do not establish that the selected cases represent production traffic. A worse observed score describes the measured run. A policy rejection alone does not establish worse performance beyond those cases; an interval may simply fail to rule out a decline larger than the allowed margin. Reporting explains the recorded decision without recalculating it from rounded chart values.

Diagram

Report lineage separates runtime observations, the canonical comparison report, verifier-owned results, signed receipts, and human renderings.

Runtime-side report

Each side has an invarlock/runtime-side-report-v1 object:

{
  "format": "invarlock/runtime-side-report-v1",
  "provider": "hf_transformers",
  "artifact_identity_sha256": "...",
  "scoring_observation_sha256": "...",
  "schedule_sha256": "...",
  "record_count": 20
}

It binds one side to its artifact identity, scoring observation, provider, and schedule. Detailed settings and runtime identity remain in the provider receipt, observation, artifact identity, and runtime manifest. The verifier reconstructs the runtime-side object exactly.

Canonical comparison report

Native exact-match/NLL and deterministic-extension evaluations write reports/evaluation.report.json as an invarlock/comparison-report-v3 object. Strict verification also accepts signed invarlock/comparison-report-v2 and invarlock/comparison-report-v1 objects. Version 2 omits side-accuracy qualification; version 1 additionally uses the legacy exact-match interval method. A verifier never silently upgrades the arithmetic or policy meaning of an existing pack.

The current report contains:

• comparison_id, metric, and record_count;
• baseline and subject mean_score;
• one metric-specific comparison object;
• one metric-specific uncertainty object;
• paired_binary for exact match, derived_measurements for normalized NLL, or scorer_extension and scorer_replay for an authorized extension;
• policy_digest;
• optional sample_qualification when the policy binds count and precision requirements;
• optional side_accuracy when exact-match policy binds a minimum accuracy for both sides; and
• verdict, either pass or fail.

A complete exact-match report has this shape:

{
  "baseline": {"mean_score": 0.75},
  "comparison": {
    "kind": "exact_match_delta_pp",
    "minimum": -15.0,
    "value": 5.0
  },
  "comparison_id": "cmp-example",
  "format": "invarlock/comparison-report-v3",
  "metric": "exact_match",
  "policy_digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
  "record_count": 20,
  "subject": {"mean_score": 0.8},
  "paired_binary": {
    "baseline_pass_subject_fail": 1,
    "baseline_fail_subject_pass": 2,
    "both_pass": 14,
    "both_fail": 3,
    "discordant_pairs": 3,
    "mcnemar_exact_two_sided_p_value": 1.0,
    "effect_size_pp": 5.0,
    "effect_size_confidence_interval": {
      "method": "newcombe_hybrid_score_paired_v2",
      "confidence_level": 0.95,
      "lower_pp": -14.975805254302834,
      "upper_pp": 24.866848542801065
    }
  },
  "sample_qualification": {
    "record_count": {"minimum": 20, "observed": 20, "passed": true},
    "interval_width": {
      "maximum": 50.0,
      "observed": 39.8426537971039,
      "unit": "percentage_points",
      "passed": true
    },
    "passed": true
  },
  "uncertainty": {
    "interval_mass": 0.95,
    "lower": -14.975805254302834,
    "method": "newcombe_hybrid_score_paired_v2",
    "scope": "paired_binary_outcomes",
    "upper": 24.866848542801065
  },
  "verdict": "pass"
}

Values are illustrative. Verification reconstructs the whole object and requires canonical equality. It does not independently trust stored means, point comparison, interval bounds, threshold, or verdict.

Exact match

{
  "kind": "exact_match_delta_pp",
  "value": 1.5,
  "minimum": -2.0
}

value = math.fsum(subject_i - baseline_i) / record_count * 100.0. The metric-bound check passes when uncertainty.lower >= comparison.minimum, where minimum comes from resolved_policy.metrics.exact_match.delta_min_pp. If sample qualification is present, its count and width checks must also pass. If side-accuracy qualification is present, both side means must meet its minimum. The final verdict is the conjunction of all applicable checks.

For this metric, baseline and subject mean_score are literal exact-match accuracies between 0 and 1. Comparison and interval bounds are percentage points between -100 and 100.

paired_binary reports baseline-pass to subject-fail regressions, baseline-fail to subject-pass improvements, both-pass and both-fail counts, the number of discordant pairs, and the exact two-sided McNemar probability. Its effect size is the same subject-minus-baseline percentage-point delta. The continuity-corrected paired Newcombe hybrid-score 95% interval is repeated as the canonical uncertainty object in a v2 or v3 report. The policy uses its lower bound; the McNemar probability does not control the verdict. A v1 report keeps method newcombe_hybrid_score_paired_v1 and is replayed only with that original method.

Here math.fsum(subject_i - baseline_i) denotes summing every paired record difference with math.fsum. Divide by the record count, then multiply by 100.0, in that order. Computing the same mathematical delta by subtracting separately rounded side means can produce different floating-point bytes. See exact replay arithmetic for the contract.

Sample and precision qualification

A metric policy may omit sample qualification. If it enables qualification, the two fields must be supplied together:

• minimum_record_count plus maximum_interval_width_pp for exact match or an authorized scorer extension; or
• minimum_record_count plus maximum_interval_width_ratio for normalized NLL.

minimum_record_count is an integer from 1 through 10,000. Percentage-point width must be positive and at most 200; ratio width must be positive. The report records the observed width as uncertainty.upper - uncertainty.lower. Its final verdict is:

metric bound passes
and observed record count >= minimum record count
and observed interval width <= maximum interval width

The report therefore distinguishes a metric rejection from insufficient count or precision without allowing a favorable point value to bypass either gate.

Side-accuracy qualification

An exact-match policy may independently set minimum_side_accuracy from 0 through 1. A v3 report then includes side_accuracy with the configured minimum, each side's observed exact-match mean and pass result, and a combined passed result. Both sides must meet the inclusive floor. This check is independent of the coupled sample controls, and the final verdict also requires it to pass. Historical v2 reports do not contain this section.

Normalized NLL per UTF-8 byte

{
  "kind": "normalized_nll_ratio",
  "value": 0.98,
  "maximum": 1.05
}

Each side's mean_score is the arithmetic mean of verifier-derived per-record teacher-forced expected-continuation -logprob_sum / utf8_byte_count. The baseline mean must be positive and value = subject_mean / baseline_mean. The metric-bound check passes when uncertainty.upper <= comparison.maximum, where maximum comes from resolved_policy.metrics.normalized_nll_per_utf8_byte.ratio_max.

This is a ratio of equally weighted record means, not pooled NLL, byte-weighted loss, or perplexity. It measures expected-continuation likelihood regression under authenticated inputs and runtime, not general model quality.

Derived perplexity interpretation

{
  "perplexity_ratio": {
    "status": "available",
    "basis": "authenticated_target_likelihood",
    "method": "target_token_weighted_perplexity_ratio_v1",
    "tokenizer_metadata_sha256": "...",
    "target_token_count": 4096,
    "baseline_perplexity": 12.3,
    "subject_perplexity": 12.6,
    "ratio": 1.0243902439
  }
}

For normalized NLL, derived_measurements.perplexity_ratio is available when both artifacts bind matching authenticated tokenizer metadata and every pair has the same positive target-token count. The verifier derives both side perplexities and their ratio from authenticated target log probabilities. When the facts are not comparable, the object carries status: unavailable and a closed reason instead.

This object is interpretation only. It is not a metric or policy input and has no threshold, confidence interval, or verdict authority.

Normalized-NLL paired resampling interval

Every normalized-NLL comparison report contains:

{
  "method": "paired_percentile_bootstrap_sha256_v1",
  "scope": "authenticated_schedule",
  "interval_mass": 0.95,
  "replicates": 2048,
  "lower": 0.98,
  "upper": 1.04
}

Index draws are derived from SHA-256 of the authenticated schedule digest, replicate number, and draw number. Baseline and subject scores at each selected schedule position are resampled together. The complete comparison is recomputed 2,048 times; lower and upper are linearly interpolated 2.5th and 97.5th percentiles. A one-record schedule yields a point interval.

The interval and verdict are deterministic verifier-replay outputs. The scope authenticated_schedule is literal: this is a finite-schedule stability interval, not a population confidence interval or representativeness claim.

Authorized deterministic scorer extension

An extension report uses:

{
  "kind": "scorer_extension_delta_pp",
  "value": 2.5,
  "minimum": -1.0
}

For each side, an explicitly authorized deterministic scorer replays exactly the authenticated expected output, output text, and output digest for every record and returns one higher-is-better value in [0, 1]. The engine owns the arithmetic mean and computes value = math.fsum(subject_i - baseline_i) / record_count * 100.0. It then applies the same fixed 2,048-replicate paired schedule-resampling method to the paired record values. The metric-bound check passes only when uncertainty.lower >= comparison.minimum, where minimum comes from resolved_policy.metrics.scorer_extension.delta_min_pp.

scorer_extension records the bound scorer ID, version, descriptor digest, and configuration digest. scorer_replay binds the per-side deterministic replay results. Verification requires the caller to authorize the same scorer in a ScorerExtensionRegistry, runs it twice, and reconstructs the complete report. A stored scorer result is never accepted as an aggregate assertion.

The scorer-extension v1 contract supports deterministic text replay. Core ships invarlock.normalized_match, invarlock.numeric_tolerance, invarlock.json_fields, invarlock.json_exact and invarlock.token_f1; the CLI can use these without installed-scorer authorization. SDK callers supply a ScorerExtensionRegistry(allow_installed=False). Other implementations, such as a separately supplied VQA normalization scorer, require explicit authorization. SQL or code execution, model-based semantic similarity, network services or externally assigned ratings, and LLM judges are outside scorer-extension acceptance replay. The built-in judge scorer replays its bounded measurement contract and, for native run/import evidence, its retained runtime capture. Its report shows both evaluated model identities, runtime settings and digests, judge model/configuration, rubric, counts and uncertainty. Other recorded judge results may be authenticated as observations, where they have no verdict authority.

Supplementary diagnostics and review

Optional NumPy diagnostics can be published as authenticated observations; verification checks their retained bytes and bindings without making them verdict inputs. An RMT observation keeps its numerical method identifier. The explicit column_standardized_smaller_gram_eigh method additionally reports gram_dimension, gram_matrix_bytes and minimum_upper_edge_distance inside its payload. The distance is descriptive, not an error bound or a significance level. See diagnostics.

The exact-match sensitivity helper returns a separate Python advisory. It is not a comparison-report field or a signed verification result. Preserve the recorded verdict and interpret its hypothetical edits and search limits separately.

The standalone score contribution recipe produces a separate explanation of an additive total. It is not emitted by invarlock report or added to a signed comparison. Validate source completeness, unit weights and score-versus-percentage-point units before using it; its residual is a gap from a supplied total, not proof of numerical error or causality.

Verification result

Native deterministic invarlock verify --json emits an invarlock/evidence-pack-verify-v1 result. Important fields include:

The high-level transaction adds signed_receipt, verifier_identity, and verifier_fingerprint. Profile-based verification also adds trust_profile_digest. Stdout is useful process output; the separately signed receipt is the portable verifier assertion.

Captured verification emits invarlock/evidence-pack-verify-v2 with kind: captured, ok, integrity_ok, policy_verdict, decision, replay_status, signed_receipt, pack_manifest_digest, and metric_summaries. A completed regression or insufficient_evidence decision has policy_verdict: fail, integrity_ok: true, and ok: false (exit 7). Local work-budget or unsupported-scoring-environment refusal is incomplete verification (exit 2) without a receipt, not evidence corruption.

Do not infer acceptance from integrity_ok alone. Native deterministic acceptance requires assurance_status: verified; captured acceptance requires replay_status: completed. Both require status 0, ok: true, integrity_ok: true, exact anchors, and a valid receipt when the result crosses a process boundary.

Signed verification receipt

The receipt contains a statement and signature. The statement format is invarlock/evidence-verification-receipt-v1 for native evidence without a request anchor; v2 additionally binds an independently supplied request digest. Example v1:

{
  "format": "invarlock/evidence-verification-receipt-v1",
  "pack_manifest_digest": "sha256:...",
  "anchors": {
    "policy_digest": "sha256:...",
    "artifact_digests": {
      "baseline": "sha256:...",
      "subject": "sha256:..."
    },
    "schedule_digest": "sha256:...",
    "runtime_digests": {
      "baseline": "sha256:...",
      "subject": "sha256:..."
    },
    "pack_signer_fingerprint": "sha256:..."
  },
  "verifier": {
    "identity": "release-verifier",
    "signing_key_fingerprint": "sha256:...",
    "trust_profile_digest": "sha256:..."
  },
  "verdict": {
    "ok": true,
    "integrity_ok": true,
    "policy_verdict": "pass",
    "verification_status": 0
  }
}

The signature envelope is invarlock/evidence-verification-receipt-signature-v1. It uses Ed25519, embeds the verifier public key in PEM form, and signs canonical statement bytes. The embedded key is verification material, not a trust source. A receipt reader must still supply the expected verifier identity and fingerprint, pack, policy, artifact digests, schedule digest, runtime digests, and expected evidence signer. For profile-based verification, it must also supply the expected canonical trust-profile digest. Explicit-option receipts record null for that field.

A receipt is written for a completed policy or integrity rejection when the transaction can form a safe statement, then the command exits nonzero. A successful receipt is internally consistent only when ok and integrity_ok are true, policy_verdict is not fail, and verification_status is zero.

Downstream readers use invarlock.engine.verify_signed_verification_receipt. The stable API returns a ReceiptVerification. Its ok authenticates the receipt, including an authentic rejection; acceptance additionally requires statement["verdict"]["ok"] to be true.

Captured packs use only invarlock/evidence-verification-receipt-v3. Its verification_scope: captured_comparison cannot authorize native acceptance or deployment. Anchors bind the expected complete baseline/subject runs, policy, normalized request, and signer; pack_manifest_digest identifies the manifest actually examined, even if it differs from those expectations. replay_status is not_started, failed, or completed. scoring_assurance is an ordered array of {name, slice, kind, scoring_assurance} entries with each assurance recomputed or recorded, populated only for complete matching replay under intact bindings; integrity rejection carries null, even after replay completed. A signed rejection may record an unsigned input pack without authenticating it. A valid signature on a rejection receipt may authenticate successfully while statement.verdict.ok remains false. Receipt authentication does not replay payloads or promote a rejection to acceptance. Native receipt v1/v2 semantics and historical bytes remain unchanged.

Console and HTML reports

Render a controlled evaluation bundle for inspection:

invarlock report evidence/ --html comparison.html
invarlock report evidence/ --explain
invarlock report evidence/ --html comparison.html --json

invarlock report checks the closed inventory, checksums, canonical JSON and embedded evidence signature before rendering. The console displays a formatted summary; the self-contained HTML works offline and can be printed. The HTML file must be outside the evidence pack. Rendering preserves every bundle byte. --json emits a rendering result object, including the HTML path when requested; it does not turn the report into an independent technical verification receipt.

Native and captured report calls return invarlock/evidence-report-v2 for every output selection, including console-only and HTML-only rendering:

{
  "format_version": "invarlock/evidence-report-v2",
  "kind": "captured",
  "ok": true,
  "pack_manifest_digest": "sha256:...",
  "requested_outputs": {"html": "report.html", "junit": "results.xml"},
  "written_outputs": {"html": "report.html", "junit": "results.xml"},
  "failed_output": null,
  "errors": []
}

kind is runtime for native v2 output. Destination maps can contain html, markdown, and junit. All paths must be distinct, new, and outside the evidence directory. Upfront collision refusal writes nothing; a later write failure returns ok: false, lists completed files in written_outputs, and identifies the failed format. Rendering does not discover or authenticate adjacent receipts. Unsigned captured packs remain explicitly unsigned local reports. JUnit records regression as failure and insufficient evidence as error.

Read the result and its requirements

After the decision context, the report shows Policy satisfied, Policy not met or More evidence needed, the recorded verdict and its explanation. HTML places the assurance checks beside the verdict, with signature validation and independent recipient acceptance kept separate. Each metric shows baseline and candidate values, change, observed pair count, an interval and the configured requirements. The interval diagram labels its scale, the policy boundary and the no-change reference: zero for a difference and one for a ratio. Shading marks only the region meeting the change requirement. Labels identify the interval endpoints, estimate, policy limit and neutral reference; axis ticks use rounded steps in the displayed units. Separate count, precision and absolute score requirements still apply. The table shows which numerical checks passed or were not met. Exact recorded values remain available in the evidence and technical details.

For captured comparisons, the technical details identify both recorded runs, their complete-run digests, attributed artifact digests and evaluator source name and version. These fields make the report traceable to the exact supplied runs. Optional model and prompt context describes the evaluator-recorded change; it does not establish checkpoint identity.

Hosted captured evidence attributes a service observation, with an explicit configuration, harness and observation window, instead of asserting an immutable weight artifact. An observed model name or exposed revision remains a source assertion. Replaying or rendering the evidence does not make new service calls or establish current service behavior.

Use the failure-state table to distinguish an authentic policy rejection from a submission that cannot be verified.

For exact match, means appear as percentages and changes as percentage points. The decision tests the paired interval's lower bound, not just the observed change. An improving candidate can still fail an absolute accuracy floor. When configured, the report shows observed and required record count, interval width, and the separate baseline and candidate accuracy checks. An absent requirement is identified as absent; rendering does not supply a new threshold.

For normalized negative log-likelihood, the change is a candidate-to-baseline ratio and the upper ratio bound is compared with the policy maximum. Its finite-schedule resampling interval describes stability on the authenticated schedule. It does not establish population uncertainty. Available derived perplexity values appear separately and do not affect acceptance; an unavailable interpretation includes its recorded reason.

Complete binary-mean results can also show exact match counts when the retained mean and record count determine an integer count consistently. Missing results and fractional scores do not acquire match counts. HTML places match counts beneath the corresponding score, and missing, included and required pair counts beneath the usable-pair count. Required counts appear only when policy is bound.

HTML includes expandable identities, exact comparison data, paired outcome analysis where available, and supplementary authenticated observations. --explain adds technical details to the text view. Paired outcome analysis can include improvement and regression counts, discordant pairs and the exact McNemar probability; those diagnostics do not replace the configured decision.

Understand the comparison

HTML, Markdown and terminal reports share the same comparison grouping and put What was compared before the results. HTML aligns sides in a table; Markdown and terminal output use paired fields so long identifiers remain readable at narrow widths. The section aligns baseline and subject fields side by side, highlights differences, and groups matching displayed fields with the recorded workflow, task, dataset and coverage information under Additional recorded context. Matching previews do not establish equality of the complete retained fields. They appear under Matching displayed fields, separately from additional context and ambiguous entries; acronyms and original ambiguous labels are preserved. On small screens, each comparison field stacks its explicitly labelled baseline and subject values while retaining the table headers for assistive technology. The recorded changes explain what the evidence establishes about the difference between the two sides.

Native run and import reports use the authenticated request and input identities. They show recorded model IDs and provider names, dataset preparation and schedule identity when available. Different artifact digests establish different artifacts; they do not establish which model setting, prompt or weight changed.

Captured reports inspect context across every record. Consistent recorded model keys, revisions, capture roles, dataset labels and message structure appear as comparison context. Missing or mixed values are explicit. A matching model key is a recorded label, not proof that model weights or runtime settings were identical. The renderer does not infer a model revision from filenames or external catalogs.

For paired prompt changes, the report compares retained message sequences by case ID. Effective messages and captured HTTP request messages are distinct projections; the latter do not reveal hidden service-side instructions. When every pair differs only by one uniform added system instruction, the report identifies that change and includes a bounded instruction preview in the details. It does not display full case prompts by default. Missing or inconsistent context cannot establish a uniform prompt change. Comparison-only SDK reports cannot recover record context without the associated evidence.

Compare multiple metrics and scopes

Reports with multiple results include an expandable overview of every metric and scope. It shows the baseline, subject, change, observed pair count, decision and any failed or unavailable checks. The overall verdict and result counts remain visible when choosing a metric; selecting one result does not change the overall decision. The overview opens automatically if any result fails or needs more evidence, so adverse checks are visible before choosing a tab.

Metric tabs group related scopes together. For example, quality and latency measured overall and on an exceptions slice produce four results under two metrics. Select a tab or an overview result to inspect its details. Use the arrow keys, Home and End to move between tabs, or Show all to read every metric in one page. Reports with one metric do not need tabs. Overlapping scope counts must not be added together as though they were separate cases.

The report remains one offline HTML file. A fixed, hash-authorized script handles navigation only; no evidence labels or values are inserted into executable code. Without JavaScript, all metric details remain visible and section links still work. Printing includes every metric, even when only one tab is selected, and the contents of closed explanations, including interval methods. Raw evidence details print when opened, keeping the default review from expanding into a long dump of technical data. Complete exact values remain in the HTML and original evidence either way.

The optional real-browser regressions check default and expanded PDF text on A4 and US Letter, including printing without backgrounds, first-page review actions, page numbering and 320-pixel overflow for native, captured, judge and combined evidence. A separate synthetic stress fixture checks keyboard navigation, hidden-result links, computed text/graphic contrast, enlarged text spacing, long identifiers, offline use and operation without JavaScript. With an installed Playwright Node module and a Python interpreter containing pypdf, run:

INVARLOCK_REPORT_PLAYWRIGHT_MODULE=/absolute/path/to/playwright \
INVARLOCK_REPORT_PDF_PYTHON=/absolute/path/to/python \
PYTHONPATH=src python -m pytest \
  tests/reporting/test_report_browser_print.py \
  tests/reporting/test_report_browser_access.py

The check makes no model calls or dependency downloads. It is skipped when the two tool paths are not supplied; ordinary unit tests do not prove browser print behavior.

Keep rendering separate from recipient acceptance

The assurance panel distinguishes embedded bundle authentication from independent recipient acceptance. A bundle's embedded signer is not a recipient-owned trust anchor. Use invarlock verify with independent policy, identity and signer inputs to produce the signed acceptance or rejection receipt. A report can show an authenticated policy failure.

Record the organizational release decision separately, with its scope and remaining conditions.

A successful report invocation indicates that rendering completed, including when the recorded policy verdict is fail. Automation should parse verified JSON or validate a receipt, never scrape the console or HTML. Renderer failures use the selected text or JSON mode; argument-parser errors remain usage errors.

Reports from captured evaluation exports

invarlock evaluate request.yaml --signing-key signing-key.pem
invarlock verify evidence/ --trust-profile recipient/trust-inputs.json \
  --receipt verification.receipt.json
invarlock report evidence/

The trust profile must independently supply the captured policy, complete-run and request pins, evidence signer, and verifier identity and key. The core commands retain JSON output for automation. report regenerates presentations from bounded evidence after checking the closed inventory, payload digests, canonical JSON, cross-file bindings and embedded signature for signed packs. It does not recompute scores or replay comparison arithmetic; use verify for independent authentication and replay.

Captured reports distinguish unsigned local evidence, an embedded signature validated without recipient authorization, and unavailable signing information in a comparison-only view. They preserve pass, regression and insufficient_evidence as the recorded decisions. A regression decision means a policy bound failed, which can be an absolute floor even when the observed candidate improved. Missing paired results remain missing; overlapping scope counts are not independent samples. Advanced configuration and missing-ID lists use labeled previews while the original values remain in bound evidence. Configuration previews use indented text with explicit limits and truncation markers; the renderer does not reinterpret them as executable content or complete configuration. Small screens stack each requirement with labelled Observed, Required and Result values. All checks remain visible without horizontal scrolling. HTML uses the light color scheme regardless of system appearance, including when printed.

Successful captured report regeneration exits 0 regardless of the stored policy decision. Evaluation exits 0 on publication unless --fail-on-policy requests the local gate; independent verification rejects adverse decisions. See the captured-results guide for the full workflow and its assurance limits.

Failure-state interpretation

• State

  Accepted

  Integrity

  True

  Policy

  pass

  Portable receipt

  Signed success receipt

  Interpretation

  Conservative interval bound and any configured sample/precision requirements met the policy under all supplied anchors

• State

  Policy rejection

  Integrity

  True

  Policy

  fail

  Portable receipt

  Signed rejection receipt when completion reached

  Interpretation

  Authentic evidence did not meet the policy

• State

  Integrity rejection

  Integrity

  False

  Policy

  Unavailable or untrusted

  Portable receipt

  Signed rejection receipt when safe completion reached

  Interpretation

  Pack must not be used

• State

  Precondition failure

  Integrity

  Not completed

  Policy

  Unavailable

  Portable receipt

  May be absent

  Interpretation

  Caller input or structure prevented completed verification

• State

  Render success only

  Integrity

  Embedded signature checked for signed packs; unsigned evidence remains local

  Policy

  Recorded verdict only

  Portable receipt

  None

  Interpretation

  Inspection of evidence, not independent technical verification

Related documentation

• Decision semantics defines the exact score, interval, and threshold arithmetic.
• Evidence artifacts maps reports to fixed bundle paths.
• Public contracts specifies canonical bytes and signature dependency order.
• Command-line interface documents JSON modes and exit status.
• Python API defines result and signed-receipt types.

For bounded frozen-answer ratings, see judge measurements. Judge reports keep offline replay, signer authentication and recipient acceptance distinct and use additive judge-specific JSON formats. Rendering replays the retained measurements; verify performs envelope signature authentication against the recipient policy. They show the first 50 case IDs by default; repeat report --case-id ID to inspect any retained cases without loading every answer and judge response into one report. Case selection does not change the complete replay or the recorded policy decision.