Decision semantics

Assurance note

In plain language: For native exact match and NLL, InvarLock recomputes scores from authenticated paired records, derives the paired interval, and applies the policy to the conservative bound. When the policy qualifies sample size, precision, or exact-match side accuracy, those checks must pass too. A favorable point value cannot override a threshold, count, width, or side- accuracy failure.

Question: How does each scoring contract turn its measurements and uncertainty into a replayable policy decision?

Decision use: Use these definitions to review thresholds, reproduce canonical report arithmetic, and interpret a pass or fail at the boundary.

Evidence: Pinned local source identity, canonical schedule, provider observations, verifier-derived paired scores, exact policy bytes, independent artifact and schedule anchors, and the metric-specific paired interval recorded in the report.

Choose the decision contract

The selected evidence family determines the arithmetic and the meaning of its result labels. Native judge selection uses the bounded judge contract, even though it shares the main commands with native exact match and NLL.

Data table with columns: Evidence family, Decision basis, Meaning when the result does not pass
Evidence family	Decision basis	Meaning when the result does not pass
Native exact match, NLL or deterministic extension	Authenticated provider records, the metric-specific paired interval and configured qualification checks	`fail` means the policy was not met; it does not by itself establish degradation
Captured deterministic, likelihood or recorded scores	Every declared metric and slice, with conservative comparison bounds and optional subject-mean bounds	`regression` means a bound check failed; missing facts, too few records or excessive width alone yield `insufficient_evidence`
Bounded judge, including native judge	Complete planned trials, declared units, bounded-score intervals and the judge analysis policy	`regression` requires an entirely adverse interval after count and precision checks; a boundary-straddling interval is inconclusive

Scroll horizontally to see every column.

A combined evidence set requires both component acceptances and preserves their statistical scopes. It does not create a new joint confidence guarantee. The detailed native derivation follows; captured and judge rules appear under Captured comparison and bounded judge decisions.

The native exact-match, normalized-NLL and deterministic extension contract makes one decision over one authenticated, ordered, finite schedule. It reports a point comparison and a verifier-replayed paired interval. Exact match uses a paired Newcombe 95% effect-size interval. Normalized NLL and an authorized scorer extension use deterministic paired resampling over the authenticated schedule.

The notation, preconditions and equations through the deterministic-extension section apply to that native contract, including its 10,000-record limit. Captured comparison capacity and judge plan limits are separate.

New evaluations under that contract emit invarlock/comparison-report-v3. Strict verification continues to replay v2 reports without side-accuracy qualification and v1 reports with their original exact-match interval method, so signed historical evidence keeps its original meaning.

Notation

Data table with columns: Symbol, Meaning
Symbol	Meaning
$n$	Number of scheduled records; $1 \le n \le 10{,}000$
$i$	Schedule position, $i \in \{1, \ldots, n\}$
$B$ , $S$	Baseline and subject sides
$y_i$	Scheduled expected output for record $i$
$\hat{y}_{X,i}$	Authenticated output text for side $X$
$L_{X,i}$	Authenticated sum of target-token log probabilities
$u_i$	Positive UTF-8 byte count of $y_i$
$w_i$	Positive target token count used only for a derived perplexity interpretation
$x_{X,i}$	Verifier-derived record score for side $X$
$\bar{x}_X$	Arithmetic mean of side $X$ record scores
$\tau_{\Delta}$	Policy floor `exact_match.delta_min_pp`
$\tau_A$	Optional exact-match floor `minimum_side_accuracy` for each side mean
$\tau_{N}$	Policy ceiling `normalized_nll_per_utf8_byte.ratio_max`
$\tau_{S}$	Policy floor `scorer_extension.delta_min_pp`
$[q_L,q_U]$	Metric-specific paired interval
$n_{\min}$	Optional policy minimum `minimum_record_count`
$w_{\max}$	Optional policy maximum interval width in percentage points or ratio units

Shared preconditions

Replay proceeds only when:

the normalized request, source identity, and canonical schedule are bounded and digest-authenticated;
record IDs are unique and both observations have exactly the schedule IDs in schedule order;
each observed input_sha256 matches the corresponding scheduled input;
every selected observation record has status: ok;
provider, artifact, observation, schedule, runtime, report, and request bindings are internally consistent;
the schedule and both providers declare the selected task and provider collection metric;
the verifier's independently supplied baseline artifact, subject artifact, and canonical schedule digests match the identities bound into the signed evidence; and
the verifier's policy bytes have the same digest as the policy bound into the signed evidence.

Failure of a precondition is a verification error, not a poor metric score. InvarLock does not impute, discard, reorder, or average around an invalid record.

Exact-match percentage-point delta

For each side and scheduled record:

x_{X,i} = \mathbf{1}\!\left[\hat{y}_{X,i}=y_i\right].

Equality is literal Unicode string equality. It does not trim whitespace, case-fold, normalize, parse an answer, or compare token sequences.

The side means and point comparison are:

\bar{x}_X = \frac{1}{n}\sum_{i=1}^{n}x_{X,i}, \qquad \Delta_{\mathrm{pp}} = 100\left(\bar{x}_S-\bar{x}_B\right).

The report records comparison.value = $\Delta_{\mathrm{pp}}$ . The policy is:

{
  "resolved_policy": {
    "metrics": {
      "exact_match": {
        "delta_min_pp": -2.0,
        "maximum_interval_width_pp": 10.0,
        "minimum_record_count": 400,
        "minimum_side_accuracy": 0.5
      }
    }
  }
}

The verdict uses the interval lower bound:

\operatorname{pass}_{\mathrm{EM}} \iff q_L \ge \tau_{\Delta} \land n \ge n_{\min} \land (q_U-q_L) \le w_{\max} \land \bar{x}_B \ge \tau_A \land \bar{x}_S \ge \tau_A.

The boundary is inclusive. delta_min_pp is an absolute percentage-point floor, not a relative percentage and not a floor applied only to the point estimate. The count and width terms apply only when the policy supplies both coupled fields; without them, the threshold-only policy remains valid. The two side-accuracy terms apply only when minimum_side_accuracy is present. That optional finite value lies from 0 through 1 and prevents a passing relative comparison between two sides whose absolute exact-match accuracy is below the policy floor.

For every pair, InvarLock also classifies the two binary outcomes:

baseline pass and subject fail: a regression;
baseline fail and subject pass: an improvement;
both pass; or
both fail.

If $r$ is the regression count and $g$ is the improvement count, then $\Delta_{\mathrm{pp}}=100(g-r)/n$ . The report records all four counts, the number of discordant pairs $r+g$ , and the exact two-sided McNemar probability conditional on those discordant pairs. The McNemar result describes evidence of asymmetry between regressions and improvements; it does not control the acceptance verdict.

The exact-match interval is a paired Newcombe hybrid-score 95% interval for the subject-minus-baseline effect. Current v3 reports use the continuity-corrected method:

{
  "method": "newcombe_hybrid_score_paired_v2",
  "scope": "paired_binary_outcomes",
  "interval_mass": 0.95,
  "lower": -1.8,
  "upper": 0.7
}

The policy reads lower; the point delta and McNemar probability cannot override a lower bound below delta_min_pp.

For a signed v1 report, strict replay instead requires newcombe_hybrid_score_paired_v1 and reconstructs the legacy result exactly. It does not apply the v2 calculation to old evidence.

Normalized NLL per UTF-8 byte

For each side and record, the provider performs teacher-forced scoring of the authenticated expected continuation and supplies finite $L_{X,i}$ and positive byte count $u_i$ . Prompt tokens are excluded from the target loss. The verifier requires the byte count to equal the UTF-8 length of the authenticated scheduled target, then derives:

x_{X,i} = -\frac{L_{X,i}}{u_i}.

Every derived value must be finite and nonnegative. The side means use equal record weight:

\bar{x}_X = \frac{1}{n}\sum_{i=1}^{n}x_{X,i}.

The baseline mean must be positive. The point ratio is:

R_{\mathrm{NLL}} = \frac{\bar{x}_S}{\bar{x}_B}.

The policy is:

{
  "resolved_policy": {
    "metrics": {
      "normalized_nll_per_utf8_byte": {
        "maximum_interval_width_ratio": 0.05,
        "minimum_record_count": 400,
        "ratio_max": 1.05
      }
    }
  }
}

The verdict uses the interval upper bound:

\operatorname{pass}_{\mathrm{NLL}} \iff q_U \le \tau_N \land n \ge n_{\min} \land (q_U-q_L) \le w_{\max}.

This statistic is a ratio of arithmetic means of per-record normalized scores. It is not pooled NLL, a byte-weighted mean, token-weighted mean, or perplexity. Each selected record contributes one score regardless of target length. It measures expected-continuation likelihood regression under the authenticated prompt, target, provider, and runtime; it is not a general model-quality score.

The count and ratio-width terms apply only when their coupled policy fields are present. Their pass does not establish that the schedule represents a broader population.

Worked normalized-NLL example

Suppose two authenticated records yield:

Data table with columns: Record, Baseline xB,ix_{B,i}xB,i, Subject xS,ix_{S,i}xS,i
Record	Baseline $x_{B,i}$	Subject $x_{S,i}$
1	$-(-5)/10 = 0.5$	$-(-5.5)/10 = 0.55$
2	$-(-20)/20 = 1.0$	$-(-22)/20 = 1.1$

Scroll horizontally to see every column.

Then $\bar{x}_B=0.75$ , $\bar{x}_S=0.825$ , and the point ratio is $1.1$ . That point fails a ratio_max of 1.05; even when a point ratio is below the ceiling, the canonical verdict still fails if its interval upper bound exceeds 1.05.

Derived perplexity interpretation

For a normalized-NLL comparison, the verifier may also derive a token-weighted perplexity ratio when both sides bind one matching authenticated tokenizer contract and report the same positive target token count $w_i$ for every pair. It derives token-normalized record NLL:

x_{X,i} = -\frac{L_{X,i}}{w_i}.

The token-weighted mean NLL and side perplexity are:

\mu_X^{(w)} = \frac{\sum_{i=1}^{n}w_i x_{X,i}}{\sum_{i=1}^{n}w_i}, \qquad P_X = \exp\!\left(\mu_X^{(w)}\right).

The derived ratio is:

R_{\mathrm{PPL}} = \frac{P_S}{P_B} = \exp\!\left(\mu_S^{(w)}-\mu_B^{(w)}\right).

The canonical derived_measurements.perplexity_ratio object records the authenticated likelihood basis, method, tokenizer digest, total target-token count, both side perplexities, and the ratio. If tokenizer contracts differ, token counts are unavailable or unequal, or exponentiation is non-finite, the object records a deterministic unavailable reason instead.

This interpretation is not a selectable metric. It has no policy threshold, confidence interval, or verdict authority. Acceptance continues to use the normalized-NLL ratio and its schedule-resampling interval.

Normalized-NLL paired resampling

The report's uncertainty object is fixed:

{
  "method": "paired_percentile_bootstrap_sha256_v1",
  "scope": "authenticated_schedule",
  "interval_mass": 0.95,
  "replicates": 2048,
  "lower": 0.98,
  "upper": 1.04
}

Let $d$ be the 32 bytes represented by the canonical schedule's lowercase SHA-256. For replicate $r \in \{0,\ldots,2047\}$ and draw $j \in \{0,\ldots,n-1\}$ , InvarLock computes:

h_{r,j} = \operatorname{SHA256} \left(d\,\|\,\operatorname{u64be}(r)\,\|\,\operatorname{u64be}(j)\right),

and selects index:

k_{r,j} = \operatorname{u64be}\!\left(h_{r,j}[0:8]\right) \bmod n.

The baseline and subject normalized-NLL scores at $k_{r,j}$ are drawn together. InvarLock recomputes the complete ratio for each replicate, sorts the 2,048 values, and uses linearly interpolated percentiles at $0.025$ and $0.975$ . When $n=1$ , the interval is the point value.

This construction provides:

deterministic replay without a platform PRNG;
pairing preservation under every draw;
a versioned algorithm and fixed replicate count; and
a conservative bound that directly controls the policy verdict.

It does not provide random-sample provenance. The SHA-256-derived index stream makes replay deterministic; it does not make the original selected schedule a probability sample.

Authorized deterministic scorer extension

A request may select one closed scorer_extension binding instead of a built-in metric. For every side and record, the provider authenticates exactly the scheduled expected output, output text, and output digest. The explicitly authorized scorer replays those facts and returns:

0 \le x_{X,i} \le 1,

where larger values are always better. The scorer does not own aggregation or policy arithmetic. The core computes:

\bar{x}_X = \frac{1}{n}\sum_{i=1}^{n}x_{X,i}, \qquad \Delta_{S,\mathrm{pp}} = 100\left(\bar{x}_S-\bar{x}_B\right).

The independent policy pins the exact scorer identity and configuration:

{
  "resolved_policy": {
    "metrics": {
      "scorer_extension": {
        "scorer_id": "example.token_f1",
        "scorer_version": "1.0.0",
        "descriptor_sha256": "5555555555555555555555555555555555555555555555555555555555555555",
        "configuration_sha256": "6666666666666666666666666666666666666666666666666666666666666666",
        "delta_min_pp": -2.0,
        "maximum_interval_width_pp": 10.0,
        "minimum_record_count": 400
      }
    }
  }
}

The scorer runs twice during replay and both canonical results must match. The core then applies the fixed 2,048-replicate paired schedule-resampling method defined above to the paired unit-interval values. The verdict is:

\operatorname{pass}_{S} \iff q_L \ge \tau_S \land n \ge n_{\min} \land (q_U-q_L) \le w_{\max}.

Potential separately implemented scorers include deterministic token F1, structured-field extraction, and VQA answer normalization. The scorer-extension v1 contract does not admit SQL or code execution, model-based semantic similarity, network services, externally assigned ratings, external models, or LLM judges. Those sources do not execute through the deterministic extension contract.

The native judge scorer uses a separate bounded measurement decision contract. It retains and replays the declared judge requests, responses, attempts, parsing, independent-unit aggregation, interval, and recipient policy. It also accepts captured evaluator answers and retained judge calls. See judge measurements for those acceptance rules; no live judge calls occur inside deterministic-extension replay.

For the native exact-match, NLL and extension metrics above, the sample controls are optional but indivisible. Exact match and scorer extensions use maximum_interval_width_pp, whose value must be positive and no greater than 200. Normalized NLL uses positive maximum_interval_width_ratio. minimum_record_count is an integer from 1 through 10,000. The canonical v3 report records the minimum, maximum, observed values, units, individual results, and combined sample_qualification.passed.

Captured comparison and bounded judge decisions

Captured exact-match and normalized-NLL requests use the invarlock/multi-metric-comparison-v1 contract. Its binary metrics use paired Newcombe intervals in score units; continuous delta metrics use paired_mean_shake256_percentile_v1, which differs from the native SHA-256 resampling method. Captured NLL reuses the native mean-ratio arithmetic and paired_percentile_bootstrap_sha256_v1, seeded from the sorted scope's retained case facts rather than a native provider schedule. Captured policy evaluates every metric and slice; its marginal intervals do not provide simultaneous family-wide coverage.

Captured recorded metrics can apply policy to upstream scores, including judge or other externally assigned ratings with explicit approved provenance and rubric binding. Replay recomputes aggregation and decision arithmetic, not the upstream judgment. Missing facts or insufficient count produce insufficient_evidence; a violated regression or subject bound produces regression; an otherwise acceptable but overly wide interval produces insufficient_evidence. The captured regression label is a policy-gate result: its conservative bound can fail even when the interval straddles the allowed-degradation boundary. It does not claim that the entire interval demonstrates adverse change. Absolute subject bounds apply to the observed subject mean, not a confidence bound on that mean. A bound violation takes precedence over excessive width; missing measurements or an unmet minimum count are checked before either. Across metric/slice rows, any regression takes precedence over insufficient_evidence. All rows must pass for the comparison to pass. See the captured policy contract.

Selecting judge instead uses fixed-benchmark-hoeffding-v1 over bounded, declared units under an assumption of independence across those units. Replay checks grouping and arithmetic; it cannot establish that independence. Replay averages repetitions within cases and cases within units before calculating equal-unit means. Repetitions and additional cases in one unit do not increase the inference sample size. The Hoeffding bound targets the fixed benchmark's average expected score under the declared independence assumption; it is not the schedule-composition bootstrap described above. Bonferroni adjustment uses the declared family size and error budget, including published advisory intervals. Constant observed ratings still have positive interval width unless the declared support itself is constant.

Judge gates distinguish pass, regression and insufficient_evidence. Within each judge gate, minimum-unit and maximum-width checks take precedence: a failure yields insufficient_evidence. Once those checks pass, an interval entirely beyond tolerated degradation establishes regression; an interval that straddles the decision boundary remains inconclusive. An optional absolute subject bound uses the subject interval with the same decision rules, unlike the captured subject-mean check above. Incomplete scheduled trials yield insufficient_evidence without complete-case inference. Across required judge gates, regression takes precedence over insufficient evidence. decision_role: required controls the conjunction of metrics; advisory metrics cannot independently authorize recipient acceptance. Optional reference-label studies assess the judge's usefulness but are not inputs required by runtime replay. See judge statistics.

Verifier-owned replay

For native pack-v1 evidence, the evidence signer cannot supply an accepted aggregate directly. Verification:

reconstructs the canonical schedule and record order;
derives exact-match or normalized-NLL record scores from authenticated facts, or requires an explicitly authorized scorer and replays it twice;
derives paired exact-match counts, exact McNemar probability, and the Newcombe interval, or the normalized-NLL or scorer-extension schedule-resampling interval;
derives a perplexity interpretation only when tokenizer and token-count facts are comparable;
reconstructs canonical paired records;
recomputes side means, point comparison, the selected interval, threshold application, optional count/width and exact-match side-accuracy qualification, and verdict; and
requires canonical equality with the stored report.

The implementation is in build_comparison_report, and independent replay is in verify_comparison_evidence.

Finite-schedule interpretation

For the native comparisons derived above, a pass means that the policy's conservative bound cleared its threshold and any configured count, width, and exact-match side-accuracy requirements passed for the authenticated records and runtime configuration. It does not establish:

equality, equivalence, or non-inferiority in a broader population;
representativeness of the selected prompts or targets;
statistical power for a separately defined hypothesis;
safety, robustness, fairness, or general model quality;
execution attestation from a runtime-image digest; or
immunity to repeated tuning or best-run selection bias.

Where population inference is necessary, define and precommit a sampling and statistical protocol outside InvarLock and retain that evidence separately. Do not extend either paired interval beyond the authenticated schedule without those assumptions.

References

RFC 3629: UTF-8, a transformation format of ISO 10646, Internet Standard, 2003.
McNemar, Q., Note on the sampling error of the difference between correlated proportions or percentages, Psychometrika 12, 153–157, 1947.
Newcombe, R. G., Improved confidence intervals for the difference between binomial proportions based on paired data, Statistics in Medicine 17(22), 2635–2650, 1998.
National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework 1.0, 2023.