A Signed Score Is Not Yet Decision Evidence

You have a signed score, but the verifier cannot use it to decide whether a model change passes. Saving a larger log or signing the score again may look like progress. Whether either helps depends on what is missing.

The retained MLflow and Garak examples stop for different reasons. MLflow's saved aggregate lacks the individual ordered records. Garak's retained output uses replay semantics that the examined contract does not support. One gap concerns missing measurements; the other concerns how to interpret them.

In InvarLock, that difference is the line between decision evidence and an authenticated observation. Decision evidence is replayable material that an authorized contract may use. An authenticated observation is preserved context that cannot vote on the verdict.

Diagnose that gap before collecting more data. A third case—an authenticated numerical diagnostic—can have intact measurements and still lack a justified decision rule.

The fork is independent replay

For the retained exact-match qualification surface, replay authority requires complete ordered records. Each record is bound to the schedule, input, output, reference, and status. InvarLock recomputes the score instead of trusting the evaluator's aggregate claim.

Diagram

Authenticated evaluator output enters paired policy only when complete ordered records can be independently replayed under an authorized contract; other output remains observation-only context.

Which output may affect policy?

Authenticate the output

Check the signature or independently pinned digest and the bound identities.

Complete, authorized records

Independently recompute the score and interval. The declared contract may then use the result in paired policy.

Otherwise: retain context

Preserve the observation for review. Missing records or unsupported replay semantics cannot determine the verdict.

Observation-only describes authority under this contract, not whether the output is useful.

Authentication preserves the output in either path. Only the complete, supported records enter paired policy; the other path keeps useful context available to the reviewer.

MLflow and Garak stop for different reasons

An aggregate cannot tell a reviewer which questions were answered correctly, whether records are missing, or how baseline and candidate outcomes pair up. For the MLflow example, the remedy begins with retaining the complete ordered records. If those were never saved, the aggregate alone cannot recover them.

For Garak, retaining more output in the same unsupported form does not define how another verifier should score it. The scoring and replay semantics need an explicit contract and validation before those records can affect a verdict. This article does not claim that such a contract has been implemented.

Both integration paths are maintained, and both outputs remain useful for provenance and review. “Observation-only” is an authority classification, not a failure label. The full support, replay, and signed-journey matrix is covered in An Evaluator Can Be Supported Without Being Decision-Ready.

There is no command-line switch that promotes an observation into authoritative records. When a caller requires replay authority, InvarLock rejects the observation-only import.

Authenticated diagnostics stay diagnostic

The standalone diagnostics package makes the same separation explicit for numerical context. Spectral, random-matrix, and variance summaries can be emitted as canonical JSON and attached to a signed evidence transaction.

Strict verification can then establish that the observation is canonical, bound to the comparison, and unchanged. A report can display it next to the policy result. Those are meaningful properties.

They still do not turn the diagnostic into a calibrated threshold.

If a diagnostic suggests a useful future rule, the team must define its semantics, calibrate its threshold, and review its failure modes. The approved rule then belongs in a new versioned policy and transaction; it cannot change the meaning of an old receipt.

The SPDX observation example applies the same rule to software-supply-chain metadata: it checks the declaration’s artifact binding while recording full schema and semantic validation as not_evaluated. The declaration remains context.

Match the next step to the missing evidence

These are requirements for future decision evidence, not claims that the retained observation-only examples already meet them. Re-signing the existing output changes none of these gaps.

Preserve context without granting a verdict

Write the follow-up as a concrete deliverable: the missing ordered records, a validated replay contract, or a calibrated rule with an approved decision meaning. Keep the original observation alongside that work so the earlier result remains inspectable.

Some observations should stay as context. The SPDX declaration can help a reviewer identify the supplied package without becoming an accuracy measurement. Promotion into paired policy is useful only when the measurement has a justified role in that decision.

Limitations

• The retained matrix demonstrates one deterministic exact-match surface, not every task or evaluator feature.
• Other replay semantics, including non-deterministic judges and human scoring, need separately defined evidence and authority contracts.
• Authentication does not establish correctness, representativeness, safety, compliance, or deployment suitability.