A Valid Evaluation Artifact Is Not Yet a Release Decision

A valid report records one comparison. Release gates decide whether its evidence is ready, sufficient, and consistent across the hosts named by the claim.

A typed evaluation artifact is a structured report that names the compared models, runtime, records, metric, and policy. Even when that report parses cleanly, a release reviewer still needs to know whether the evidence is usable, sufficient, and stable across the intended hosts.

The AIware '26 paper studied those questions across 340 runs, seven edit families, four model families, ten seeds, and matched H100/H200 execution. An edit family is one class of model change, such as pruning or weight scaling.

Its clearest operational result was about where to spend the next run. Under the studied workload and policy, repeated seeds stopped changing promote-or-block outcomes before broader edit-family coverage did. The studied stop rule reduced scheduled compute from 17.77 to 1.78 GPU-hours. Host parity also held 20 otherwise-passing candidates at the declared 0.001 tolerance. These figures belong to that workload; they are not universal budgets or thresholds.

Historical study: The paper used InvarLock v0.3.10. The mapping to today’s signed-evidence workflow below is an architectural interpretation, not a study of the current implementation.

Diagram

In the studied 340-run matrix, the seed stop rule used 1.78 GPU-hours against 17.77 for fixed ten-pass evaluation; a separate host-parity gate held 20 otherwise-passing candidates at tolerance 0.001.

Two findings from the studied workload

Seed stopping: 17.77 → 1.78 GPU-hours

Fixed ten-pass evaluation versus the seed stop rule in the published matrix. Additional seed depth did not change the decisions.

Host parity: 20 candidates held

At the declared 0.001 tolerance, parity held 20 otherwise-passing candidates in the augmented H100/H200 matrix.

The 340-run study is workload- and policy-specific. The parity count is a separate result from the budget comparison.

The study separates three practical questions:

1. Is the artifact coherent enough to enter policy analysis?
2. Would more evidence still change the promote-or-block result?
3. Does that result survive a change of execution host?

The paper, Artifact Readiness Gates with Saturation Stop Rules and Host-Parity Admissibility for FM Release Evaluation, was published in the ACM AIware '26 proceedings on July 5, 2026. It uses the historical reports as typed artifacts and applies a separate release-policy layer. The gates consume the reported metrics; they do not rewrite them.

Three gates, three different questions

Scroll horizontally to see every column.

These are not three ways to score the model. They are three checks on whether a release process may act on the available evidence.

The artifact is the input, not the verdict

The report used by the paper grouped three things: model identities, evaluation inputs, and decision metadata. Runtime provenance travelled beside the report for container-backed evaluations.

That surface matters because downstream policy needs more than a floating-point value. It needs to know which comparison produced the value, under what runtime, and whether the surrounding evidence is internally coherent.

Clearing all three gates can support a bounded promote-or-block result under the declared policy. It does not establish task correctness, general safety, or deployment approval.

Gate one: Can this artifact enter policy?

Artifact readiness comes first because malformed or mismatched evidence should not acquire authority merely by reaching a dashboard.

The paper's readiness gate checks report parsing, model identity, baseline identity, runtime payload, and metric finiteness. Error-level findings keep a row out of the policy analysis. Warning-level findings remain visible instead of being flattened into a clean-looking score.

Imagine that the primary metric is present but the runtime payload is missing. The number may be real. It may even be useful during an investigation. But the release process no longer has the complete evidence object it expected, so the stronger claim must stop there.

Readiness asks a concrete question: does the report contain the structured, identified evidence that this policy requires?

Gate two: Would another run change the decision?

A release process also needs to distinguish having evidence from having enough evidence. More repetitions can improve confidence, but a fixed run count is not automatically the best use of compute.

In the paper's 340-run matrix, adding seed repetitions did not change the promote-or-block outcomes, while broader edit-family coverage remained decision-informative.

It suggests a better operational question: under this workload and this declared policy, which next run is still capable of moving the decision?

A saturation stop rule watches decision movement as evidence is added. If another seed no longer changes any relevant outcome, the process can stop spending compute on that dimension. If another edit family still exposes new boundary behavior, breadth remains worth buying.

The rule makes the reason for stopping inspectable. Its result remains conditional on the workload and policy.

Gate three: Does the result survive a host change?

Near a strict threshold, a small host-dependent difference can flip a release result. That makes host parity a policy concern, not a footnote beside the score.

The paper models matched H100/H200 admissibility as a separate promotion condition. A candidate may satisfy the promote rule on one host and still be held because the other host fails it, or because the matched pair exceeds the declared parity tolerance.

When a release claim names both hosts, the team declares an acceptable difference before seeing the result. It then evaluates and records parity as its own policy outcome.

Without that gate, a team can quietly choose whichever host produces the convenient side of a boundary. With it, the release claim has to survive the runtime scope it names.

What the study supports—and what it does not

The paper's useful claims are narrow enough to challenge:

The supported/unsupported split identifies which findings can be reused and which decisions still require separate evidence or authority.

Mapping the study to current evidence

The exact InvarLock files and operator flow have changed since the paper’s pinned implementation. The following mapping is our interpretation of how the study’s separation of responsibilities applies to the current system.

Scroll horizontally to see every column.

Evaluation records the comparison. Verification checks the evidence. External policy decides whether the evidence is admissible and sufficient. Recipient acceptance and deployment approval remain separate decisions.

To follow the transition, read the paper, inspect the v0.13 transaction model, and use the current evidence workflow.

Limitations

• The run-level evidence is limited to the paper's published 340-run matrix.
• Numeric thresholds, parity tolerances, and boundary bands are workload-specific rather than universal constants.
• The study evaluates one toolchain; it does not empirically establish cross-runner replication or downstream task correctness.
• Saturation is conditional on the measured workload and policy. A new model, edit family, metric, host, or threshold can reopen the evidence question.
• Passing all three gates does not establish general safety, compliance, organizational acceptance, or deployment suitability.

Sources

Website documentation explains the maintained workflow. The paper studied InvarLock v0.3.10 in a 340-run matrix; the paper and release record preserve that evidence basis. The mapping to the maintained workflow is an architectural interpretation, not a measurement of a later implementation.

• Documentation
• Reports and receipts
• Evidence artifacts
• ACM DOI record
• OpenReview full text
• InvarLock v0.3.10 release record