Recheck the Evidence, or Rerun the Model?

A candidate can answer more questions correctly and still fail a stricter evaluation requirement. In 400 saved MMLU-Pro question pairs, a Qwen3.5 9B candidate answered 219 correctly against the baseline's 212. Reanalysing those answers with a tighter rule changes the policy verdict, even though no answer or score changes.

That is a useful distinction when someone says, “Run the evaluation again.” A clearer report, a stricter requirement and an updated service need different work. The question is which facts are missing: do the saved records already answer it, or does it require new observations?

InvarLock's evaluation workflow separates producing evidence, verifying it, and presenting it. Those established roles help decide which part to repeat.

Start with what changed

These steps can be combined. A new collection still needs comparison and verification. A new report does not perform all the work of an independent verifier.

A new report needs no new answers

The reporting command makes retained evidence readable. It checks the bundle's internal integrity and embedded signature before rendering, but it does not establish that your organization has authorized that signer or validate the separate record of a verification check.

The HTML is therefore a view of the signed comparison. A different renderer can change how the result is explained without changing the evidence, scores, or official verdict. Keep generated views outside the evidence directory so that presentation files do not alter the signed inventory.

The v0.17 release article demonstrates this with a newer report view of an earlier saved comparison. The original model was not rerun. If the question is how to understand that result, begin with its records and report rather than collecting different answers.

A new receipt checks the saved comparison

Verification checks the evidence's signatures and contents, then recomputes its decision under the declared rules. The reviewer must separately establish the intended inputs, requirements and authorized signer; copying those expectations from the submission would make the check circular. A verification receipt is the signed record of this check.

The CPU example introduced in v0.15 makes this distinction tangible. We ran its released v0.17.0 version after installation with operating-system network access denied. It returned Decision: pass, issued a new signed verification receipt, and rendered HTML. Rendering the same evidence again produced identical HTML bytes.

That check used one saved test example and the approved-input and signer settings supplied for the demonstration. It made no model calls. It shows that this saved comparison can be checked locally; it does not show that other workloads will pass, prove how the original outputs were produced, or establish current service behavior. Identical HTML concerns this input and renderer, not general model reproducibility.

A new receipt records another verification of the evidence. It does not change when the original outputs were collected. The receipt alone does not prove when it was issued. If your organization requires recent evidence, it needs trusted timing records and explicit rules about age.

A changed threshold can reuse the same observations

A comparison policy contains the technical requirements applied to the records. Changing it creates a new analysis. It must not silently change the meaning of an old signed result.

The published GGUF comparison provides a real example: all 400 Qwen3.5 9B answer pairs from MMLU-Pro, including wrong and empty answers. We checked the public extract against its original records, then ran InvarLock 0.17.0 on the saved answers and reference letters. No models or judges were called.

The reanalysis reproduced the original 53% baseline accuracy and 54.75% candidate accuracy. Seventeen answers changed from wrong to correct, and ten changed the other way. The resulting gain is 1.75 percentage points, with a paired 95% interval of −0.83 to +4.32 points. The interval, not just the positive average, matters to the requirement.

The published tolerance allows a loss of up to two percentage points: the interval's lower end must stay at or above −2. It does. For a second evaluation, we changed only that tolerance to zero. The same lower end is below zero, so the stricter requirement is not met.

Diagram

The same paired 95% interval, from minus 0.83 to plus 4.32 percentage points, clears the minus 2 point requirement but crosses the stricter zero point requirement. The point estimate remains plus 1.75 points in both evaluations.

Same answers. A stricter requirement.

Unchanged observations

All 400 real Qwen3.5 9B answer pairs: 53% baseline accuracy, 54.75% candidate accuracy. Difference: +1.75 percentage points; paired 95% interval: −0.83 to +4.32 points.

Two-point tolerance: pass

The interval’s lower bound is above −2 percentage points. The recreated score requirements pass.

Zero-loss illustration: regression

The same lower bound is below zero. InvarLock returns regression under this stricter rule; the interval does not prove the candidate is worse.

Actual unsigned reanalysis with no new model calls. The zero-loss rule is a retrospective illustration, not an approved deployment requirement. The original signed study remains unchanged.

InvarLock returned pass and regression in two separate evidence directories. Here, regression means the evidence fails the stricter rule; it does not prove that the candidate is worse. The interval includes zero. The zero-loss scenario was chosen for this explanation after the results were known; it was not an original study requirement.

This distinction changes the next step. Reanalysis answers whether the existing evidence meets the stricter requirement. It cannot add information or narrow the interval. If that remaining uncertainty matters, more appropriately chosen paired cases may be needed. Replaying these same 400 records cannot supply them.

Both evaluations and their reports ran with network access denied. All first-evaluation files remained unchanged after the second. These are new unsigned reanalysis artefacts: they have no digital signature and do not replace or independently verify the original signed study. We recreated its score requirements in the captured-results format; the original baseline accuracy floor was checked separately because that format does not express it. The package notes specify every requirement and the source conversion.

Read the actual two-point-tolerance report and zero-tolerance report, or download the complete reproduction. It includes all 400 answer pairs, source identities, both policies, both evidence packs, both HTML reports and the checker. With Python 3.12 or newer and invarlock==0.17.0 installed, unzip it and run:

python reproduce.py --output reproduced

The checker invokes InvarLock twice, regenerates both reports, matches the original statistics and confirms that the first evaluation remains unchanged. The input identifiers and prompt hashes are enough to match the saved records; they do not provide prompt text for a new model run.

For a signed handoff, sign the new comparison and obtain a new verification receipt using separately approved inputs, requirements and signers. Preserve the earlier evidence and explain why the policy changed. A threshold selected after seeing the answers must not be presented as a requirement fixed before collection.

This also differs from a recipient changing its own acceptance rules. The recipient checks article explains how an organization can decline an authentic historical pass without rewriting its technical verdict.

Saved records cannot supply facts they never contained

Reusing records depends on what the new question requires:

Using another model to judge the answers adds a distinction. Replaying saved ratings does not ask the judge to assess the answers again. A different rubric may require new judge ratings while keeping the candidate answers fixed. That repeats the judging step, not necessarily the candidate model's generation. The captured-results guide describes the required records.

Public reproduction downloads have a boundary too. The coding study separates arithmetic checks over published extracts from signature verification of its full signed archive. Recalculating a CSV does not authenticate its private source records.

New behavior needs fresh collection

If the question is whether a changed model, prompt, retrieval source, tool setup, or hosted service still meets your requirements, old outputs do not measure that change. Collect records for the setup you intend to assess, identify its cases and configuration, and record when the outputs were collected.

A frozen baseline can remain useful even when its old endpoint is unavailable. Be explicit that the baseline consists of saved outputs; do not describe it as a fresh execution of that service. The hosted-service workflow keeps the earlier assessment intact while giving each later comparison its own records and receipt.

Before repeating expensive work, state what changed and what evidence would answer the new question. Re-render when the view is the problem, reverify when the check is needed, reanalyse when the rules change and the records suffice, and collect again when the required behavior has not been observed.

Limitations

• The reanalysis uses one existing 400-question MMLU-Pro comparison, with literal exact match and the original paired interval method. It adds no model runs, latency measurements or evidence about another workload. A separate saved, signed test example demonstrates local verification.
• The original runs allowed one output token per answer. The baseline used Transformers/PyTorch on GPU; the quantized candidate used llama.cpp on CPU. Model representation, software and hardware changed together, so these results do not isolate a causal effect of quantization.
• Replay checks retained evidence against its declared rules. It does not prove genuine original execution, representative cases, correct judge ratings, or unchanged service behavior.
• Record reuse requires complete inputs in a format the evaluator supports. Missing measurements cannot be repaired by a new report or signature.
• Technical verification, recipient acceptance, and deployment approval remain separate; changing one does not silently update the others.