Which Answers Changed Behind the Score?

A release reviewer sees accuracy rise from 53.00% to 54.75%. That difference does not say how many previously correct answers were lost, or how many outputs changed without changing the score.

In one retained Qwen3.5 9B deployment comparison, 54 answer strings changed. Only 27 of those changes crossed the exact-match correctness boundary: 17 improvements and 10 regressions. The other 27 changed answers remained wrong. The net gain was seven correct answers across 400 records.

Those counts give a supplier and reviewer different work to do. The net score summarizes the balance; the regressions identify answers to inspect; the changed-but-wrong records expose behavior that the binary score does not distinguish. A practical review preserves all three views.

Reconstruct the four correctness outcomes

The example compares a BF16 checkpoint with its source-derived Q5_K_M GGUF deployment on 400 paired, balanced MMLU-Pro records. These are retained results already introduced in the three-deployment comparison. The question here is how to inspect their constituent answers, not whether a new model run passed.

Start with the canonical report's paired_binary counts. “Correct” below means that the recorded output exactly equals the scheduled expected answer. It is a scoring definition, not an independent judgment of the question or answer key.

The four cells sum to 400. Baseline correctness is 202 + 10 = 212; subject correctness is 202 + 17 = 219. Their difference is seven records, or +1.75 percentage points. The 27 discordant pairs are 10 + 17, not the net difference.

A positive net difference therefore coexists with lost successes. It does not show that every previously correct behavior was preserved. The report's paired 95% interval is [−0.83, 4.32] percentage points, rounded to two decimals. It includes zero, so the point estimate does not demonstrate improvement or establish equivalence.

Count output changes separately

The correctness table cannot reveal whether the two wrong answers were identical. To answer that, join the retained baseline and subject observations to the schedule by record identity and input digest, then compare their output_text values.

For this example, that comparison finds 54 unequal answer strings. The 27 correctness transitions account for half. The remaining 27 sit inside the 171-record both-wrong cell; the other 144 both-wrong records retain the same answer string. This output-change count is descriptive; it is not an additional acceptance rule in the canonical report.

Three records show the distinction directly. These are the first retained examples encountered in schedule order for each listed category, not a representative sample of errors.

The last row contributes zero to each side's accuracy and nothing to the net difference. Yet its output changed. Exact match deliberately treats both wrong strings as failures; it does not measure whether one error is more serious, more plausible, or closer to the expected answer.

“54 changed answers,” “27 changed correctness outcomes,” and “seven net additional correct answers” are consequently compatible descriptions of the same records. They should never be substituted for one another in a release summary.

Turn the counts into a review queue

First confirm the comparison's identity and scoring rule. A record ID alone is not enough if the input changed. Check the ordered schedule, input digests, expected outputs, provider observations, and scoring contract together. The released exact-match rule uses literal string equality; it does not trim, case-fold, or extract an answer before comparison.

Then retain separate lists of regressions, improvements, and both-wrong output changes. Include the record ID, bound input, expected answer, both outputs, and classification. Reconcile the list totals with the report before interpreting individual examples. Do not silently drop failed or missing observations to produce a cleaner table.

Review the regressions against the actual task requirements. A net gain cannot tell a recipient whether a lost success matters more than a gained one. The changed-but-wrong list can also guide error analysis, but calling one wrong answer better requires an explicit criterion and supporting evidence beyond this exact-match score.

Keep that follow-up analysis separate from the original verdict. Selecting examples after seeing their outcomes can help find a hypothesis; it cannot establish a subgroup effect or justify a newly chosen threshold as though it had governed the original comparison. Preserve the original result, then define and evaluate any new criterion explicitly.

This procedure produces an inspectable review queue. It does not diagnose why the outputs changed. The comparison changes artifact representation and runtime together, so the observed transitions do not isolate a causal effect of quantization alone.

Keep the policy result and the inspection result distinct

The retained report passes its declared exact-match policy. That policy requires at least 400 records, a paired interval no wider than 10 percentage points, a lower interval bound of at least −2 percentage points, and at least 20% accuracy on both sides.

It does not require zero regressions or identical output strings. Finding the ten regressions does not contradict its pass. Conversely, the pass does not settle the importance of those ten cases for a particular recipient. The policy thresholds belong to this comparison; they are not general capability requirements.

The report contract specifies which measurements control the verdict. A reviewer can use the retained observations to ask additional questions without implying that those questions were already answered by the policy. Recipient acceptance and organizational deployment approval remain separate decisions.

Limitations

• One curated, text-only comparison supplies the example. Its results do not establish population error rates, a model ranking, or general quantization behavior.
• Exact match follows the retained answer key. Both-wrong output changes do not measure error severity; the three example records are illustrative selections from the retained data.
• The interval includes zero, and artifact and runtime change together. The results do not demonstrate improvement, establish equivalence, or isolate the cause of a changed answer.
• Replay does not rerun inference or prove genuine execution. No latency, memory, throughput, or cost benefit is measured, and a policy pass is not deployment approval.

Sources

Website documentation explains the maintained workflow. This methods note re-extracts answer transitions from the Qwen3.5 9B comparison retained in InvarLock v0.15.0, commit 4f036ffe99cfaa82019c90ea527b8c4291a091b3. The immutable records, report, and receipt preserve the example; independent replay used the release's separately retained reference inputs and did not rerun inference.

• Reports and receipts
• Pairing and replay
• Public evidence guide
• Evidence artifacts
• Three GGUF Deployment Comparisons Under One Review Standard
• Released Qwen3.5 9B canonical report
• Released Qwen3.5 9B paired records
• Released Qwen3.5 9B schedule
• Released baseline observations
• Released subject observations
• Retained Qwen3.5 9B verification receipt
• Released reference inputs
• Released replay implementation