Which Answers Changed Behind the Score?

A release reviewer sees accuracy rise from 53.00% to 54.75%. That difference does not say how many previously correct answers were lost, or how many outputs changed without changing the score.

In one retained Qwen3.5 9B deployment comparison, 54 answer strings changed. Only 27 of those changes crossed the exact-match correctness boundary: 17 improvements and 10 regressions. The other 27 changed answers remained wrong. The net gain was seven correct answers across 400 records.

The net score summarizes the balance. Inspecting the offered choices adds a useful finding: every lost success selects a valid but wrong option. Simple whitespace or case differences explain none of the scores.

Looking inside those groups adds another distinction: two previously wrong answers become empty outputs. Their scores stay at zero, so the accuracy comparison does not reveal that loss of an answer. The downloadable checks reproduce every category from the original records, with no new inference or changed results.

Reconstruct the four correctness outcomes

The original comparison uses a BF16 checkpoint and its source-derived Q5_K_M GGUF deployment on 400 paired, balanced MMLU-Pro records.

Start with the canonical report's paired_binary counts. “Correct” below means that the recorded output exactly equals the scheduled expected answer. It is a scoring definition, not an independent judgment of the question or answer key.

Diagram

Four correctness outcomes: 202 preserved successes, 10 regressions, 17 improvements, and 171 failures on both sides. Within the 171, 27 answer strings changed and 144 stayed the same.

Four outcomes across 400 paired answers

202 correct → correct

Preserved exact-match successes.

10 correct → wrong

Regressions to inspect.

17 wrong → correct

Improvements to inspect.

171 wrong → wrong

27 changed their answer string; 144 kept the same wrong string.

Retained Qwen3.5 9B BF16-to-Q5_K_M GGUF comparison. There are 27 correctness changes and 54 changed answer strings, with a net gain of 7 correct answers. The paired 95% interval includes zero; these counts do not demonstrate improvement or measure error severity.

The four cells sum to 400. Baseline correctness is 202 + 10 = 212; subject correctness is 202 + 17 = 219. Their difference is seven records, or +1.75 percentage points. The 27 pairs whose correctness changed are 10 + 17, not the net difference.

A positive net difference therefore coexists with lost successes. It does not show that every previously correct behavior was preserved. The report's paired 95% interval is [−0.83, 4.32] percentage points, rounded to two decimals. It includes zero, so the point estimate does not demonstrate improvement or establish equivalence.

Count output changes separately

Correctness alone cannot reveal whether the two wrong answers were identical. To add the answer-string detail shown in the matrix, join the retained baseline and subject observations to the schedule by record identity and input digest, then compare their output_text values.

The 54 changed strings split equally between correctness transitions and changes within the both-wrong cell. This descriptive count is not an additional acceptance rule.

Three records show the distinction directly. These are the first retained examples encountered in schedule order for each listed category, not a representative sample of errors.

• Record ID

  mmlu_pro_00111

  Expected

  A

  Baseline

  A

  Subject

  F

  Classification

  Correct → wrong

• Record ID

  mmlu_pro_00375

  Expected

  B

  Baseline

  H

  Subject

  B

  Classification

  Wrong → correct

• Record ID

  mmlu_pro_00166

  Expected

  B

  Baseline

  A

  Subject

  J

  Classification

  Changed answer, both wrong

The last row changes its answer while contributing zero on both sides. Exact match does not say whether one wrong answer is more serious or closer to the expected answer.

What two changed answers actually mean

The first loss and first gain above are simple arithmetic questions. Their saved prompts make the answer-key distinction understandable without guessing how the models reasoned.

• Loss, mmlu_pro_00111: free retained earnings start at 50,000. Increasing the contingency reserve by 10,000 leaves 40,000 free. The baseline selects A, the 40,000 option; the subject selects F, the 55,000 option.
• Gain, mmlu_pro_00375: a price falls from 300 to 174. The discount is (300 − 174) / 300 = 42%. The baseline selects H, 40%; the subject selects B, 42%.

These are the original first-in-schedule examples, not an arithmetic subgroup assessment. The calculations explain the expected answers; the final outputs do not reveal the models' reasoning. Readers can inspect the full prompts and choices in the original schedule.

A valid choice can still be a regression

The schedule gives the actual offered options for each question. They vary from three to ten choices, so checking for any A–J letter is insufficient. All ten lost successes select another option that was actually offered. All 17 gains also select an offered option; none is a formatting repair.

Diagram

Among 54 changed Qwen3.5 answers, all ten losses and 17 gains select an offered option. Of 27 changed-but-wrong answers, 25 select an offered option and two are empty.

All ten lost successes select another offered option

10 losses

All ten correct-to-wrong changes select an actual offered option. None is empty.

17 gains

All seventeen wrong-to-correct changes select an offered option.

27 changed, still wrong

25 select an offered option; two are empty. Their scores remain zero.

54 changed Qwen3.5 strings. Whitespace trimming and case normalization change none of the 800 original scores. These checks do not identify failure causes.

As a separate sensitivity check, trimming surrounding whitespace and converting letters to uppercase changes none of the 800 original scores. The published scores remain literal exact matches. This rules out those two formatting differences as an explanation of the losses; it does not reveal how the models selected their answers.

A wrong answer and no answer can have the same score

Two of the 27 changed-but-wrong records have an empty subject output:

• Record ID

  mmlu_pro_01020

  Expected

  G

  Baseline output

  E

  Subject output

  Empty string

  Exact-match score

  0 → 0

• Record ID

  mmlu_pro_09998

  Expected

  J

  Baseline output

  B

  Subject output

  Empty string

  Exact-match score

  0 → 0

The other 25 changed-but-wrong records move between distinct offered choices. The two empty outputs are a separate change hidden inside unchanged failure scores.

Both empty outputs are retained observations with status ok, not absent records. That status does not guarantee a usable answer, and these records do not establish why the output is empty. No records are excluded or rescored. If an application needs an answer on every request, answer availability deserves its own review alongside accuracy; this retrospective observation does not add a requirement to the original policy.

The limits of a one-token comparison

Both runtime receipts specify a one-token output budget, with a prompt asking for exactly one option letter. This bounds the finding to constrained choice generation. The saved observations contain neither reasoning traces nor token probabilities, and do not identify a stopping reason for empty text. The budget is relevant context, but it does not establish why these two outputs are empty.

Artifact representation, execution software and hardware change together: the baseline uses Transformers/PyTorch on CUDA, while the GGUF side uses llama.cpp on CPU. The retained runtime settings preserve those settings. These records cannot isolate quantization, the runtime or hardware as the cause of an answer change.

Turn the counts into a review queue

Keep gains, losses and changed-but-wrong outputs in separate lists, with their input identities and expected answers. Reconcile the totals with the original report, then review the losses against the recipient's requirements. A net gain cannot tell a reviewer whether a lost success matters more than a gained one.

The downloadable checker performs the complete ordered join and literal string comparison: no trimming, case folding, answer extraction or record dropping. The companion comparison also examines errors that remain unchanged across deployments. A review queue limited to changed answers would miss those persistent failures.

Keep the policy result and the inspection result distinct

The retained report passes its declared exact-match policy. That policy requires at least 400 records, a paired interval no wider than 10 percentage points, a lower interval bound of at least −2 percentage points, and at least 20% accuracy on both sides.

It does not require zero regressions or identical output strings. Finding the ten regressions does not contradict its pass. Conversely, the pass does not settle the importance of those ten cases for a particular recipient. The policy thresholds belong to this comparison; they are not general capability requirements.

Follow-up interpretation does not change those rules. Recipient acceptance and deployment approval remain separate decisions.

Reproduce the review from public records

The reproduction guide and download include all 400 Qwen3.5 records, including unchanged answers, and the corresponding records for the other two deployments. The CSV preserves exact strings, expected answers, input digests, original scores and categories. A separate 400-row file retains each question’s offered letters and question-body digest, without copying benchmark prompts.

With Python 3.12 or later, unzip the package and run:

python3 reproduce.py --output reproduced

The offline command rebuilds the categories, original statistics, validity checks and all four package charts. It uses the released interval implementation. The optional --fetch-sources flag checks the projections against 27 immutable public source files, including the six runtime receipts and their manifests. Neither command runs models.

This October 14 analysis extends the inspection of the original evidence. It preserves the reports, thresholds and verdicts. The new commands check records and arithmetic; they do not repeat the earlier signature-verification replay or establish genuine original execution.

Limitations

• One curated, text-only comparison supplies the example. Its results do not establish population error rates, a model ranking, or general quantization behavior.
• Exact match follows the retained answer key. The worked examples are selected cases, not a subgroup estimate; the records do not reveal internal reasoning, error severity or the cause of empty output.
• The interval includes zero, and artifact and runtime change together. The results do not demonstrate improvement, establish equivalence, or isolate the cause of a changed answer.
• Replay does not rerun inference or prove genuine execution. No latency, memory, throughput, or cost benefit is measured, and a policy pass is not deployment approval.