Three GGUF Comparisons: Different Regressions, Shared Errors

Five different losses follow a shared correct starting point, while 30 questions keep one wrong answer across all six outputs. Reproduce the findings from unchanged public records.

Published September 7, 2026Updated October 14, 202610 min readInvarLock Team

This card shows the Ministral comparison only. The article also analyzes two Qwen comparisons and shared errors across the three. These benchmark results do not establish production performance or a model ranking.

A model-change review needs to find both newly wrong answers and errors that persist. Three saved GGUF comparisons show why: the deployments lose different previously correct answers, while 30 questions keep the same wrong option across every baseline and candidate output.

The comparisons cover Qwen3.5 9B, Qwen3.8 27B and Ministral 3 8B on the same 400 MMLU-Pro questions. Each compares a BF16 checkpoint with its source-derived Q5_K_M GGUF version. This analysis uses their original public records; it does not rerun models, rank them or isolate an effect of quantization.

Agreement can preserve errors

Across the 400 questions, 92 are wrong in all six outputs: three models, each before and after the change. On 30 of those questions, every output selects the same wrong option according to the original answer key.

What to noticeEach question is counted once. Agreement and stability can preserve an error: the 30 identical wrong answers create no within-model regression to flag. Six related outputs are not six independent trials, and no ensemble or routing strategy was tested.

Of 400 shared questions, 132 are correct in all six outputs, 176 have mixed correctness and 92 are wrong in all six. Among the 92, 30 have one identical wrong option and 62 have differing wrong outputs. — What to noticeEach question is counted once. Agreement and stability can preserve an error: the 30 identical wrong answers create no within-model regression to flag. Six related outputs are not six independent trials, and no ensemble or routing strategy was tested.

A review limited to changed answers would miss those 30 persistent, identical errors. Check absolute quality alongside regressions. The answer key defines correctness here; it has not been independently validated, and these records do not establish error causes or severity.

One deployment's loss list does not cover the others

The comparisons lose ten, seven and four previously correct answers respectively: 21 distinct question IDs, with no overlap. Reviewing only one deployment's losses would therefore miss those in the other two.

A loss is possible only where the baseline was correct. Among the 137 questions all three baselines got right, the GGUF outputs lose two, one and two different questions. All three preserve the other 132.

What to noticeEvery row shown started correct in all three baselines. These are all five rows with a subsequent loss. The selected 137-question subset makes loss eligibility comparable; it does not establish independent failure mechanisms or replace the original 400-question results.

Five different losses among 137 questions all baselines answered correctly: Qwen3.5 loses mmlu_pro_04628 and mmlu_pro_10700, Qwen3.8 loses mmlu_pro_03861, and Ministral loses mmlu_pro_05992 and mmlu_pro_06633. The remaining 132 stay correct in all three. — What to noticeEvery row shown started correct in all three baselines. These are all five rows with a subsequent loss. The selected 137-question subset makes loss eligibility comparable; it does not establish independent failure mechanisms or replace the original 400-question results.

Different losses remain even from this shared correct starting point. With only five losses, their lack of overlap does not establish a general pattern or distinct causes. The complete lists include all 137 questions.

The same question can improve in one comparison and regress in another

Three questions are gains in one comparison and losses in another. These are all three cases:

Question ID
mmlu_pro_04748
Expected
F
Qwen3.5
I → F, gain
Qwen3.8
F → F
Ministral
F → G, loss
Question ID
mmlu_pro_11770
Expected
D
Qwen3.5
A → D, gain
Qwen3.8
D → A, loss
Ministral
D → D
Question ID
mmlu_pro_12199
Expected
D
Qwen3.5
D → B, loss
Qwen3.8
B → D, gain
Ministral
B → B

Table 1: The same question can improve in one comparison and regress in another. Data table with columns: Question ID, Expected, Qwen3.5, Qwen3.8, Ministral
Question ID	Expected	Qwen3.5	Qwen3.8	Ministral
`mmlu_pro_04748`	F	I → F, gain	F → F	F → G, loss
`mmlu_pro_11770`	D	A → D, gain	D → A, loss	D → D
`mmlu_pro_12199`	D	D → B, loss	B → D, gain	B → B

The last two rows show exact reversals between the same two options in the Qwen comparisons. The records establish the transitions, not their causes or a general direction of change under GGUF. Question text, offered choices and answer keys match across the original schedules after removing prompt wrappers.

Three results, each with its own uncertainty

All three comparisons pass their declared exact-match policy. Accuracy below means the share of answers matching the expected letter. The paired effect is GGUF accuracy minus BF16 accuracy; pp means percentage points.

Comparison
Qwen3.5 9B
BF16
53.00%
Q5_K_M
54.75%
Paired effect
+1.75 pp
Paired 95% interval
[−0.83, 4.32] pp
Policy
Pass
Comparison
Qwen3.8 27B
BF16
63.00%
Q5_K_M
64.50%
Paired effect
+1.50 pp
Paired 95% interval
[−0.74, 3.75] pp
Policy
Pass
Comparison
Ministral 3 8B
BF16
44.75%
Q5_K_M
45.25%
Paired effect
+0.50 pp
Paired 95% interval
[−1.12, 2.12] pp
Policy
Pass

Table 2: Three results, each with its own uncertainty. Data table with columns: Comparison, BF16, Q5_K_M, Paired effect, Paired 95% interval, Policy
Comparison	BF16	Q5_K_M	Paired effect	Paired 95% interval	Policy
Qwen3.5 9B	53.00%	54.75%	+1.75 pp	[−0.83, 4.32] pp	Pass
Qwen3.8 27B	63.00%	64.50%	+1.50 pp	[−0.74, 3.75] pp	Pass
Ministral 3 8B	44.75%	45.25%	+0.50 pp	[−1.12, 2.12] pp	Pass

These original report values are rounded to two decimals. All three intervals include zero: the positive estimates do not demonstrate improvement, and passing the policy does not establish equivalence. The reports contain no latency, memory, throughput, energy or cost measurements, so they establish no benefit on those measures.

How to read and reproduce these results

Shared questions do not mean identical input bytes

All three comparisons use the same ordered 400 question IDs and answer key, balanced across 14 MMLU-Pro domains and choices A–J. Within each comparison, baseline and candidate receive the same ordered inputs.

Across comparisons, the two Qwen schedules also have matching per-question input hashes; Ministral uses a different prompt format. Each schedule keeps its own identity. Shared questions do not erase differences in model files, prompts, conversion or runtime. The original deployment guide describes each text-only setup; vision capabilities were not evaluated.

Keep answer validity and runtime context visible

Qwen3.5 has two empty GGUF outputs, both already wrong in the baseline. Qwen3.8 has four, including one previously correct answer. Ministral has no empty strings; one baseline ** becomes a wrong offered option. Every other output selects an offered option. Trimming whitespace or changing case alters none of the 2,400 original scores. The changed-answer article examines this distinction.

All six runtime records specify a one-token output budget. The baselines use Transformers/PyTorch on CUDA; GGUF uses llama.cpp on CPU. Model representation, software and hardware change together. The saved records include no token probabilities or stopping reasons to establish why an answer changed or became empty.

A shared policy makes the passes interpretable

The three original reports use invarlock/comparison-report-v3 and the same four requirements:

Table 3: A shared policy makes the passes interpretable. Data table with columns: Check, Declared requirement
Check	Declared requirement
Paired answers	At least 400
Width of the paired 95% interval	At most 10 percentage points
Lower bound of the GGUF-minus-BF16 interval	At least −2 percentage points
Accuracy of both baseline and GGUF	At least 0.20, or 20%, on each side

The interval widths are 5.15, 4.49 and 3.24 percentage points, and every lower bound is above −2 points. Each comparison has 400 pairs, and all six accuracies exceed 20%. All therefore pass.

Checking accuracy on each side prevents a small difference between two poorly scoring versions from deciding the result alone. The 20% floor belongs to this policy; it is not a general capability or safety threshold. See the report requirements.

Each result retains its own evidence and authority

Each evidence package identifies its model files, runtime, conversion, inputs, metric and policy. A receipt for the original setup does not verify a changed setup. Ministral remains a limited text-only canary example outside the evaluator qualification matrix.

For the September publication, all three packages passed replay with the published 0.15.0 wheel and separately retained release reference inputs. Fresh signed receipts validated, and repeated HTML reports matched byte for byte. Those checks covered saved evidence and arithmetic; they did not prove original execution or representative performance. The public evidence guide explains how reviewers supply their own trusted inputs. A policy pass, a recipient's acceptance and deployment approval remain separate decisions.

Reproduce the original results and new counts

The reproduction download contains 1,200 answer pairs across the three comparisons, three unchanged original reports, the released statistics implementation and analysis code. With Python 3.12 or later, unzip it and run:

python3 reproduce.py --output reproduced

The command rebuilds the original counts, accuracies, paired 95% intervals, exact McNemar statistics and policy passes. It also reproduces answer validity, normalization checks, shared starting points, persistent errors, opposite-direction cases and all four package charts.

Add --fetch-sources to compare the extracted data with 27 immutable public source files: schedules, saved answers, paired scores, runtime receipts and manifests. Neither command runs models or verifies signatures.

This October 14 extension adds descriptive analysis to the original evidence. It leaves the reports and acceptance rules unchanged; repeated inspection supplies no additional observations.

Limitations

These are three curated comparisons on 400 shared questions, with a one-token output budget. Six related outputs are not independent trials. The selected 137-question subset and descriptive overlaps do not establish failure causes, a model ranking, general quantization behavior or vision capability.
All three intervals include zero; the results do not demonstrate improvement or establish equivalence. The reports contain no latency, memory, throughput, energy, or cost measurements.
Replay checks retained evidence and report arithmetic. It does not rerun inference or conversion, or independently establish genuine original execution.
Policy passes remain bound to their transactions. Safety, recipient acceptance, and organizational deployment approval require separate evidence or authority.

Sources

Website documentation explains the maintained workflow. The comparisons and receipts use InvarLock v0.15.0, commit 4f036ffe99cfaa82019c90ea527b8c4291a091b3. The October 14 extensions inspect answer validity, shared starting points, persistent errors and runtime settings from that same evidence without rerunning inference or changing reports. The original signature replay used release-maintained reference inputs held separately from the packs; recipients still choose which identities and policies to trust.