# Reproduce the GGUF answer review This package supports **Which Answers Changed Behind the Score?** and **Three GGUF Deployment Comparisons Under One Review Standard**. It analyzes existing public records; no model, conversion tool or benchmark is invoked. ## Run offline Unzip `reproduction.zip`, open the `gguf-answer-review` directory, then run with Python **3.12 or later**. No third-party packages are required: ```sh python3 reproduce.py --output reproduced ``` The output directory must not already exist. The command checks package hashes, all 1,200 rows, literal-equality scores, outcome categories and three original reports. It reproduces the original accuracies, paired Newcombe v2 95% intervals, exact two-sided McNemar statistics and policy passes, then writes the original `summary.json` and `answer-changes.svg`, plus `review-summary.json` and three new charts. The interval implementation is an unchanged copy of the original released module, so this is a reproduction using the same method, not an independent validation of that statistical method. ## Check the extraction against the public originals This optional command reads 27 public JSON files over the network, each from commit `4f036ffe99cfaa82019c90ea527b8c4291a091b3` (v0.15.0): ```sh python3 reproduce.py --fetch-sources ``` It checks the source hashes in `sources.json`, requires complete ordered joins between each schedule, both observations and paired scores, then compares the result with every answer/option CSV field and unchanged report. Six runtime receipts and their manifests add the settings and observation bindings. The three extracted question bodies, offered choices and keys must match exactly after removing the model-specific wrappers. It does not execute downloaded code. A local checkout of that revision also works without network access: ```sh python3 reproduce.py --source-root /path/to/invarlock ``` Hashes establish file identity, not truthful original execution or authority. These commands do not verify signatures or issue a verification receipt. Use the separately scoped public-evidence workflow for those checks: https://www.invarlock.ai/docs/user-guide/public-evidence ## What the files contain - `answers.csv`: every record from all three 400-question comparisons, in schedule order, including unchanged and unfavorable outcomes. Fields retain model, order, record ID, input digest, expected answer, both exact output strings, original binary scores and the derived category. Empty CSV answer fields mean recorded empty output strings, not missing observations. - `*-report.json`: three byte-identical original canonical reports. - `summary.json`: recomputed counts, original statistics, empty/non-letter outputs, complete loss lists and cross-comparison overlap. - `sources.json`: immutable locations and hashes for all 27 extraction inputs. - `options.csv`: all 400 IDs, answer keys, offered letters and digests of the question/choices text. Prompt bodies are not duplicated. Source verification parses the choice labels and checks that all three question bodies match. - `runtime-context.json`: settings, backend/device and measurement availability from all six source receipts and observations. The one-token output budget is collection context, not evidence of why an answer is empty. - `review-summary.json`: complete lists for the 137 shared baseline successes, their five distinct losses, 92 shared failures, 30 identical wrong answers and all three questions with opposite gain/loss directions. It also records offered option validity and the trim/uppercase score-sensitivity check for every model. - `review.py`: projection, descriptive follow-up and three chart generators. - `answer-validity.svg`, `common-correct.svg`, `shared-errors.svg`: the new figures. - `reproduce.py`: the extraction, analysis and chart code. - `paired_exact_match.py`: unchanged released statistics code; its Apache 2.0 license is in `LICENSE.txt`. Original source: https://github.com/invarlock/invarlock/blob/4f036ffe99cfaa82019c90ea527b8c4291a091b3/src/invarlock/paired_exact_match.py - `answer-changes.svg`: the figure used in the revised GGUF article. - `package-manifest.json`: identities of the package members, excluding itself. ## Definitions and interpretation `gain` means wrong → correct; `loss` means correct → wrong; `preserved` means correct on both sides. `changed_wrong` means unequal strings that both fail literal exact match; `same_wrong` means the same wrong string. No trimming, case folding, answer extraction, empty-output exclusion or rescoring is applied to the original results. A separate sensitivity calculation tries surrounding whitespace trimming and uppercase conversion: none of the 2,400 scores changes. It does not test arbitrary alternative scorers or identify model reasoning. Across comparisons, the 1,200 rows use 400 shared question IDs and answer keys; they are not 1,200 distinct questions or independent samples. The Qwen schedules share input digests; Ministral uses a different prompt format. Full schedule identities remain separate. The disjoint loss lists describe these records only; they do not establish independent failure mechanisms or general model ranking. Two worked examples in the changed-answer article paraphrase the first loss and first gain in Qwen3.5 schedule order, preserving their original selection rule: - `mmlu_pro_00111`: 50,000 free retained earnings minus a 10,000 increase in reserves gives 40,000 (A); the subject output F identifies 55,000. - `mmlu_pro_00375`: (300 − 174) / 300 = 42% (B); the baseline output H identifies 40%. These calculations explain the answer-key distinction, not the models' reasoning. The complete prompts and choices remain in the pinned public schedule identified in `sources.json`; they are not duplicated here. Two Qwen3.5 subject outputs (`mmlu_pro_01020`, `mmlu_pro_09998`) are empty while their baselines also fail the answer key. Exact match assigns zero to both. Their records have status `ok`; these fields do not explain why text is empty. ## Scope All analyses use the original 400 records per deployment. They are descriptive follow-up checks, not newly registered endpoints or changed acceptance rules. No repeated-run stability, causal quantization effect, resource savings, error severity, representative workload coverage or deployment suitability is inferred. The original intervals include zero. Results, thresholds and verdicts are unchanged. The shared-correct subset is selected using all three baseline outcomes. Its five disjoint losses do not establish independent failure mechanisms. The six outputs per question are related observations, not independent trials, repeats or a tested ensemble. Shared wrong answers follow the original key, which this analysis does not independently validate. No stopping reason or token probability is retained in the observations used here. No cause is inferred from hardware, runtime, artifact format or the one-token budget.