# Reanalyse 400 real MMLU-Pro answer pairs

This package accompanies **Recheck the Evidence, or Rerun the Model?** It uses
all 400 Qwen3.5 9B answer pairs from the published GGUF deployment comparison,
not generated examples. No model or judge is called.

With Python 3.12 or newer:

```sh
python -m pip install invarlock==0.17.0
python reproduce.py --output reproduced
```

Installation can need a network. Reproduction itself is local. Use a new output
directory. The script checks the package hashes, every answer/reference score,
all IDs and order; creates complete captured-results inputs; invokes InvarLock
`evaluate --unsigned --json` for both requests; and invokes `report --html` for
both resulting evidence packs. It checks the original statistics and that all
first-evaluation files remain unchanged after the second evaluation.

## What the run establishes

The original answers give 212/400 correct baseline answers (53%) and 219/400
candidate answers (54.75%): 17 gains, 10 losses and a net +1.75 percentage points.
The paired Newcombe v2 95% interval is −0.8298925546 to +4.3217785714 percentage
points, matching the original report. Both evaluations use literal equality:
no stripping, case conversion, missing-row exclusion or changed reference keys.

The first score policy uses a two-point tolerated regression (0.02 in score
units); the second changes only that value to zero. Both require 400 records,
at most a ten-point interval width and at least 20% candidate accuracy. The
original native comparison also checks a 20% baseline accuracy floor. Captured
policies do not express that baseline check, so this script checks the unchanged
53% baseline separately against the original report's requirement.

InvarLock returns `pass` for the two-point tolerance and `regression` for zero.
Here `regression` means the lower confidence bound fails the requirement; it is
not proof of a negative effect. The interval includes zero. The zero-loss rule
is a retrospective illustration, not a preregistered study endpoint or an
approved deployment requirement. These are new unsigned captured-results packs,
not the original signed native comparison or a replay of all its contracts.

## Contents and provenance

- `answers.csv`: all 400 pairs, including unchanged, wrong and empty answers,
  with original order, IDs, input hashes, reference letters and literal scores.
- `original-report.json`: byte-identical canonical report from the original
  public comparison. `original-runtime-context.json` preserves both runtime
  settings. The baseline used Transformers/Torch on GPU; the Q5_K_M candidate
  used llama.cpp on CPU. This is not an isolated causal quantization experiment.
- `sources.json`: original artifact/observation identities and nine immutable
  source locations at v0.15.0 commit `4f036ffe99cfaa82019c90ea527b8c4291a091b3`.
  This comparison is first in the existing package order; all 400 rows were used.
- Both policies and request files, and `reproduce.py`.
- `retained/`: both executed evidence packs, their HTML reports, complete derived
  inputs inside each evidence pack and the checked result. No keys, signatures or verification receipt.
- `package-manifest.json`: SHA-256 identities of every included file except itself.

The source extraction was checked against the complete public originals before
publication. The existing [GGUF answer-review package](https://www.invarlock.ai/evidence/gguf-answer-review/README.md)
provides an optional source-verification command and all three comparisons.
Hashes identify bytes; this reproduction does not authenticate the original
signer, establish truthful execution or independently verify the original study.

The derived input field contains only the original record ID and prompt hash;
rescoring needs the saved answers and reference letters, not the prompt text.
This download cannot execute the original models from those hashes. Full prompts
remain in the public source schedule. The 400 retained MMLU-Pro questions do not
establish performance on another workload, current service behavior, latency,
resource savings or deployment suitability. The earlier signed report and verdict
remain unchanged.

Download: [Complete reproduction](policy-example.zip). ZIP SHA-256: `776faed46a639f9dad1d3c3588a43525d132b8fa66790041514d1b600c913567`.

Read the [two-point-tolerance report](original-tolerance.html) or [zero-tolerance report](zero-tolerance.html). Both are actual InvarLock 0.17.0 outputs.