Recorded comparison with signed evidence

Ministral 3 8B: BF16 → Q5_K_M GGUF

400 paired MMLU-Pro records

Opening this report does not run verification or authorize a release.

Scope and interpretation

Authored commentary · Fictional client scenario · 24 October 2026 · Unsigned

Northstar Learning and Maya Chen are fictional. The business context, rationale, recommendation and organizational decision below illustrate a completed handover; measurements and linked evidence come from the retained comparison.

Illustrative decision: approved for a supervised trial. Maya Chen, Learning Systems Lead at Northstar Learning, approves a two-week internal quiz-authoring trial with five staff reviewers. Recommendation: proceed with human review of every answer. The recorded benchmark policy passes, with 4 regressions and 219 of 400 answers missed. Next: authenticate the retained evidence, then evaluate 200 internal practice questions before the release review.

Intended use and scope

Northstar Learning wants staff to draft answers for internal training quizzes on a locally hosted model. Staff check every answer against an approved answer key before publication. This handover uses exact-match accuracy on 400 paired MMLU-Pro cases as an initial regression screen for the proposed trial.

Change assessed

Ministral 3 8B: BF16 → Q5_K_M GGUF. The baseline uses Hugging Face Transformers; the candidate uses llama.cpp. Both use the same 400 paired MMLU-Pro cases, so this compares the recorded model and runtime configurations together.

Scenario purpose: explore a smaller model artifact for local hosting. Memory use, latency and operating cost will be measured during the trial.

Scoped technical result

Recorded result: PASS under this example’s policy. Exact-match accuracy was 44.75% for the baseline and 45.25% for the candidate (+0.50 percentage points). 95% interval: −1.12 to +2.12 percentage points. The measured accuracy is slightly higher, but the interval includes zero. This supports the allowed-loss check, not a claim of improved accuracy.

At least 400 pairs, a lower effect bound of −2 percentage points, an interval no wider than 10 percentage points, and at least 20% accuracy on both sides. These are this example’s requirements, not recommended defaults.

Why these 400 cases

Documented selection facts: the retained request identifies 400 selected MMLU-Pro test records under the split name test-balanced-400. The schedule fixes the same cases for both configurations.

Illustrative rationale: Northstar uses a reproducible first screen before committing staff time to internal questions. The 400-pair minimum bounds the work; the interval-width check limits imprecision. This scenario rationale is separate from the historical selection facts.

Threshold rationale

Illustrative rationale: for a trial with mandatory answer checking, Maya accepts the recorded 2-percentage-point allowed loss as a screening tolerance. The lower effect bound must clear −2 points; the observed bound is −1.12. The 10-point interval-width ceiling limits imprecision, while the 20% accuracy floor catches a severely degraded configuration. Staff-reviewed internal questions determine fitness for quiz authoring.

Important failures

4 of 400 cases regressed: the baseline matched the expected answer and the candidate did not. 6 cases improved; 175 matched on both sides; 215 matched on neither. The candidate missed 219 of 400 answers despite the passing policy result. These counts describe answer matching, not the severity of each error.

Scenario response: reviewers examine all four regressions and classify missed answers by topic and consequence. Mandatory answer-key review remains in place throughout the trial.

Limitations

This is a materialized GGUF comparison on a fixed public benchmark. It does not establish performance on customer traffic or preservation of every model capability. The trial will separately assess quiz relevance, error severity, instruction following, safe handling of internal material, latency, memory and cost. The accuracy interval includes zero, so improved accuracy is not established.

Acceptance and approval

Technical result — retained evidence
The recorded checks pass for this policy and these cases.
Recipient-policy acceptance — evidence status
The retained receipt belongs to the source example. Independent recipient verification remains the trial’s entry check: Maya supplies approved identities, policy and trust inputs and retains the resulting receipt.
Organizational decision — fictional example
Approved on 24 October 2026 by Maya Chen, Learning Systems Lead: a two-week, five-reviewer trial with mandatory answer-key review. Wider rollout requires the trial’s release review.

Next action and owner

Owner: Maya Chen. Complete independent verification, then assign two reviewers to score 200 internal practice questions. Retain paired answers, adjudicated errors and operating measurements. After two weeks, review error severity, latency, memory and cost; record the rollout decision and archive the trial evidence.

InvarLockRuntime evaluation evidence

Recorded policy result

Policy satisfied

The uncertainty interval is within the required range, and the other configured checks passed. The subject matched the expected answer in 181 of 400 cases, compared with 179 for the baseline. Accuracy changed from 44.75% to 45.25%, a change of +0.5 pp.

What was compared

Recorded fieldBaselineSubject
ArtifactDiffershf://mistralai/Ministral-3-8B-Instruct-2512-BF16@f6fae9795746f63c9be8344932f01275f3c63734derived://mistralai/Ministral-3-8B-Instruct-2512-BF16@f6fae9795746f63c9be8344932f01275f3c63734#llama.cpp-b10015-q5_k_m@sha256:b2c56b655b15577c30734b28b820d054e43df06c7712b74b9c170bff5f10f253
Model IDDiffersmistralai/Ministral-3-8B-Instruct-2512-BF16gguf-sha256-b2c56b655b15577c30734b28b820d054e43df06c7712b74b9c170bff5f10f253.gguf
ProviderDiffershf_transformersllama_cpp
Additional recorded context (10)

Additional fields

Workflow
Runtime execution
Task
text_causal
Paired records
400
Schedule digest
sha256:9ac24cf5e4b904197117fa8d0d9661d88fdf0dae1cc39ddfa711aa820de60fad
Dataset
TIGER-Lab/MMLU-Pro/ministral3-instruct
Dataset split
test-balanced-400
Dataset source SHA-256
c3d83209d6f36023f0a5aef5ee9be895891cc66ecc1b7196e83227558a38fade
Dataset source format
jsonl
Selected records
400
Selection limit
Not specified

Results and requirements

Exact-match accuracy

Scope: All paired records

Policy satisfied

The uncertainty interval is within the required range, and the other configured checks passed.

Baseline
44.75%179 of 400 matched
Subject
45.25%181 of 400 matched
Change
+0.5 ppPaired 95% CI: -1.12 to +2.119 pp
Observed pairs
400
EstimatePaired 95% confidence interval (Newcombe hybrid score) Policy thresholdNo changeAllowed change region

How to read this comparison

Change
The marker shows the observed change between subject and baseline. A change in percentage points (pp) is the subject percentage minus the baseline percentage.
Interval
The bar shows uncertainty around that change. Its left endpoint is the lower bound; its right endpoint is the upper bound. The interval level does not tell you how often model runs would pass or fail the policy.
Policy rule
The paired lower bound (left endpoint) must be at least the policy minimum. The whole interval must lie on the allowed side of the threshold for this bound check to pass.
No change
The dashed line marks no change. An interval crossing it includes changes in either direction. A worse observed score describes this run; policy rejection alone does not prove worse performance beyond these cases.
How this interval was calculated

Baseline and subject are paired on the same 400 cases. Newcombe hybrid score uses their paired match outcomes to form a nominal 95% confidence interval for the accuracy change.

Both matched: 175; baseline only: 4; subject only: 6; neither matched: 215.

Change is subject accuracy minus baseline accuracy in percentage points, not relative percent change. The 95% confidence level describes the interval method, not the share of correct answers.

This exact-match method uses a fixed 95% confidence level; the allowed-loss threshold is a separate policy choice. Under suitable sampling and independent-pair assumptions, the method aims for intervals to cover the underlying accuracy difference in about 95% of repeated studies. This is not the probability that the subject exceeds the allowed loss, and these cases are not automatically representative of production.

Decision checks
CheckObservedRequiredResult
Paired lower boundThe lower bound must meet the allowed change, not just the point estimate.-1.11994 pp>= -2 ppPassed
Record countThe comparison must include the configured minimum number of paired records.400>= 400Passed
Interval widthThe interval must be narrow enough for the configured precision requirement.3.23886 pp<= 10 ppPassed
Baseline accuracyEach side must meet the configured absolute accuracy floor.44.75%>= 20%Passed
Candidate accuracyEach side must meet the configured absolute accuracy floor.45.25%>= 20%Passed

Next steps

  1. Review the decision checks and their requirements.
  2. Run invarlock verify with your independently supplied trust profile and receipt destination to create the signed acceptance or rejection receipt.
  3. Keep this report alongside the original immutable evidence and the separate verification receipt.
Scope and limitations
  • This report summarizes the evidence bundle. Rendering checks its integrity and embedded signature; independent acceptance requires verification with your own trust inputs.
  • Results apply to the recorded cases, metric and policy. A pass does not establish general model quality, safety or representative production performance.
  • The interval describes paired binary outcomes under its stated method. Population interpretation requires an appropriate sampling design.

Evidence details

Identities, methods and exact recorded values

Comparison and policy identities
Comparison
comparison-4472c601a80f319859c4bf0c505c399c
Metric
exact_match
Policy
sha256:48e929519390c41e91c5e907e7e3f3d6d0b26e5ac4345136c4f6b85244b191d9
Evidence signer
sha256:e1f2aab12ee457abad9b0c74b1b9dcd0ce0429e541c15f17b814071b7fd9abab
Paired outcome analysis
{
  "baseline_fail_subject_pass": 6,
  "baseline_pass_subject_fail": 4,
  "both_fail": 215,
  "both_pass": 175,
  "discordant_pairs": 10,
  "effect_size_confidence_interval": {
    "confidence_level": 0.95,
    "lower_pp": -1.119943249438997,
    "method": "newcombe_hybrid_score_paired_v2",
    "upper_pp": 2.118915687037841
  },
  "effect_size_pp": 0.5,
  "mcnemar_exact_two_sided_p_value": 0.75390625
}
Authenticated observations
[
  {
    "authority": "observation",
    "bindings": {
      "artifact_digests": {
        "baseline": "sha256:2bba1d6c1331980047b18d9b00da21d997dc9ca26a183434e512bfc348b2e891",
        "subject": "sha256:fa3ab58b49a8e08b4901a3f990c0d0a401ec7930bd90a632f564ff7f71f1b1a6"
      },
      "comparison_id": "comparison-4472c601a80f319859c4bf0c505c399c",
      "policy_digest": "sha256:48e929519390c41e91c5e907e7e3f3d6d0b26e5ac4345136c4f6b85244b191d9",
      "schedule_digest": "sha256:9ac24cf5e4b904197117fa8d0d9661d88fdf0dae1cc39ddfa711aa820de60fad"
    },
    "format": "invarlock/evidence-observation-v1",
    "kind": "artifact_transformation",
    "observation_id": "ministral3-8b-bf16-to-gguf-q5-k-m",
    "payload": {
      "conversion": {
        "output": {
          "byte_length": 16987563264,
          "filename": "Ministral-3-8B-Instruct-BF16.gguf",
          "sha256": "b6dc36267813cef3a367b3e23528f5e367db5713ca6f2dd1f1a40a3eb1344ada",
          "type": "BF16"
        },
        "runtime_image_digest": "sha256:b905cf3b278531333d47fc80ba5b110793f80717fc2acc526e8dd291ae8f3301",
        "tool": {
          "name": "llama.cpp/convert_hf_to_gguf.py",
          "source_commit": "12127defda4f41b7679cb2477a4b0d65ee6a0c8f",
          "source_sha256": "5ab75e394f4c71425ecce64a213dab3b8e3e9cfe0f19d0dcda4d5a4f7733da83",
          "source_tag": "b10015"
        }
      },
      "format": "invarlock/example-gguf-deployment-transformation-v1",
      "quantization": {
        "runtime_image_digest": "sha256:aa5d7b75a807e786175e9edf9927cbbc0bb3099bcc2d95fe07e56f80bbfeedb6",
        "tool": "llama-quantize",
        "type": "Q5_K_M"
      },
      "source": {
        "checkpoint_tree_sha256": "sha256:6cbddcebc289550569cc3f6a93676a8f4f605d8574b8aec8448d61594a283996",
        "repository": "mistralai/Ministral-3-8B-Instruct-2512-BF16",
        "revision": "f6fae9795746f63c9be8344932f01275f3c63734",
        "tokenizer_contract_sha256": "b9e3906504b6235b5c289fe9d3f7a86512f968dfe300771ec660080924615dbc"
      },
      "subject": {
        "byte_length": 6058747136,
        "filename": "Ministral-3-8B-Instruct-Q5_K_M.gguf",
        "sha256": "b2c56b655b15577c30734b28b820d054e43df06c7712b74b9c170bff5f10f253"
      }
    },
    "scope": "subject"
  }
]
Exact comparison data
{
  "baseline": {
    "mean_score": 0.4475
  },
  "comparison": {
    "kind": "exact_match_delta_pp",
    "minimum": -2.0,
    "value": 0.5
  },
  "comparison_id": "comparison-4472c601a80f319859c4bf0c505c399c",
  "format": "invarlock/comparison-report-v3",
  "metric": "exact_match",
  "paired_binary": {
    "baseline_fail_subject_pass": 6,
    "baseline_pass_subject_fail": 4,
    "both_fail": 215,
    "both_pass": 175,
    "discordant_pairs": 10,
    "effect_size_confidence_interval": {
      "confidence_level": 0.95,
      "lower_pp": -1.119943249438997,
      "method": "newcombe_hybrid_score_paired_v2",
      "upper_pp": 2.118915687037841
    },
    "effect_size_pp": 0.5,
    "mcnemar_exact_two_sided_p_value": 0.75390625
  },
  "policy_digest": "sha256:48e929519390c41e91c5e907e7e3f3d6d0b26e5ac4345136c4f6b85244b191d9",
  "record_count": 400,
  "sample_qualification": {
    "interval_width": {
      "maximum": 10.0,
      "observed": 3.238858936476838,
      "passed": true,
      "unit": "percentage_points"
    },
    "passed": true,
    "record_count": {
      "minimum": 400,
      "observed": 400,
      "passed": true
    }
  },
  "side_accuracy": {
    "baseline": {
      "observed": 0.4475,
      "passed": true
    },
    "minimum": 0.2,
    "passed": true,
    "subject": {
      "observed": 0.4525,
      "passed": true
    }
  },
  "subject": {
    "mean_score": 0.4525
  },
  "uncertainty": {
    "interval_mass": 0.95,
    "lower": -1.119943249438997,
    "method": "newcombe_hybrid_score_paired_v2",
    "scope": "paired_binary_outcomes",
    "upper": 2.118915687037841
  },
  "verdict": "pass"
}