Policy not met

The observed change was within the policy limit, but its uncertainty interval extended beyond it. Across 400 usable pairs, the subject's accuracy was 54.75%, compared with 53% for the baseline, a change of +1.75 pp. The 95% interval for the change runs from -0.8299 to 4.322 pp. The policy requires the interval to stay at or above -0 pp. The subject score must be at least 20%.

What was compared

Recorded field	Baseline	Subject
Recorded runDiffers	qwen3.5-9b-baseline-retained	qwen3.5-9b-subject-retained

Additional recorded context (8)

Matching displayed fields

Matching previews do not establish equality of the complete retained fields.

Evaluator: published-mmlu-pro-qwen3.5-9b-answer-extract 4f036ffe99cfaa82019c90ea527b8c4291a091b3
Model revision: Unavailable in recorded context
Capture role: Unavailable in recorded context
Workflow: Unavailable in recorded context
Dataset: Unavailable in recorded context
Records: 400
Effective message roles: Unavailable in recorded context

Additional fields

Evaluation mode: Captured evaluator outputs

Model-key comparison is unavailable because recorded values are missing or mixed.
Prompt comparison unavailable in this report projection: no complete, uniquely paired supported message view is available.

Results and requirements

accuracy

Scope: overall

Policy not met

The observed change was within the policy limit, but its uncertainty interval extended beyond it.

Baseline: 53%212 of 400 matched
Subject: 54.75%219 of 400 matched
Change: +1.75 ppPaired 95% CI: -0.8299 to +4.322 pp
Usable pairs: 4000 missing · 400 included · ≥ 400 required

EstimatePaired 95% confidence interval (Newcombe hybrid score) Policy thresholdNo changeAllowed change region

How to read this comparison

Change: The marker shows the observed change between subject and baseline. A change in percentage points (pp) is the subject percentage minus the baseline percentage.
Interval: The bar shows uncertainty around that change. Its left endpoint is the lower bound; its right endpoint is the upper bound. The interval level does not tell you how often model runs would pass or fail the policy.
Policy rule: The paired lower bound (left endpoint) must be at least the policy minimum. The whole interval must lie on the allowed side of the threshold for this bound check to pass.
No change: The dashed line marks no change. An interval crossing it includes changes in either direction. A worse observed score describes this run; policy rejection alone does not prove worse performance beyond these cases.

How this interval was calculated

Baseline and subject are paired on the same 400 cases in this scope. Newcombe hybrid score uses their paired match outcomes to form a nominal 95% confidence interval for the accuracy change.

Change is subject accuracy minus baseline accuracy in percentage points, not relative percent change. The confidence level describes the interval method, not the share of correct answers.

Each metric and scope has its own interval; these intervals do not provide a simultaneous confidence guarantee across all results or establish a representative production sample.

Decision checks
Check	Observed	Required	Result
Complete paired resultsMissing results are retained; they cannot be dropped to obtain a pass.	400 of 400	All included pairs	Passed
Included pair count	400	>= 400	Passed
Allowed changeUses the interval bound, not just the observed change.	-0.8299 pp	>= -0 pp	Not met
Interval width	5.152 pp	<= 10 pp	Passed
Subject minimum	54.75%	>= 20%	Passed

Higher values are better.
Recorded scoring basis: expected and output values, scored when the comparison was created.
400 usable pairs; 0 missing results; 400 included pairs. Counts in overlapping slices must not be added together.
Matches: baseline 212 of 400; subject 219 of 400.

Next steps

Review the recorded checks against the approved requirements.
For a signed handoff, run invarlock verify with independent signer, policy, request and complete-run anchors and an external signed receipt.
Use the comparison JSON and JUnit outputs in CI; preserve missing results and the original policy.

Scope and limitations

Recorded scores retain source judgments; only their aggregation is recomputed.
Intervals describe the paired schedule under the stated method; representativeness requires an external sampling design.
Metric and slice intervals are marginal, not a simultaneous family-wide confidence guarantee.
A policy pass does not establish truthful model execution, general quality, safety or compliance.
Model and prompt context is evaluator-recorded provenance, not independent execution or model-identity attestation.

Evidence details

Identities, methods and exact recorded values

Comparison and policy identities

Manifest: sha256:0c2a38d748146d868af9b0070598b324391434b1cd9cf110a2112a788f820e7f
Comparison: sha256:24a88125ec133e8176023568c820663a037477333e13e0d6e6d1767641e34ede
Evidence signer: Unsigned local evidence
Baseline run: qwen3.5-9b-baseline-retained
Baseline run digest: sha256:6bb727f831d82e9344911b0d609ddeb265c3f2ca0ce6f32fbfe232332b76d744
Baseline attributed artifact: sha256:9774e232a74e1c4c1d28ba22bec451bef4a956f031e1fefc8326e255b998bb78
Baseline evaluator: published-mmlu-pro-qwen3.5-9b-answer-extract 4f036ffe99cfaa82019c90ea527b8c4291a091b3
Subject run: qwen3.5-9b-subject-retained
Subject run digest: sha256:8ad6f016beaff54fbc7af1411a4b292066311b9a5dea0c6470bff0b01cfa75ab
Subject attributed artifact: sha256:be1b745caa7acd5c0bb0837f2e0f668d3c7929a9fc8600a7118cc62db4bf4e2a
Subject evaluator: published-mmlu-pro-qwen3.5-9b-answer-extract 4f036ffe99cfaa82019c90ea527b8c4291a091b3
Baseline binding: sha256:6bb727f831d82e9344911b0d609ddeb265c3f2ca0ce6f32fbfe232332b76d744
Policy binding: sha256:262ffe34593a309341f32c054c315509cce788a44991b7930d6e95e17d2a2b83
Subject binding: sha256:8ad6f016beaff54fbc7af1411a4b292066311b9a5dea0c6470bff0b01cfa75ab

Bound policy (configuration preview)

{
  "format": "invarlock/comparison-policy-v1",
  "metrics": [
    {
      "aggregation": "mean",
      "configuration": {
        "limits": {
          "depth": 6,
          "nodes": 128,
          "string_characters": 256
        },
        "note": "Display preview, not a replacement configuration. Full configuration remains in evidence under the original policy binding.",
        "preview_only": true,
        "text": "{}"
      },
      "direction": "higher",
      "kind": "exact_match",
      "maximum_interval_width": 0.1,
      "maximum_regression": 0.0,
      "minimum_count": 400,
      "name": "accuracy",
      "subject_minimum": 0.2,
      "unit": "score"
    }
  ],
  "slices": []
}

Exact comparison data

{
  "bindings": {
    "baseline": "sha256:6bb727f831d82e9344911b0d609ddeb265c3f2ca0ce6f32fbfe232332b76d744",
    "policy": "sha256:262ffe34593a309341f32c054c315509cce788a44991b7930d6e95e17d2a2b83",
    "subject": "sha256:8ad6f016beaff54fbc7af1411a4b292066311b9a5dea0c6470bff0b01cfa75ab"
  },
  "decision": "regression",
  "format": "invarlock/multi-metric-comparison-v1",
  "limitations": [
    "Recorded scores retain source judgments; only their aggregation is recomputed.",
    "Intervals describe the paired schedule under the stated method; representativeness requires an external sampling design.",
    "Metric and slice intervals are marginal, not a simultaneous family-wide confidence guarantee.",
    "A policy pass does not establish truthful model execution, general quality, safety or compliance."
  ],
  "metrics": [
    {
      "aggregation": "mean",
      "baseline_mean": 0.53,
      "count": 400,
      "decision": "regression",
      "delta": 0.01749999999999996,
      "direction": "higher",
      "interval": {
        "lower": -0.008298925545805375,
        "mass": 0.95,
        "method": "newcombe_hybrid_score_paired_v2",
        "replicates": 0,
        "upper": 0.043217785714311986
      },
      "kind": "exact_match",
      "missing_count": 0,
      "missing_ids_preview": [],
      "missing_ids_preview_is_complete": true,
      "name": "accuracy",
      "reasons": [
        "lower interval bound exceeds allowed regression"
      ],
      "scoring_assurance": "recomputed",
      "slice": "overall",
      "subject_mean": 0.5475,
      "unit": "score"
    }
  ]
}

accuracy configuration preview

Display preview, not a replacement configuration. Full configuration remains in evidence under the original policy binding.

{}

Preview limits: {"depth": 6, "nodes": 128, "string_characters": 256}