Policy not met

The observed NLL ratio exceeded the policy limit. Lower NLL is better. Across 400 usable pairs, the subject's Reference NLL was 0.5943 nats / byte, compared with 0.5446 nats / byte for the baseline, giving a subject-to-baseline ratio of 1.091. The 95% interval for the ratio runs from 1.085 to 1.098. The policy requires the interval to stay at or below 1.05.

What was compared

Recorded field	Baseline	Subject
Recorded model IDDiffers	mistralai/Mistral-7B-v0.1	mistralai/Mistral-7B-Instruct-v0.1
Model revisionDiffers	27d67f1b5f57dc0953326b2601d68371d40ea8da	ec5deb64f2c6e6fa90c1abf74a91d5c93a9669ca

Additional recorded context (7)

Matching displayed fields

Matching previews do not establish equality of the complete retained fields.

Evaluator: lm-eval 0.4.12
Capture role: Unavailable in recorded context
Workflow: Unavailable in recorded context
Dataset: Unavailable in recorded context
Records: 400
Effective message roles: Unavailable in recorded context

Additional fields

Evaluation mode: Captured evaluator outputs

Model-key comparison is unavailable because recorded values are missing or mixed.
Prompt comparison unavailable in this report projection: no complete, uniquely paired supported message view is available.

Results and requirements

Reference NLL

Scope: overall

Policy not met

The observed NLL ratio exceeded the policy limit. Lower NLL is better.

Baseline: 0.5446 nats / byte
Subject: 0.5943 nats / byte
Change: 1.091 ratio95% paired-schedule ratio resampling interval: 1.085 to 1.098 ratio
Usable pairs: 4000 missing · 400 included · ≥ 400 required

Estimate95% paired-schedule ratio resampling interval (Paired percentile resampling) Policy thresholdNo changeAllowed change region

How to read this comparison

Change: The marker shows the observed change between subject and baseline. A ratio divides the subject value by the baseline value; 1 means no change.
Interval: The bar shows uncertainty around that change. Its left endpoint is the lower bound; its right endpoint is the upper bound. The interval level does not tell you how often model runs would pass or fail the policy.
Policy rule: The upper bound (right endpoint) must be at most the policy maximum. The whole interval must lie on the allowed side of the threshold for this bound check to pass.
No change: The dashed line marks no change. An interval crossing it includes changes in either direction. A worse observed score describes this run; policy rejection alone does not prove worse performance beyond these cases.

How this interval was calculated

The declared percentile method resamples the same 400 paired records in this scope with replacement, keeping each baseline and subject result together. Its retained resampling count is 2,048.

The central 95% of that distribution describes the subject-to-baseline mean NLL ratio. Each side's NLL uses nats per expected UTF-8 byte.

This is a fixed-schedule resampling interval, not a population confidence interval. Separate metric and scope intervals do not give a simultaneous confidence guarantee or establish a representative production sample.

Decision checks
Check	Observed	Required	Result
Complete paired resultsMissing results are retained; they cannot be dropped to obtain a pass.	400 of 400	All included pairs	Passed
Included pair count	400	>= 400	Passed
Maximum NLL ratioUses the interval bound, not just the observed change.	1.098 ratio	<= 1.05 ratio	Not met
Interval width	0.0135 ratio	<= 0.1 ratio	Passed

Lower values are better.
Recorded scoring basis: captured reference-continuation likelihood facts; NLL means, ratio and interval were recomputed when the comparison was created. Model execution and the supplied likelihood facts are not independently established.
400 usable pairs; 0 missing results; 400 included pairs. Counts in overlapping slices must not be added together.

Next steps

Review the recorded checks against the approved requirements.
For a signed handoff, run invarlock verify with independent signer, policy, request and complete-run anchors and an external signed receipt.
Use the comparison JSON and JUnit outputs in CI; preserve missing results and the original policy.

Scope and limitations

Recorded scores retain source judgments; only their aggregation is recomputed.
Intervals describe the paired schedule under the stated method; representativeness requires an external sampling design.
Metric and slice intervals are marginal, not a simultaneous family-wide confidence guarantee.
A policy pass does not establish truthful model execution, general quality, safety or compliance.
Normalized NLL replays captured reference-continuation likelihood facts; their model execution, tokenizer and configuration claims are source assertions, not independently established runtime facts.
Model and prompt context is evaluator-recorded provenance, not independent execution or model-identity attestation.

Evidence details

Identities, methods and exact recorded values

Comparison and policy identities

Manifest: sha256:f42638bfa9ec5cb2240aa40881fd0039a842cc531bdb10c1a336ce891c445680
Comparison: sha256:9fab658a8634ea309a1d4e911355eba577b357eaa8966e497378a0922a1cabc7
Evidence signer: sha256:d60106c6c8f43c3d5f64498cca0d06bbd9282b15edf94995d65d004a32e3bf15
Baseline run: harness-baseline
Baseline run digest: sha256:257b27bea9017e8ff6a422ccbbc752ba783b253e719a5aedfe4ca467ff5c2b4d
Baseline attributed artifact: sha256:c8e5d83cef2df1fa38f189fbbaab100571622addb05bb812fcc324cadbf099ec
Baseline evaluator: lm-eval 0.4.12
Subject run: harness-subject
Subject run digest: sha256:8f6b8a296f01ae50bb31f4ae956ad2e085e0f15118a18af661c3675538bcad12
Subject attributed artifact: sha256:301a07d113d1964b69f497b54a8bc32a44f3b8ce0899e03e74809fa660c04067
Subject evaluator: lm-eval 0.4.12
Baseline binding: sha256:257b27bea9017e8ff6a422ccbbc752ba783b253e719a5aedfe4ca467ff5c2b4d
Policy binding: sha256:1b722ca1f7bb36840614f66891bd076f61070a37fe855338adefae16c80b3ba8
Subject binding: sha256:8f6b8a296f01ae50bb31f4ae956ad2e085e0f15118a18af661c3675538bcad12

Recorded metric and scope identifiers

[
  {
    "display_metric": "Reference NLL",
    "display_scope": "overall",
    "metric": "reference_nll",
    "result": 1,
    "scope": "overall"
  }
]

Bound policy (configuration preview)

{
  "expected_case_set_digest": "sha256:04d191a1d781754dd593c9bf647989a12a900be50dcccf1ab80f3fc3e3afa5b8",
  "format": "invarlock/comparison-policy-v1",
  "metrics": [
    {
      "aggregation": "mean",
      "configuration": {
        "limits": {
          "depth": 6,
          "nodes": 128,
          "string_characters": 256
        },
        "note": "Display preview, not a replacement configuration. Full configuration remains in evidence under the original policy binding.",
        "preview_only": true,
        "text": "{\n  \"baseline_tokenizer_digest\": \"sha256:11539bde3507db70de31a5931152d0907f24972b41b17468b3ea0f4d9fa70299\",\n  \"configuration_digest\": \"sha256:c4505212e0733ea2aefd2e65d240d8a9449f06a08851c871f3be8593eaee601b\",\n  \"subject_tokenizer_digest\": \"sha256:bef458384298d886e4e30ac9262f18d7cfc14adf71b2643a2935237b7bd3f7c1\"\n}"
      },
      "direction": "lower",
      "kind": "normalized_nll_per_utf8_byte",
      "maximum_interval_width": 0.1,
      "minimum_count": 400,
      "name": "reference_nll",
      "ratio_max": 1.05,
      "unit": "nats_per_utf8_byte"
    }
  ],
  "slices": []
}

Exact comparison data

{
  "bindings": {
    "baseline": "sha256:257b27bea9017e8ff6a422ccbbc752ba783b253e719a5aedfe4ca467ff5c2b4d",
    "policy": "sha256:1b722ca1f7bb36840614f66891bd076f61070a37fe855338adefae16c80b3ba8",
    "subject": "sha256:8f6b8a296f01ae50bb31f4ae956ad2e085e0f15118a18af661c3675538bcad12"
  },
  "decision": "regression",
  "format": "invarlock/multi-metric-comparison-v1",
  "limitations": [
    "Recorded scores retain source judgments; only their aggregation is recomputed.",
    "Intervals describe the paired schedule under the stated method; representativeness requires an external sampling design.",
    "Metric and slice intervals are marginal, not a simultaneous family-wide confidence guarantee.",
    "A policy pass does not establish truthful model execution, general quality, safety or compliance.",
    "Normalized NLL replays captured reference-continuation likelihood facts; their model execution, tokenizer and configuration claims are source assertions, not independently established runtime facts."
  ],
  "metrics": [
    {
      "aggregation": "mean",
      "baseline_mean": 0.5446367517489973,
      "count": 400,
      "decision": "regression",
      "delta": 0.04965006758952362,
      "direction": "lower",
      "interval": {
        "lower": 1.0845635913082117,
        "mass": 0.95,
        "method": "paired_percentile_bootstrap_sha256_v1",
        "replicates": 2048,
        "upper": 1.0980647428628072
      },
      "interval_unit": "ratio",
      "kind": "normalized_nll_per_utf8_byte",
      "missing_count": 0,
      "missing_ids_preview": [],
      "missing_ids_preview_is_complete": true,
      "name": "reference_nll",
      "ratio": 1.091161801751501,
      "reasons": [
        "upper ratio interval bound exceeds ratio_max"
      ],
      "scoring_assurance": "recomputed",
      "slice": "overall",
      "subject_mean": 0.5942868193385209,
      "unit": "nats_per_utf8_byte"
    }
  ]
}

Reference NLL configuration preview

Display preview, not a replacement configuration. Full configuration remains in evidence under the original policy binding.

{
  "baseline_tokenizer_digest": "sha256:11539bde3507db70de31a5931152d0907f24972b41b17468b3ea0f4d9fa70299",
  "configuration_digest": "sha256:c4505212e0733ea2aefd2e65d240d8a9449f06a08851c871f3be8593eaee601b",
  "subject_tokenizer_digest": "sha256:bef458384298d886e4e30ac9262f18d7cfc14adf71b2643a2935237b7bd3f7c1"
}

Preview limits: {"depth": 6, "nodes": 128, "string_characters": 256}