Four Evidence Packs, Four Narrow Answers

Historical release note: This article records the four-pack v0.14 inventory. In v0.15, three retained BF16-to-Q5_K_M GGUF deployment journeys expand the separate public inventory to seven packs. See the current public evidence guide and v0.15 release note.

“All four comparisons passed” is accurate. It is not enough information to understand what happened.

Two packs compare checkpoints with derivatives made by scaling every weight by 0.99. One compares vision-text checkpoints of different sizes. One exercises two TensorRT-LLM engines whose exact-match scores are both zero on the selected schedule. They use different metrics and answer different comparison questions.

What they share is a review contract, not a model ranking. A public evidence pack binds the artifacts and schedule to record-level results. It also carries the comparison policy, evidence signature, and separate verification receipt.

Start with the four bounded results

Diagram

The two likelihood intervals remain below their authenticated maximum ratio, while the two exact-match intervals remain above their authenticated minimum delta.

Separate metrics and policy boundaries

Normalized NLL · maximum 1.05

Mistral 7B: 0.9979, interval 0.9970–0.9988. Qwen2.5 14B: 0.9985, interval 0.9955–1.0015. Both upper endpoints pass.

Exact-match delta · minimum −2 pp

Qwen2-VL: +11.0 pp, interval +6.34 to +15.67 pp. TinyLlama: 0.0 pp, interval −0.95 to +0.95 pp. Both lower endpoints pass.

Ratios and percentage-point deltas have different units. The zero-accuracy TinyLlama comparison demonstrates the runtime path, not useful task performance.

The panels use separate scales because normalized-NLL ratios and exact-match percentage-point deltas have different units and policy thresholds. Positions in one panel therefore cannot be compared with positions in the other.

The text packs ask whether likelihood regressed

The Mistral 7B and Qwen2.5 14B packs use normalized negative log likelihood (NLL): a per-byte measure of how much probability the model assigns to the expected continuation. Lower is better.

For Mistral, the baseline mean is 1.281 and the subject mean is 1.279. Their subject-to-baseline ratio is 0.9979, with a paired 95% interval of [0.9970, 0.9988].

For Qwen2.5, the corresponding means are 0.7424 and 0.7414. The ratio is 0.9985, with an interval of [0.9955, 1.0015].

In both cases, policy reads the conservative upper endpoint of the paired interval. Both endpoints remain below the authenticated maximum ratio of 1.05, so both finite-schedule comparisons pass.

What may a reader conclude? Under the exact artifacts, runtime identities, records, metric, and policy in each pack, the signed normalized-NLL comparison did not exceed the allowed regression boundary.

What may a reader not conclude? The subjects are universally better, the scaling transformation is safe for every task, or the models have equivalent behavior outside those schedules.

The vision-text pack shows a positive finite-schedule effect

The Qwen2-VL pack compares 2B and 7B checkpoints over a balanced MMMU-Pro Vision schedule spanning 30 subjects and answer choices A–J.

The baseline exact-match score is 14.5%; the subject score is 25.5%. The paired effect is +11.0 pp, with a paired 95% interval of [6.34, 15.67] pp.

Unlike the two likelihood rows, this interval is entirely positive. The pack therefore records a positive effect on this exact 400-record schedule while also exercising the signed transaction through the first-party vision-text runtime.

That is a substantial result for the named comparison. It still is not a universal claim that the 7B checkpoint dominates the 2B checkpoint across every multimodal task.

The zero-score row is a runtime-path result

The TinyLlama pack is the easiest row to overread because both exact-match means are zero. If we looked only at model performance, the row would tell us very little.

The retained transaction answers a narrower systems question. Two authenticated checkpoint engines ran through the TensorRT-LLM provider on the same ordered schedule. The pack preserves their outputs, record identities, runtime bindings, policy arithmetic, signatures, and independent receipt. The paired delta is zero, and its interval of [−0.95, 0.95] pp remains above the authenticated −2 pp non-regression boundary.

So the signed predicate passes. That does not turn zero accuracy into a quality success. It establishes finite-schedule non-regression between the two named engines and qualifies the demonstrated TensorRT-LLM evidence path.

This is why a public index needs the result, interval, policy, and intended qualification in the same row. “Pass” without those fields invites the wrong story.

What remains stable across all four packs

The four comparisons vary in model family, metric, and runtime. Their evidence lifecycle remains the same: bind the inputs, retain paired records, compute the versioned result, sign the canonical directory, and replay it with independently supplied anchors.

After publication, the index records the immutable pack and receipt by content hash. A later result can supersede that entry without rewriting its identity.

That common review contract lets a reviewer reconstruct each result from its evidence. The public evidence guide describes publication and supersession.

How to read a public evidence row

Before repeating a result, ask five questions:

If a summary drops those details, return to the pack or the public index before widening the claim.

Limitations

• Every result is scoped to one authenticated 400-record schedule.
• The two metrics do not form a shared leaderboard or common notion of model quality.
• The TinyLlama row qualifies the named non-regression and runtime path; it does not establish useful task performance.
• At v0.14, four additional comparisons remained explicitly labelled Evidence not yet created.
• Signatures and replay do not establish schedule representativeness, runtime honesty, general safety, or deployment approval.

Sources

Website documentation explains the maintained workflow. The four results and policy semantics described here use InvarLock v0.14.0; the tagged index and evidence directories preserve that evidence basis. Later inventory additions do not update those measurements or verdicts.

• Public evidence guide
• Public evidence guide: published evidence
• Pairing and replay
• Tagged v0.14.0 public evidence index
• v0.14 evidence directories