Four Evidence Packs, Four Narrow Answers

Historical release note: This article records the four-pack v0.14 inventory. In v0.15, three retained BF16-to-Q5_K_M GGUF deployment journeys expand the separate public inventory to seven packs. See the current public evidence guide and v0.15 release note.

Two TinyLlama engines answered none of their 400 test questions correctly, yet their comparison passed its declared policy. The rule tested whether one had regressed relative to the other; it did not require useful accuracy.

That result makes “all four comparisons passed” an incomplete summary of this historical evidence set. Two other packs test likelihood after scaling weights by 0.99, and one compares differently sized vision-text checkpoints. Each pass answers its own question.

A public evidence pack records the exact model identities, cases, results and rules behind that answer, with signed evidence and a separate verification receipt.

The zero-score row is a runtime-path result

The TinyLlama pack demonstrates a systems path despite its zero exact-match scores. Two authenticated checkpoint engines ran through the TensorRT-LLM provider on the same ordered schedule. The pack preserves their outputs, record identities, runtime bindings, policy arithmetic, signatures, and independent receipt. The paired delta is zero, and its interval of [−0.95, 0.95] pp remains above the authenticated −2 pp non-regression boundary.

The lower endpoint clears −2 pp, so the relative-change rule passes. Neither engine answered a case correctly. A reviewer who needs a minimum accuracy must state that separately; the later v0.15 accuracy-floor control addresses that gap without changing this historical result.

This is why a public index needs the result, interval, policy, and intended qualification in the same row. “Pass” without those fields invites the wrong story.

Start with the four bounded results

• Comparison

  Mistral 7B checkpoint vs 0.99-scaled derivative

  Test setup

  Hugging Face text · 400 MMLU-Pro

  Result

  NLL ratio 0.9979

  95% interval

  [0.9970, 0.9988]

  Policy

  Maximum 1.05 · Pass

• Comparison

  Qwen2.5 14B checkpoint vs 0.99-scaled derivative

  Test setup

  Hugging Face text · 400 MMLU-Pro

  Result

  NLL ratio 0.9985

  95% interval

  [0.9955, 1.0015]

  Policy

  Maximum 1.05 · Pass

• Comparison

  Qwen2-VL 2B vs 7B checkpoints

  Test setup

  Hugging Face vision-text · 400 MMMU-Pro Vision

  Result

  Exact-match delta +11.0 pp

  95% interval

  [6.34, 15.67] pp

  Policy

  Minimum −2 pp · Pass

• Comparison

  Two TinyLlama 1.1B checkpoint engines

  Test setup

  TensorRT-LLM · 400 MMLU-Pro

  Result

  Exact-match delta 0.0 pp

  95% interval

  [−0.95, 0.95] pp

  Policy

  Minimum −2 pp · Pass

Diagram

The two likelihood intervals remain below their authenticated maximum ratio, while the two exact-match intervals remain above their authenticated minimum delta.

Separate metrics and policy boundaries

Normalized NLL · maximum 1.05

Mistral 7B: 0.9979, interval 0.9970–0.9988. Qwen2.5 14B: 0.9985, interval 0.9955–1.0015. Both upper endpoints pass.

Exact-match delta · minimum −2 pp

Qwen2-VL: +11.0 pp, interval +6.34 to +15.67 pp. TinyLlama: 0.0 pp, interval −0.95 to +0.95 pp. Both lower endpoints pass.

Ratios and percentage-point deltas have different units. The zero-accuracy TinyLlama comparison demonstrates the runtime path, not useful task performance.

The panels use separate scales because normalized-NLL ratios and exact-match percentage-point deltas have different units and policy thresholds. Positions in one panel therefore cannot be compared with positions in the other.

The text packs ask whether likelihood regressed

The Mistral 7B and Qwen2.5 14B packs use normalized negative log likelihood (NLL): a per-byte measure of how much probability the model assigns to the expected continuation. Lower is better.

Mistral’s mean moves from 1.281 to 1.279; Qwen2.5’s moves from 0.7424 to 0.7414. The table and chart show their subject-to-baseline ratios and paired intervals.

Both upper endpoints remain below the maximum ratio of 1.05. That meets the allowed regression bound on these records; it does not establish that scaling weights improves behavior on other tasks.

The vision-text pack shows a positive finite-schedule effect

The Qwen2-VL pack compares 2B and 7B checkpoints over a balanced MMMU-Pro Vision schedule spanning 30 subjects and answer choices A–J.

The baseline exact-match score is 14.5%; the subject score is 25.5%. The paired effect is +11.0 pp, with a paired 95% interval of [6.34, 15.67] pp.

Unlike the two likelihood rows, this interval is entirely positive. The pack therefore records a positive effect on this exact 400-record schedule while also exercising the signed transaction through the first-party vision-text runtime.

That is a substantial result for the named comparison. It still is not a universal claim that the 7B checkpoint dominates the 2B checkpoint across every multimodal task.

What remains stable across all four packs

The same verification workflow preserves all four results, including the zero-score pass. It is doing its job when it reproduces the declared rule accurately. Whether that rule asks enough is a separate question for the person choosing the requirement.

The public evidence guide explains how each pack and receipt is retained. For interpreting these rows, the important detail is which condition each pass actually satisfies.

How to read a public evidence row

Read the word “pass” with its condition attached:

Before using a relative-change rule, decide whether you also require an absolute minimum. The TinyLlama row shows why those requirements cannot substitute for each other. The finite-schedule article addresses the next question: how far a passing result can apply beyond its tested cases.

Limitations

• Every result is scoped to one authenticated 400-record schedule.
• The two metrics do not form a shared leaderboard or common notion of model quality.
• The TinyLlama row qualifies the named non-regression and runtime path; it does not establish useful task performance.
• At v0.14, four additional comparisons remained explicitly labelled Evidence not yet created.
• Signatures and replay do not establish schedule representativeness, runtime honesty, general safety, or deployment approval.