Why Paired Evaluation Beats Before/After Benchmarks

Historical context: This article preserves the product model and examples described at its original publication date. The v0.13.0 transition replaced that operator workflow. Documentation links lead to maintained guidance; the retired commands and contracts below are not present-day instructions.

Comparison quality is part of the result

Highlights

• Pairing is enforced through three report invariants: window_match_fraction == 1.0, window_overlap_fraction == 0.0, paired_windows > 0.
• CI and Release profiles hard-abort when pairing falls short of the tier floor; failures surface as E001 and related codes.
• evaluation.report.json exposes the pairing trace — dataset window stats, counts, overlap behavior — so the comparison is inspectable, not just asserted.

A model-edit benchmark can look precise while still being structurally weak. The usual failure mode is simple: a reader sees one baseline score, one edited-model score, and a difference between them, but the underlying comparison was not held fixed tightly enough to justify that difference.

That is why pairing matters.

In InvarLock, the point is not only to compare a baseline and an edited subject. It is to compare them on the same deterministic windows, with overlap and count checks surfaced in the report and enforced by the verifier. That turns "before/after" from a loose benchmark habit into a constrained measurement procedure.

The Problem With Naive Before/After Numbers

The phrase "before and after" sounds stronger than it is.

If the baseline and edited runs do not reuse the same windows, then some of the observed difference may come from the schedule itself rather than from the edit. Different slices of text, different coverage, silent overlap, or mismatched counts can all contaminate the comparison. The resulting number may still be interesting, but it is no longer the clean statement many readers assume it is.

InvarLock's public docs therefore treat pairing as part of the method surface. They do not describe evaluation as "run two benchmarks and compare them later." They describe evaluation as a paired comparison against a fixed baseline with deterministic windows and explicit guard contracts.

What Pairing Changes

The baseline and edited subject reuse the same preview and final windows. The schematic contrasts that alignment with two runs drawn from different windows; the letters are illustrative window IDs, not measured results.

Diagram

Unpaired runs use window IDs A, B, C and D, E, F; paired runs align baseline and subject on the same A, B, C windows.

Align the comparison windows

Different windows

Baseline windows A, B, C and subject windows D, E, F change both the subject and the evaluation sample.

Paired windows

Baseline and subject reuse A, B, C so each comparison shares a window identity.

Inspect the alignment

Review window identities, counts, and overlap before interpreting the score difference.

Illustrative IDs in the historical workflow. Pairing alone does not establish representative coverage or model quality.

The assurance case ties the historical paired metrics to log-space comparison and bootstrap intervals. Reported counts, identities, and overlap statistics let a reviewer inspect whether that comparison surface was held fixed.

Why InvarLock Treats Pairing As A Runtime Contract

The most important thing in the pairing docs is not the definition. It is the fact that pairing is enforced.

The public coverage-and-pairing note is explicit: valid schedules use fixed seeds, non-overlapping windows, and exact reuse of baseline window IDs for edited runs. CI and Release profiles fail closed if pairing is insufficient, overlap is present, or window counts do not meet the tier floor.

The datasets reference makes the same posture concrete through invariants. window_match_fraction must be 1.0. window_overlap_fraction must be 0.0. paired_windows must be greater than zero. Missing or invalid baseline evidence cannot support a hand-wavy comparison; it stops the stronger claim from being made.

That distinction matters. A serious evaluation system should not rely on author intent to keep comparisons clean. It should make the bad comparison harder to ship.

What The Report Makes Visible

The reports reference is useful here because it shows how pairing leaves a trail.

evaluation.report.json carries dataset and window statistics, primary-metric fields, and the validation surface the verifier uses. Pairing is not buried in a side note. It appears in the evidence flow, in the dataset windows stats, and in the verify path that checks pairing, count logic, and ratio math.

That is the practical difference between "paired evaluation" as a slogan and pairing as an evidence system. In the first case, readers are asked to trust that the comparison was careful. In the second case, they can inspect whether the schedule actually matched.

What Pairing Still Does Not Solve

Pairing is strong, but it is not magic.

It does not tell you whether the dataset was a good choice. It does not rescue a tokenizer mismatch. It does not solve task validity, deployment realism, or broader questions about whether the benchmark reflects the behavior you actually care about. Pairing improves the fairness of the comparison. It does not automatically make the comparison globally important.

Pairing makes a deliberately narrow promise. It does not make benchmarking objective. It makes exact window reuse a much stronger comparison surface than a loose before/after number when the question is regression from a weight edit.

Limitations

• Pairing tightens the comparison surface, not the question being asked: a paired evaluation of an unrepresentative dataset is still unrepresentative.
• The cited pairing docs define the complete window identity, coverage, and overlap contracts.
• Whether a given benchmark should be the comparison is out of scope.

Sources

Website documentation explains the maintained workflow. The examples described here preserve the historical workflow at the article’s original publication date. These maintained guides explain the related concepts; they do not reinstate the retired commands or contracts.

• Documentation
• Assurance case
• Pairing and replay
• Schedule and policy
• Reports and receipts