An Evaluator Can Be Supported Without Being Decision-Ready
Evaluator support has three independent meanings: a maintained adapter, replayable records, and a retained signed journey. None implies the other two.
Updated September 6, 2026: This article was revised to describe the v0.15.0 release snapshot and its three independent axes. The original publication date is retained; the earlier v0.14 “levels” framing is historical.
Evaluator support has three independent meanings. A reviewer needs to know whether the adapter is maintained, whether its records can be replayed, and whether a named model run completed the signed release path.
The retained matrix records those questions on separate axes:
- Adapter support: the source-specific execution path is maintained.
- Replay authority: another implementation can reconstruct the result from complete retained records.
- Signed-journey maturity: a named model-running transaction crossed the full signed release path.
Three independent evaluator properties
- 19 maintained profiles
- A pinned upstream entry point, runner, and dependency surface crossed the adapter contract.
- 17 replayable profiles
- Complete per-record exact-match evidence; the other two profiles, MLflow and Garak, remain observation-only.
- 4 signed transactions
- Retained model-running journeys across two evaluators. Each records its own integrity and policy outcome.
Read each property separately. These counts are not stages on a maturity ladder.
The axes can combine in different ways. A profile can be maintained without being replay-authorized. It can be replay-authorized without having a retained model-running journey. And a signed journey can still end in policy rejection.
A maintained adapter answers only the first question
Adapter support means the repository can run a pinned evaluator through a maintained integration. The linked qualification record carries the exact version, runner, and dependency details.
The retained matrix contains 19 maintained profiles. Each named path executed and crossed the generic contract under the retained fixture.
Adapter maintenance is an engineering claim about an integration surface. It says nothing by itself about complete records, independent recomputation, a signed model run, or a policy verdict.
Can another verifier rebuild the result?
Replay authority begins with the records, not the adapter badge.
Replay authority means the verifier can recalculate the score from every ordered record. InvarLock uses those retained inputs, outputs, expected answers, and statuses instead of trusting the evaluator's aggregate.
Seventeen profiles retain that surface. They share one pinned 102-record Qwen3.5 0.8B qualification corpus containing 61 matches and 41 mismatches. The corpus demonstrates one deterministic exact-match import and replay boundary. It does not rank the evaluators or establish their quality on other tasks.
The other two profiles make the boundary easier to see:
| Profile | What the adapter retains | Why it remains observation-only |
|---|---|---|
| MLflow | Aggregate output | The aggregate cannot reconstruct the complete ordered record set needed for independent recomputation. |
| Garak | Maintained source-specific output | The retained surface has replay semantics that the qualification contract in this snapshot does not support. |
Scroll horizontally to see every column.
These integrations retain useful context without supplying records to paired policy. A Signed Score Is Not Yet Decision Evidence explains that admission rule and the different MLflow and Garak limitations.
Has the full signed journey actually been run?
Replayable qualification evidence still does not prove that a model-running transaction crossed the full release path. Signed-journey maturity asks a separate, operational question:
Does the repository retain a named model and corpus run that crossed
evaluate→verify→report, with the exact record count and signed outcomes available for review?
This snapshot retains four 400-record transactions across LM Evaluation Harness and Inspect AI. Three MMLU-Pro transactions authenticate correctly and then miss their −2 pp lower-bound requirement. The separate LAMBADA deployment transaction passes its declared −20 pp bound.
Two examples show why the axis is not a pass/fail ladder:
| Retained transaction | What the signed outcome shows |
|---|---|
| Qwen3.5 9B · LM Evaluation Harness | Integrity is valid; the paired policy rejects [−6.42, 1.43] pp. |
| Qwen3.5 0.8B deployment · Inspect AI | The declared LAMBADA deployment policy passes [−8.99, −2.47] pp. |
The release note for this snapshot contains the full four-row table. It also records that LM Evaluation Harness and Inspect AI agree on every baseline and subject record in the shared Qwen3.5 comparison. That is cross-evaluator replay consistency, not another verdict.
Reading the matrix without inventing a ladder
OpenAI Evals, for example, has maintained adapter support and replayable per-record qualification evidence but no retained model-running signed journey. MLflow and Garak have maintained adapters while remaining observation-only. LM Evaluation Harness and Inspect AI have retained signed journeys, yet each transaction carries its own integrity and policy outcome.
The seven public evidence packs are another inventory with another purpose. They retain model comparisons and deployment paths. They are not seven evaluator transactions, and they do not change the 19/17/2 qualification counts.
A retained journey records integrity and a comparison-policy result. Recipient acceptance and deployment approval remain separate decisions; neither is an additional evaluator-support level.
Applying the model to a private evaluator
A proprietary SDK, CLI, or API does not need a special authority model. Record the immutable version, entry point, runner inputs, and normalized output for adapter support. Retain complete ordered records when independent replay is required. Retain a separate signed model-running transaction when the full journey needs to be demonstrated.
The evaluator can remain private. The evidence boundary still needs to be explicit.
Before allowing its output into a release decision, ask:
- Is the exact upstream version, entry point, runner, and dependency surface maintained?
- Are complete ordered records available for independent recomputation?
- Does a retained model-running transaction exist for the exact evaluator, corpus, and record count being claimed?
- Are integrity, policy, recipient acceptance, and deployment approval still represented separately?
Those answers locate the integration on three axes without granting it a fictional all-in-one maturity level.
Limitations
- The 102-record qualification corpus demonstrates one pinned exact-match surface, not general evaluator quality.
- The four signed transactions cover two evaluators and their named 400-record corpora; other tools, tasks, models, and metrics need their own journeys.
- MLflow and Garak remain observation-only under the semantics retained in this snapshot.
- Judge-based, human-scored, or otherwise non-deterministic surfaces require separately defined evidence and replay contracts.
- The three axes do not establish general model safety, compliance, or deployment suitability.
Sources
Website documentation explains the maintained workflow. The counts and classifications described here use InvarLock v0.15.0, adopted in the September 6 revision; the tagged sources preserve that evidence basis. They are not a rolling inventory of later integrations.
More in Research Note
Explore nearby related posts.
Research note
Four Evidence Packs, Four Narrow Answers
Four retained evidence packs passed four different tests. Their metrics, intervals, runtimes, and policies show exactly how far each result reaches.
Research Note
Invariants Are Necessary but Not Sufficient
Structural checks matter because they prevent obviously broken edits from reaching evaluation. They do not, by themselves, show that an edit preserved quality.
Research note
A Signed Score Is Not Yet Decision Evidence
A signed evaluator output becomes decision evidence only when its records can be independently replayed under an authorized contract.