An Evaluator Can Be Supported Without Being Decision-Ready

Revision note: This article was revised to describe the v0.15.0 release snapshot and its three independent axes. The original publication date is retained; the earlier v0.14 “levels” framing is historical.

OpenAI Evals has a maintained integration and replayable qualification records in the snapshot examined here, but no retained model-running signed comparison. LM Evaluation Harness has such comparisons, including one that correctly ends in policy rejection. Neither can be described adequately by a single “supported” badge.

Ask three separate questions: does the integration work, can someone else recalculate the score, and has an actual model comparison completed the signed workflow? The matrix names them:

• Adapter support: the source-specific execution path is maintained.
• Replay authority: another implementation can reconstruct the result from complete retained records.
• Signed-journey maturity: a named model-running transaction crossed the full signed release path.

Diagram

Evaluator evidence is recorded on three independent axes: 19 maintained adapters, 17 profiles with per-record replay authority, and four retained signed transactions.

Three independent properties in the v0.15.0 snapshot

19 maintained profiles

The pinned integrations crossed the adapter contract.

17 replay-authorized profiles

Complete per-record facts permit recomputation. Two other profiles remain observation-only: MLflow and Garak.

4 signed transactions

Two each through LM Evaluation Harness and Inspect AI. Integrity and policy outcome remain separate.

Read the columns independently: they count profiles and transactions in the v0.15.0 snapshot.

Before choosing an integration for a signed comparison, read across its row. An empty comparison field identifies work still to demonstrate; a completed comparison that rejects a model can be evidence that the integration works.

A maintained adapter answers only the first question

Adapter support means the repository can run a pinned evaluator through a maintained integration. The linked qualification record carries the exact version, runner, and dependency details.

The retained matrix contains 19 maintained profiles. Each named path executed and crossed the generic contract under the retained fixture.

Can another verifier rebuild the result?

Replay authority means the verifier can recalculate the score from every ordered record. InvarLock uses those retained inputs, outputs, expected answers, and statuses instead of trusting the evaluator's aggregate.

Seventeen profiles retain that surface. Their qualification uses the same pinned 102-record Qwen3.5 0.8B corpus: 61 matches and 41 mismatches. Agreement on those saved answers demonstrates the examined exact-match import path; it does not rank the evaluators or establish their quality on other tasks.

MLflow and Garak have maintained adapters but remain observation-only in this snapshot. Their outputs cannot supply the replay this contract requires. A Signed Score Is Not Yet Decision Evidence examines the different gaps and what addressing them would involve.

Has the full signed journey actually been run?

Being able to recalculate a fixture's scores does not show that an actual model comparison completed the full workflow. The signed-journey record answers a separate question:

Does the repository retain a named model and corpus run that crossed evaluate → verify → report, with the exact record count and signed outcomes available for review?

This snapshot retains four 400-record transactions across LM Evaluation Harness and Inspect AI. Three MMLU-Pro transactions authenticate correctly and then miss their −2 pp lower-bound requirement. The separate LAMBADA deployment transaction passes its declared −20 pp bound.

Two examples show why the axis is not a pass/fail ladder:

The release note for this snapshot contains the full four-row table. It also records that LM Evaluation Harness and Inspect AI agree on every baseline and subject record in the shared Qwen3.5 comparison. That is cross-evaluator replay consistency, not another verdict.

Reading the matrix without inventing a ladder

These three retained profiles lead to different next steps:

Scroll horizontally to see every column.

Read OpenAI Evals’ empty signed-journey field as “no retained comparison here,” not as a failed model evaluation. Read the Qwen3.5 rejection as a completed comparison with an unfavorable policy outcome, not as a broken adapter.

The seven public evidence packs are a separate comparison inventory. Adding one does not by itself qualify another evaluator integration.

Applying the model to a private evaluator

A proprietary SDK, CLI, or API does not need a special authority model. Record the immutable version, entry point, runner inputs, and normalized output for adapter support. Retain complete ordered records when independent replay is required. Retain a separate signed model-running transaction when the full journey needs to be demonstrated.

The evaluator can remain private. Reviewers still need to know exactly which records they can check and what those checks establish.

Use the three columns to estimate the work between “supported” and the comparison you need. Request the pinned integration record, replay inputs and relevant signed example separately. That prevents an integration choice from resting on a badge whose meaning changes from one tool to another.

Limitations

• The 102-record qualification corpus demonstrates one pinned exact-match surface, not general evaluator quality.
• The four signed transactions cover two evaluators and their named 400-record corpora; other tools, tasks, models, and metrics need their own journeys.
• MLflow and Garak remain observation-only under the semantics retained in this snapshot.
• Judge-based, human-scored, or otherwise non-deterministic surfaces require separately defined evidence and replay contracts.
• The three axes do not establish general model safety, compliance, or deployment suitability.