Evaluator qualification

InvarLock qualifies evaluator evidence through one closed, evaluator-neutral boundary. qualify_evaluator_export consumes normalized contracts without dispatching to an evaluator SDK. Example-owned runners execute representative upstream tools and normalize their results into the same four contracts.

Reference

Surface: Four canonical JSON contracts, the invarlock-qualify-evaluator companion CLI, and invarlock.engine.qualify_evaluator_export

Stability: Closed v1 qualification formats

Use this page when: Integrating an open or proprietary evaluator, or reviewing the maintained upstream-execution matrix

A matrix row demonstrates the named version and entry point; it does not make that evaluator a built-in InvarLock plugin.

The captured-results integration adds installed parsers for selected native export shapes and a multi-metric comparison behind the core evaluation commands. The qualification boundary documented here also accepts normalized matching, numeric tolerance, structured fields and token F1 when independent reference text is available for recomputation. Existing retained matrix rows continue to prove their declared exact-match profiles; they are not evidence for every new metric.

Capture for exact match, likelihood and judge scoring

The installed load_run parsers support canonical JSON, generic JSONL, inspect-json, lm-eval-samples and promptfoo-jsonl. Other workflows can map original per-case facts through invarlock.engine.capture_evaluator_run and inspect evaluator_input_capabilities. These are captured-input interfaces, separate from qualification and the example-owned signed OCI bridges.

The maintained capture helper covers all 19 ecosystems through explicit canonical records. It can also join 17 retained deterministic exports to their original cases and independent schedule. MLflow aggregate observations and Garak detector summaries cannot supply original per-case predictions, references or likelihoods; those workflows must capture the actual records separately. Historical matrix identities and authority remain unchanged.

InvarLock owns the selected scorer. Exact match requires string answers and references. Judge requires task and answer text plus a declared recipe and complete retained judge calls; a scalar rating is insufficient. Structured task inputs require an explicit text projection that preserves the original facts. Per-case judge references are opt-in and stay separate from the model input. Normalized NLL requires typed reference-continuation log probabilities, byte and token counts, and bound source, artifact, configuration and tokenizer identities. Naming an evaluator cannot supply these facts.

The separate Harness likelihood reference retains real unmodified Harness 0.4.12 HFLM measurements on six authored same-model CPU pairs, followed by signed evaluation and independent replay. It establishes that likelihood compatibility profile. Synthetic canonical NLL tests exercise the shared contract across the shortlist; neither those tests nor the retained exact-match matrix qualify all 19 native likelihood or judge paths.

Diagram

The maintained evaluator matrix keeps adapter support, replay authority, and signed-journey maturity independent: 19 maintained profiles, 17 replayable per-record profiles, two observation-only profiles, and four retained transactions across two evaluators.

Three independent status axes

The repository does not assign one cumulative “integration level.” Support, replay authority, and retained transaction evidence answer different questions and cannot safely substitute for one another.

Scroll horizontally to see every column.

The stable qualification-result contract continues to emit outcome: qualified_for_import with authority: verdict_authority for an independently replayable result. It emits observation_only for both fields when replay is unavailable. Those wire values do not claim that InvarLock maintains the adapter or that a signed OCI journey has been demonstrated.

Qualification matrix

The small conformance corpus uses two local records: one exact match and one mismatch. It proves every maintained upstream entry point and both authority modes without downloading a model or calling a hosted evaluator.

The independently replayable rows also execute against a retained 102-record evaluation produced by the immutable Qwen/Qwen3.5-0.8B revision recorded in the corpus. The model produced 61 exact matches and 41 mismatches. Each evaluator scores all 102 model outputs through its real upstream entry point. InvarLock then recomputes every score and replays all 102 normalized records through the runtime-import authoring boundary.

Each row identifies a maintained evaluator entry point with retained upstream execution. The rows cover distinct evaluator ecosystems and workflows; matrix.json records their source, version, authority, and status metadata.

LM Evaluation Harness and Inspect AI additionally have independently replayable 102-record shared-output imports. Their retained current-model OCI journeys execute Qwen3.5 9B over the same ordered 400-record MMLU-Pro schedule. LM Evaluation Harness also retains a Gemma 4 12B instruction-to-QAT portability transaction. The other rows retain qualification and compatibility evidence without claiming a native signed journey.

The compact evidence packs, signed verifier receipts, independent policies, and builder-signed OCI attestations are retained under examples/evaluator-qualification/signed-transactions/ and replayed by make evaluator-qualification.

The 400-item semantic suite is selected deterministically from a revision- and hash-pinned TIGER-Lab MMLU-Pro dataset. It balances all 14 subject categories and answer labels A–J. Qwen and Gemma prompts derive from those same semantic records using separately pinned model-native no-thinking renderings. The 400-record size supports the maximum 10-percentage-point paired interval width declared in advance: Qwen achieved a 7.85-point width and Gemma achieved a 4.68-point width.

The retained flagship comparison confirms an identical ordered schedule and reports native agreement without creating another acceptance verdict. LM Evaluation Harness and Inspect AI agree on every baseline and subject output and score across all 800 side records. The shared-output replay and native signed journeys answer complementary questions: the first isolates evaluator scoring across the wider matrix, while the second exercises both flagship frameworks' real model paths.

The matrix represents the Microsoft PromptFlow lineage with Azure AI Evaluation rather than preserving the deprecated promptflow-evals package as a second legacy row. The OpenAI Evals qualification profile is installed from an immutable source revision, while its signed transaction integration runs the unchanged upstream basic.Match evaluator from an explicitly versioned wheel derived from the hash-pinned evals==3.0.1.post1 input. The isolated image removes the unused NLTK dependency; its narrower dependency set does not change the retained qualification profile.

The generated matrix below describes the retained generic qualification profiles. The signed bridges execute LM Evaluation Harness and the native Inspect task (inspect_ai.eval plus inspect_ai.scorer.match) in clean OCI transactions. Their build attestations, worker protocol, and native adapters remain example-owned; the installed core receives evaluator-neutral runtime-import and signed transaction contracts.

The signed bridges retain native evaluator facts but keep the transaction metric independent. Inspect's pinned causal HF decoder has an explicit authenticated boundary recovery, and InvarLock replays byte-exact equality for the signed acceptance decision after checking each evaluator's native event semantics.

Inspect literal-pair boundary

Fresh literal qualification uses the separate inspect-ai-literal-pairs-v1 profile in examples/evaluator-qualification/maintained/inspect-profile.json. Its pinned scorer is Inspect 0.3.254 match(location="exact", ignore_case=False, numeric=False) with one string target. Inspect strips boundary whitespace and ASCII punctuation even with these settings, so its native notion of exact matching can differ from literal equality.

The supported domain is defined for each output/reference pair. Literal equality must agree with equality after Inspect's boundary stripping. Equal strings containing leading whitespace or punctuation remain supported, as do distinct strings that remain distinct after stripping. Unequal strings that collapse together, such as " answer" and "answer", fail explicitly. Array or multiple targets are outside this profile. Native score, answer and explanation contradictions also fail before an export is written. The independent qualification metric remains byte-exact equality.

Before scoring, the current runner checks the complete supplied public profile against its local, source-bound definition: package version, dependency lock, runner bundle, upstream URL and authority must all agree. Even a matching candidate package installation cannot reuse this profile identity. The runner also freezes its inputs before scoring and publication.

Run the pinned 28-case differential corpus and a fresh qualification of the retained 102 model outputs with:

uv run --no-project --with-requirements examples/evaluator-qualification/locks/inspect-ai.txt python examples/evaluator-qualification/maintained/inspect_differential.py --output /tmp/inspect-boundaries.json
PYTHONPATH=src python examples/evaluator-qualification/maintained/qualify_inspect.py --output /tmp/inspect-literal-qualification

Both output paths must be new. The differential covers boundary and internal whitespace and punctuation, case, Unicode, empty outputs and multiple targets. It reports semantic drift with exit 2 and always declares authority: none. Running it with a candidate dependency version cannot promote that dependency to a qualified profile. The fresh qualification command separately executes the pinned upstream scorer and independently checks the resulting export. This is scoring of existing outputs, without new model inference or a new signed OCI transaction.

The separate artifacts under examples/evaluator-qualification/maintained/artifacts/inspect-ai-literal-pairs-v1/ retain the 28-case observation, the fresh 102-record qualification and a source hash manifest. All 102 scores and the normalized runtime-record digest agree with the historical Inspect import: 61 matches and 41 mismatches.

The original inspect-ai profile, runner bytes, qualification artifacts and signed transaction outcomes remain unchanged and replay under their original identities. They are not relabeled as evidence for the new runner. The matrix below continues to describe those retained identities.

Current strict batch profiles

Fresh batch qualification uses the separately named *-strict-batch-v1 profiles in examples/evaluator-qualification/maintained/batch-profiles.json. These profiles preserve the original matrix's historical identities while adding strict validation of the actual native rows:

Scroll horizontally to see every column.

Each supported record has one string output and reference, and a unique ID in the independent schedule order. Repeated prompts can carry different outputs; they do not overwrite one another. The runner rejects missing, duplicate, extra or reordered native rows, changed references or outputs, wrong metric configuration, score/detail contradictions, and aggregate substitution. Source inputs are copied before execution so a later source edit cannot change which records are exported. The complete supplied public profile must match its local definition, including package identity, dependency and runner hashes, upstream URL, and authority. Changing the installed package and supplied profile together does not bypass this check.

Literal equality preserves case, whitespace, punctuation, Unicode and empty strings. Promptfoo's echo path has additional renderer restrictions: an output ending in LF is rejected because the pinned variable renderer removes one final newline. Output or reference text containing template opening delimiters, file:// or package: is also rejected before execution. Leading whitespace and internal whitespace remain supported. Multiple targets and non-string outputs are outside all five profiles.

For example, run the pinned Evidently boundary audit and separately qualify the retained model outputs:

uv run --no-project --with-requirements examples/evaluator-qualification/locks/evidently.txt python examples/evaluator-qualification/maintained/batch_differential.py --provider evidently --output /tmp/evidently-boundaries.json
PYTHONPATH=src python examples/evaluator-qualification/maintained/qualify_batch.py --provider evidently --output /tmp/evidently-batch-qualification

Use the matching lock and provider name for the other Python evaluators. Promptfoo runs the same commands with the repository Python environment and requires Node.js with npm and npx; its runner verifies the exact package version and registry integrity in locks/promptfoo.txt. Every output path must be new. The differential observation covers 34 boundary cases, including explicitly unsupported inputs; it always has authority: none and returns exit 2 for drift. A candidate Python environment can run this observation, but the current qualification runner rejects a changed package/profile until its definition and source bindings are deliberately reviewed.

The five separate directories under maintained/artifacts/ retain actual pinned executions, native row projections, source/artifact hash manifests, and fresh 102-record qualifications. All five match the historical 102-score vectors and runtime-record digests. These are new scorer executions over the same captured outputs, without new model inference or signed transactions. The historical matrix below is not evidence that arbitrary domains or newer dependencies have been qualified.

Current scalar profiles

For new scalar evaluations, use the *-literal-scalar-v1 profiles in examples/evaluator-qualification/maintained/scalar-profiles.json and the qualify_scalar.py command. These profiles execute one native scalar call for each immutable, scheduled input/output/reference pair. Their package version, runner bundle, dependency declaration, scorer configuration and native module content are checked before any score is exported. They validate the returned numeric or Boolean type, metric name, success status and error fields where the upstream API provides them. A scalar return value does not supply native batch-row identity assurance.

Scroll horizontally to see every column.

The restrictions are defined per pair. Equal strings with leading whitespace remain valid for DeepEval; an empty LightEval output with a different reference remains a valid mismatch. Multiple references and non-string values are rejected before native execution. Hugging Face Evaluate loads metric code separately from its package, so the profile also binds that metric file's exact content hash. OpenAI Evals additionally checks its installed source revision.

Run a boundary audit in a fresh working directory, then independently qualify new outputs or the retained 102-record corpus:

repo_root="$PWD"
audit_dir="$(mktemp -d)"
cd "$audit_dir"
uv run --no-project --python python3.12 --with-requirements "$repo_root/examples/evaluator-qualification/locks/lm-evaluation-harness.txt" python "$repo_root/examples/evaluator-qualification/maintained/scalar_differential.py" --provider lm-evaluation-harness --output "$audit_dir/observation.json"
cd "$repo_root"
PYTHONPATH=src python examples/evaluator-qualification/maintained/qualify_scalar.py --provider lm-evaluation-harness --output /tmp/lm-literal-qualification

Choose the matching provider and lock for each row. The qualification command uses the invoking Python interpreter for its isolated upstream environment. Use make evaluator-scalar-semantics PYTHON=python3.12 to audit all eleven providers in a temporary working directory with their matching dependency locks. Pass --cases cases.json --schedule schedule.json to qualify a new evaluation; both inputs are frozen before execution, and the destination must be new. Atomic publication refuses to replace even a directory created concurrently.

The 39-case differential checks both historical and current native semantics, including whitespace, case, Unicode, empty strings, multiple targets and NUL characters. Its observations always have authority: none; candidate package or module changes cannot become qualified through the audit command. Fresh qualification separately enforces the complete current profile and returns independently recomputed per-record results. All eleven retained current profiles reproduce the historical 102-score vectors and runtime-record digest: 61 matches and 41 mismatches. Their source/artifact manifests and native observations live in the corresponding maintained/artifacts/ directories. These executions reuse captured model outputs without new model inference or new signed transactions.

Application evaluation SDKs

Benchmark harnesses

Evaluation and observability platforms

General metric libraries

Security and red-team evaluators

Each row under examples/evaluator-qualification/artifacts/ retains:

• a profile binding the upstream package, project, runner bundle, dependency declaration, and authority classification;
• the normalized upstream execution output, including the resolved Python package inventory where applicable;
• an export binding the profile, independent schedule, raw output, runner, and dependency declaration by SHA-256; and
• the independently recomputed qualification result.

Native outputs may contain timestamps, temporary paths, run IDs, or large payloads. The example runner retains a deterministic projection of the native result rather than claiming byte-for-byte preservation of those volatile fields. The reviewed runner bundle and its digest define that projection.

Run the retained, network-free verification:

make evaluator-qualification

This verifies the historical matrix profiles and rechecks every retained independently replayable import. The following command reproduces the two historical corpora with their original runner identities and refreshes those retained artifacts:

make evaluator-upstream-qualification

Use the current Inspect, batch or scalar commands above for new evaluations. The historical reproduction command requires uv, Node.js with npx, and network access on a cold cache. The retained model outputs do not require a model during evaluator execution.

To independently regenerate and compare the 102 model outputs, make the pinned model snapshot available locally and run the repository's locked Hugging Face environment:

make evaluator-replayable-corpus

The model corpus binds the immutable model revision, curated snapshot-tree digest, exact generation settings, dataset digest, generation package versions, and every output. This command performs model execution; ordinary offline qualification does not.

Claim boundary

The independently replayable demonstrations prove full per-record exact-match imports for one real, pinned model evaluation. They do not demonstrate every metric, task type, hosted mode, or model-judge feature offered by those evaluators. They also do not turn example runners into engine plugins.

Profiles marked Retained in the matrix separately carry model execution and signed release-assurance evidence with an explicit record count. A replayable row without a retained package begins with the already authenticated model-output corpus and demonstrates evaluator execution, normalization, qualification, and strict runtime-import replay.

Authority rules

deterministic_per_record recomputes the supported metric declared by the profile. The export must cover the independent schedule in exact order, bind each input and output, and carry successful record status. Exact match compares output_sha256 == reference_output_sha256 for every record. Normalized match, numeric tolerance, structured fields and token F1 require independently supplied reference text whose digest agrees with the schedule. Their metric configuration also forms part of the qualified profile. InvarLock rejects aggregate substitution and disagreement between an upstream-reported score and independent replay. The retained evaluator demonstrations continue to use their declared exact-match profiles; support for other metrics does not change those historical semantics.

observation_only requires one explicit reason:

• aggregate_only
• human_judgment
• nondeterministic_judge
• unsupported_replay_semantics

An observation-only result contains no import records and cannot be promoted with a CLI flag. --require-verdict-authority instead rejects it.

Proprietary evaluator path

A private SDK, CLI, or HTTP-backed evaluator uses the same boundary:

1. Execute the proprietary evaluator outside InvarLock.
2. Retain its response bytes or a reviewed deterministic projection as upstream-output.json.
3. Create a canonical profile with package ecosystem private, its immutable version, project URI, the runner-bundle digest, and dependency-declaration digest.
4. Normalize complete ordered records into invarlock/evaluator-qualification-export-v1, or declare the result observation-only when replay is unavailable.
5. Qualify the four inputs through the CLI or Python SDK.

invarlock-qualify-evaluator qualify \
  profile.json schedule.json export.json upstream-output.json \
  --output qualification-result.json \
  --require-verdict-authority --json

The runner can call a Python SDK, spawn a proprietary CLI, or call an API. That transport is outside the core contract. Do not put credentials, endpoint URLs with secrets, request headers, or private paths in retained public artifacts. The profile and export must bind the immutable evaluator identity and the exact runner/dependency inputs used to produce the normalized evidence.

The JSON schemas are evaluator_qualification_profile.schema.json, evaluator_qualification_schedule.schema.json, evaluator_qualification_export.schema.json, and evaluator_qualification_result.schema.json. Package-owned copies ship with InvarLock and are the validation authority.