Evaluator qualification

InvarLock qualifies evaluator evidence through one closed, evaluator-neutral boundary. qualify_evaluator_export consumes normalized contracts without dispatching to an evaluator SDK. Example-owned runners execute representative upstream tools and normalize their results into the same four contracts.

Reference

Surface: Four canonical JSON contracts, the invarlock-qualify-evaluator companion CLI, and invarlock.engine.qualify_evaluator_export

Stability: Closed v1 qualification formats

Use this page when: Integrating an open or proprietary evaluator, or reviewing the maintained upstream-execution matrix

A matrix row demonstrates the named version and entry point; it does not make that evaluator a built-in InvarLock plugin.

The captured-results integration adds installed parsers for selected native export shapes and a multi-metric comparison behind the core evaluation commands. The qualification boundary documented here also accepts normalized matching, numeric tolerance, structured fields and token F1 when independent reference text is available for recomputation. Existing retained matrix rows continue to prove their declared exact-match profiles; they are not evidence for every new metric.

Capture for exact match, likelihood and judge scoring

Dedicated native mappings cover all 19 ecosystems below. Export supported JSON cases, reports, prediction tables or attempts from your existing environment, then import them with adapter: evaluator-native-json. The optional installed invarlock.engine.export_evaluator_result helper prepares a common envelope for adapter: evaluator-json when SDK/core dependencies permit co-installation. Every profile uses the same captured evaluate, independent verify and report flow, with per-case metadata slices and attributed numeric metrics when supplied. Recipients need no evaluator SDK or account. The native shape recipes and public API example describe the capture boundary. Canonical JSON, generic JSONL and the existing Inspect, Harness, Promptfoo and Langfuse parser names remain available.

The evidence sources have different scopes. Tests that place retained model answers and likelihoods, plus synthetic complete judge-call fixtures, into each native shape exercise the shared export/import, scoring and recipient flow; they do not establish that each SDK measured those facts. Pinned SDK checks also serialize these retained records through actual upstream data objects and serializers, then pass the resulting files through a separately installed recipient for all three scorers and both import routes. Local evaluator smokes exercise their public APIs without a new model campaign. SDK-required wrapper fields are identified as serialization fixtures, not extra measurements. The historical qualification matrix retains its original 17 per-case deterministic exports and two observation-only profiles: MLflow aggregate results and Garak detector summaries. Their identities and authority do not change. New native MLflow prediction-table and Garak attempt capture paths require the actual rows; the historical summaries cannot supply them.

The fresh Mistral 7B sentinel executes both models through all 19 maintained profiles, retaining 304 model executions and 114 scorer/import journeys. Eight fixed cases cover narrative continuation, answerable questions and unanswerable questions. Its 992 usable judge ratings were collected against those retained answers; the original 32 blocked attempts remain separate. Independent recipients authenticate and replay the resulting comparisons. These small-sample integration checks preserve policy failures and insufficient-evidence outcomes; they do not establish production model quality, hosted storage support or arbitrary evaluator configurations.

The priority workflow reference extends Inspect, Harness, Promptfoo and Langfuse with 64-case local comparisons and eight-case controlled HTTP-service comparisons. Its 576 model executions support 32 independently verified exact-match/NLL comparisons and 24 judge comparisons, including reference-free, repetition and budget controls. All 1,288 admitted judge calls produced usable ratings; four stop/resume checks preserved the original checkpoint bytes. The HTTP profile qualifies the declared task service, not arbitrary cloud APIs or hosted evaluator storage. Policy rejections and insufficient-evidence outcomes remain visible in the retained references.

InvarLock owns the selected scorer. Exact match requires string answers and references. Judge requires task and answer text plus a declared recipe and complete retained judge calls; a scalar rating is insufficient. Structured task inputs require an explicit text projection that preserves the original facts. Per-case judge references are opt-in and stay separate from the model input. Normalized NLL requires typed reference-continuation log probabilities, byte and token counts, and bound source, artifact, configuration and tokenizer identities. Naming an evaluator cannot supply these facts.

The Mistral 7B likelihood reference retains actual baseline-to-Instruct measurements for 400 paired narrative continuations, both model identities, and signed evaluation and independent recipient replay. The smaller Harness likelihood control retains the earlier six same-model CPU pairs. Both establish their declared measurement profiles. The 19-profile contract replays adapt retained facts and source bindings into each native shape to test the shared interface; they do not claim new SDK likelihood measurements. Their synthetic complete judge calls exercise recipe and replay contracts, not a newly measured judge campaign.

The installed gate also evaluates unchanged real Luna measurements from the grounded-QA held-out study (422 cases, 2,532 ratings, pass) and corrected pilot (40 cases, 240 ratings, insufficient evidence). Both complete preflight, signed evaluation, independent verification, reporting and measurement-tamper rejection. Their original canonical runs and policies remain bound to the ratings. These separate journeys establish real retained-judge execution; they do not relabel old calls as new evaluator-specific measurements. See the runnable real judge journey.

Three independent status axes

The repository does not assign one cumulative “integration level.” Support, replay authority, and retained transaction evidence answer different questions and cannot safely substitute for one another.

Table 1: Three independent status axes. Data table with columns: Axis, Values, What it proves
Axis	Values	What it proves
Adapter support	`maintained_adapter` or an external adapter	Whether this repository maintains the source-specific runner, dependency lock, and upstream entry point; support grants no replay authority
Replay authority	`deterministic_per_record` or `observation_only`	Whether complete ordered facts can be independently recomputed and imported, or only retained as authenticated context
Signed OCI-journey maturity	Retained with a named corpus and record count, or `None retained`	Whether a model-running, signed `evaluate` → `verify` → `report` OCI transaction has completed and been retained as release evidence

Scroll horizontally to see every column.

The stable qualification-result contract continues to emit outcome: qualified_for_import with authority: verdict_authority for an independently replayable result. It emits observation_only for both fields when replay is unavailable. Those wire values do not claim that InvarLock maintains the adapter or that a signed OCI journey has been demonstrated.

Qualification matrix

The small conformance corpus uses two local records: one exact match and one mismatch. It proves every maintained upstream entry point and both authority modes without downloading a model or calling a hosted evaluator.

The independently replayable rows also execute against a retained 102-record evaluation produced by the immutable Qwen/Qwen3.5-0.8B revision recorded in the corpus. The model produced 61 exact matches and 41 mismatches. Each evaluator scores all 102 model outputs through its real upstream entry point. InvarLock then recomputes every score and replays all 102 normalized records through the runtime-import authoring boundary.

Each row identifies a maintained evaluator entry point with retained upstream execution. The rows cover distinct evaluator ecosystems and workflows; matrix.json records their source, version, authority, and status metadata.

LM Evaluation Harness and Inspect AI additionally have independently replayable 102-record shared-output imports. Their retained current-model OCI journeys execute Qwen3.5 9B over the same ordered 400-record MMLU-Pro schedule. LM Evaluation Harness also retains a Gemma 4 12B instruction-to-QAT portability transaction. The other rows retain qualification and compatibility evidence without claiming a native signed journey.

The compact evidence packs, signed verifier receipts, independent policies, and builder-signed OCI attestations are retained under examples/evaluator-qualification/signed-transactions/ and replayed by make evaluator-qualification.

The 400-item semantic suite is selected deterministically from a revision- and hash-pinned TIGER-Lab MMLU-Pro dataset. It balances all 14 subject categories and answer labels A–J. Qwen and Gemma prompts derive from those same semantic records using separately pinned model-native no-thinking renderings. The 400-record size supports the maximum 10-percentage-point paired interval width declared in advance: Qwen achieved a 7.85-point width and Gemma achieved a 4.68-point width.

The retained flagship comparison confirms an identical ordered schedule and reports native agreement without creating another acceptance verdict. LM Evaluation Harness and Inspect AI agree on every baseline and subject output and score across all 800 side records. The shared-output replay and native signed journeys answer complementary questions: the first isolates evaluator scoring across the wider matrix, while the second exercises both flagship frameworks' real model paths.

The matrix represents the Microsoft PromptFlow lineage with Azure AI Evaluation rather than preserving the deprecated promptflow-evals package as a second legacy row. The OpenAI Evals qualification profile is installed from an immutable source revision, while its signed transaction integration runs the unchanged upstream basic.Match evaluator from an explicitly versioned wheel derived from the hash-pinned evals==3.0.1.post1 input. The isolated image removes the unused NLTK dependency; its narrower dependency set does not change the retained qualification profile.

The generated matrix below describes the retained generic qualification profiles. The signed bridges execute LM Evaluation Harness and the native Inspect task (inspect_ai.eval plus inspect_ai.scorer.match) in clean OCI transactions. Their build attestations, worker protocol, and native adapters remain example-owned; the installed core receives evaluator-neutral runtime-import and signed transaction contracts.

The signed bridges retain native evaluator facts but keep the transaction metric independent. Inspect's pinned causal HF decoder has an explicit authenticated boundary recovery, and InvarLock replays byte-exact equality for the signed acceptance decision after checking each evaluator's native event semantics.

Inspect literal-pair boundary

Fresh literal qualification uses the separate inspect-ai-literal-pairs-v1 profile in examples/evaluator-qualification/maintained/inspect-profile.json. Its pinned scorer is Inspect 0.3.254 match(location="exact", ignore_case=False, numeric=False) with one string target. Inspect strips boundary whitespace and ASCII punctuation even with these settings, so its native notion of exact matching can differ from literal equality.

The supported domain is defined for each output/reference pair. Literal equality must agree with equality after Inspect's boundary stripping. Equal strings containing leading whitespace or punctuation remain supported, as do distinct strings that remain distinct after stripping. Unequal strings that collapse together, such as " answer" and "answer", fail explicitly. Array or multiple targets are outside this profile. Native score, answer and explanation contradictions also fail before an export is written. The independent qualification metric remains byte-exact equality.

Before scoring, the current runner checks the complete supplied public profile against its local, source-bound definition: package version, dependency lock, runner bundle, upstream URL and authority must all agree. Even a matching candidate package installation cannot reuse this profile identity. The runner also freezes its inputs before scoring and publication.

Run the pinned 28-case differential corpus and a fresh qualification of the retained 102 model outputs with:

uv run --no-project --with-requirements examples/evaluator-qualification/locks/inspect-ai.txt python examples/evaluator-qualification/maintained/inspect_differential.py --output /tmp/inspect-boundaries.json
PYTHONPATH=src python examples/evaluator-qualification/maintained/qualify_inspect.py --output /tmp/inspect-literal-qualification

Both output paths must be new. The differential covers boundary and internal whitespace and punctuation, case, Unicode, empty outputs and multiple targets. It reports semantic drift with exit 2 and always declares authority: none. Running it with a candidate dependency version cannot promote that dependency to a qualified profile. The fresh qualification command separately executes the pinned upstream scorer and independently checks the resulting export. This is scoring of existing outputs, without new model inference or a new signed OCI transaction.

The separate artifacts under examples/evaluator-qualification/maintained/artifacts/inspect-ai-literal-pairs-v1/ retain the 28-case observation, the fresh 102-record qualification and a source hash manifest. All 102 scores and the normalized runtime-record digest agree with the historical Inspect import: 61 matches and 41 mismatches.

The original inspect-ai profile, runner bytes, qualification artifacts and signed transaction outcomes remain unchanged and replay under their original identities. They are not relabeled as evidence for the new runner. The matrix below continues to describe those retained identities.

Current strict batch profiles

Fresh batch qualification uses the separately named *-strict-batch-v1 profiles in examples/evaluator-qualification/maintained/batch-profiles.json. These profiles preserve the original matrix's historical identities while adding strict validation of the actual native rows:

Table 2: Current strict batch profiles. Data table with columns: Evaluator, Pinned version, Current native path
Evaluator	Pinned version	Current native path
Promptfoo	0.121.19	Local echo provider with one string `equals` assertion
Evidently	0.7.21	`ExactMatch` descriptor over output and reference columns
Langfuse	4.14.1	Local experiment with one named Boolean equality callback
Azure AI Evaluation	1.18.1	Batch orchestration with explicitly mapped equality callback
Pydantic Evals	2.18.0	`EqualsExpected` with case identity and captured output in each input

Scroll horizontally to see every column.

Each supported record has one string output and reference, and a unique ID in the independent schedule order. Repeated prompts can carry different outputs; they do not overwrite one another. The runner rejects missing, duplicate, extra or reordered native rows, changed references or outputs, wrong metric configuration, score/detail contradictions, and aggregate substitution. Source inputs are copied before execution so a later source edit cannot change which records are exported. The complete supplied public profile must match its local definition, including package identity, dependency and runner hashes, upstream URL, and authority. Changing the installed package and supplied profile together does not bypass this check.

Literal equality preserves case, whitespace, punctuation, Unicode and empty strings. Promptfoo's echo path has additional renderer restrictions: an output ending in LF is rejected because the pinned variable renderer removes one final newline. Output or reference text containing template opening delimiters, file:// or package: is also rejected before execution. Leading whitespace and internal whitespace remain supported. Multiple targets and non-string outputs are outside all five profiles.

For example, run the pinned Evidently boundary audit and separately qualify the retained model outputs:

uv run --no-project --with-requirements examples/evaluator-qualification/locks/evidently.txt python examples/evaluator-qualification/maintained/batch_differential.py --provider evidently --output /tmp/evidently-boundaries.json
PYTHONPATH=src python examples/evaluator-qualification/maintained/qualify_batch.py --provider evidently --output /tmp/evidently-batch-qualification

Use the matching lock and provider name for the other Python evaluators. Promptfoo runs the same commands with the repository Python environment and requires Node.js with npm and npx; its runner verifies the exact package version and registry integrity in locks/promptfoo.txt. Every output path must be new. The separate SDK capture gate installs the locked dependencies needed for its Promptfoo profile from locks/promptfoo/package-lock.json with npm ci --omit=optional. Unused optional integrations are outside this SDK test. The gate checks the top-level archive against the same registry integrity pin. The isolated SDK environment pins the patched js-yaml 5.2.2 dependency, and its CI gate checks the lock for current high-severity advisories. The differential and retained qualification records still identify the original Promptfoo 0.121.19 package; the SDK gate's dependency override applies only to its disposable test installation. The differential observation covers 34 boundary cases, including explicitly unsupported inputs; it always has authority: none and returns exit 2 for drift. A candidate Python environment can run this observation, but the current qualification runner rejects a changed package/profile until its definition and source bindings are deliberately reviewed.

The five separate directories under maintained/artifacts/ retain actual pinned executions, native row projections, source/artifact hash manifests, and fresh 102-record qualifications. All five match the historical 102-score vectors and runtime-record digests. These are new scorer executions over the same captured outputs, without new model inference or signed transactions. The historical matrix below is not evidence that arbitrary domains or newer dependencies have been qualified.

Current scalar profiles

For new scalar evaluations, use the *-literal-scalar-v1 profiles in examples/evaluator-qualification/maintained/scalar-profiles.json and the qualify_scalar.py command. These profiles execute one native scalar call for each immutable, scheduled input/output/reference pair. Their package version, runner bundle, dependency declaration, scorer configuration and native module content are checked before any score is exported. They validate the returned numeric or Boolean type, metric name, success status and error fields where the upstream API provides them. A scalar return value does not supply native batch-row identity assurance.

Table 3: Current scalar profiles. Data table with columns: Provider name, Pinned version, Current literal domain
Provider name	Pinned version	Current literal domain
`lm-evaluation-harness`	0.4.12	Explicit NumPy object arrays preserve trailing NUL characters
`deepeval`	4.1.3	Nonempty outputs; literal equality must agree with boundary-whitespace stripping
`ragas`	0.4.3	One string output and reference
`lighteval`	0.13.0	Full matching without normalization; equal empty strings are unsupported
`hugging-face-evaluate`	0.4.6	The loaded metric file is content-bound; trailing-NUL equality collisions are unsupported
`autoevals`	0.3.0	One string output and reference
`openevals`	0.2.0	One string output and reference, with a named Boolean result
`openai-evals`	3.0.1.post1	Literal match function and the locked source revision
`arize-phoenix-evals`	3.3.0	One string output and reference
`opik`	2.2.7	Explicit case-sensitive matching with tracking disabled
`trulens`	2.9.0	Named local literal-equality function through the upstream Metric wrapper

Scroll horizontally to see every column.

The restrictions are defined per pair. Equal strings with leading whitespace remain valid for DeepEval; an empty LightEval output with a different reference remains a valid mismatch. Multiple references and non-string values are rejected before native execution. Hugging Face Evaluate loads metric code separately from its package, so the profile also binds that metric file's exact content hash. OpenAI Evals additionally checks its installed source revision.

Run a boundary audit in a fresh working directory, then independently qualify new outputs or the retained 102-record corpus:

repo_root="$PWD"
audit_dir="$(mktemp -d)"
cd "$audit_dir"
uv run --no-project --python python3.12 --with-requirements "$repo_root/examples/evaluator-qualification/locks/lm-evaluation-harness.txt" python "$repo_root/examples/evaluator-qualification/maintained/scalar_differential.py" --provider lm-evaluation-harness --output "$audit_dir/observation.json"
cd "$repo_root"
PYTHONPATH=src python examples/evaluator-qualification/maintained/qualify_scalar.py --provider lm-evaluation-harness --output /tmp/lm-literal-qualification

Choose the matching provider and lock for each row. The qualification command uses the invoking Python interpreter for its isolated upstream environment. Use make evaluator-scalar-semantics PYTHON=python3.12 to audit all eleven providers in a temporary working directory with their matching dependency locks. Pass --cases cases.json --schedule schedule.json to qualify a new evaluation; both inputs are frozen before execution, and the destination must be new. Atomic publication refuses to replace even a directory created concurrently.

The 39-case differential checks both historical and current native semantics, including whitespace, case, Unicode, empty strings, multiple targets and NUL characters. Its observations always have authority: none; candidate package or module changes cannot become qualified through the audit command. Fresh qualification separately enforces the complete current profile and returns independently recomputed per-record results. All eleven retained current profiles reproduce the historical 102-score vectors and runtime-record digest: 61 matches and 41 mismatches. Their source/artifact manifests and native observations live in the corresponding maintained/artifacts/ directories. These executions reuse captured model outputs without new model inference or new signed transactions.

Application evaluation SDKs

Upstream evaluator
Promptfoo
Pinned version
[email protected]
Executed upstream entry point
promptfoo eval
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
DeepEval
Pinned version
deepeval==4.1.3
Executed upstream entry point
deepeval.metrics.ExactMatchMetric.measure
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
Ragas
Pinned version
ragas==0.4.3
Executed upstream entry point
ragas.metrics.collections.ExactMatch.ascore
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
Pydantic Evals
Pinned version
pydantic-evals==2.18.0
Executed upstream entry point
pydantic_evals.Dataset.evaluate_sync/EqualsExpected
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
Braintrust AutoEvals
Pinned version
autoevals==0.3.0
Executed upstream entry point
autoevals.ExactMatch.__call__
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
OpenEvals
Pinned version
openevals==0.2.0
Executed upstream entry point
openevals.exact.exact_match
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
Azure AI Evaluation
Pinned version
azure-ai-evaluation==1.18.1
Executed upstream entry point
azure.ai.evaluation.evaluate
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained

Table 4: Application evaluation SDKs. Data table with columns: Upstream evaluator, Pinned version, Executed upstream entry point, Adapter support, Replay authority, Retained signed OCI transactions
Upstream evaluator	Pinned version	Executed upstream entry point	Adapter support	Replay authority	Retained signed OCI transactions
Promptfoo	`[email protected]`	`promptfoo eval`	Maintained	Independently replayable (102 shared outputs)	None retained
DeepEval	`deepeval==4.1.3`	`deepeval.metrics.ExactMatchMetric.measure`	Maintained	Independently replayable (102 shared outputs)	None retained
Ragas	`ragas==0.4.3`	`ragas.metrics.collections.ExactMatch.ascore`	Maintained	Independently replayable (102 shared outputs)	None retained
Pydantic Evals	`pydantic-evals==2.18.0`	`pydantic_evals.Dataset.evaluate_sync/EqualsExpected`	Maintained	Independently replayable (102 shared outputs)	None retained
Braintrust AutoEvals	`autoevals==0.3.0`	`autoevals.ExactMatch.__call__`	Maintained	Independently replayable (102 shared outputs)	None retained
OpenEvals	`openevals==0.2.0`	`openevals.exact.exact_match`	Maintained	Independently replayable (102 shared outputs)	None retained
Azure AI Evaluation	`azure-ai-evaluation==1.18.1`	`azure.ai.evaluation.evaluate`	Maintained	Independently replayable (102 shared outputs)	None retained

Benchmark harnesses

Upstream evaluator
LM Evaluation Harness
Pinned version
lm-eval==0.4.12
Executed upstream entry point
lm_eval.api.metrics.exact_match_hf_evaluate
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
Retained (2 signed transactions, 400 records each)
Upstream evaluator
Inspect AI
Pinned version
inspect-ai==0.3.254
Executed upstream entry point
inspect_ai.scorer.match
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
Retained (2 signed transactions, 400 records each)
Upstream evaluator
LightEval
Pinned version
lighteval==0.13.0
Executed upstream entry point
lighteval.metrics.metrics_sample.ExactMatches.compute
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
OpenAI Evals
Pinned version
source revision 8eac7a7 (3.0.1.post1)
Executed upstream entry point
evals.elsuite.modelgraded.classify_utils.MATCH_FNS['exact']
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained

Table 5: Benchmark harnesses. Data table with columns: Upstream evaluator, Pinned version, Executed upstream entry point, Adapter support, Replay authority, Retained signed OCI transactions
Upstream evaluator	Pinned version	Executed upstream entry point	Adapter support	Replay authority	Retained signed OCI transactions
LM Evaluation Harness	`lm-eval==0.4.12`	`lm_eval.api.metrics.exact_match_hf_evaluate`	Maintained	Independently replayable (102 shared outputs)	Retained (2 signed transactions, 400 records each)
Inspect AI	`inspect-ai==0.3.254`	`inspect_ai.scorer.match`	Maintained	Independently replayable (102 shared outputs)	Retained (2 signed transactions, 400 records each)
LightEval	`lighteval==0.13.0`	`lighteval.metrics.metrics_sample.ExactMatches.compute`	Maintained	Independently replayable (102 shared outputs)	None retained
OpenAI Evals	source revision `8eac7a7` (`3.0.1.post1`)	`evals.elsuite.modelgraded.classify_utils.MATCH_FNS['exact']`	Maintained	Independently replayable (102 shared outputs)	None retained

Evaluation and observability platforms

Upstream evaluator
MLflow Model Evaluation
Pinned version
mlflow==3.14.0
Executed upstream entry point
mlflow.models.evaluate
Adapter support
Maintained
Replay authority
Observation-only: aggregate only
Retained signed OCI transactions
None retained
Upstream evaluator
Arize Phoenix Evals
Pinned version
arize-phoenix-evals==3.3.0
Executed upstream entry point
phoenix.evals.metrics.exact_match
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
Langfuse
Pinned version
langfuse==4.14.1
Executed upstream entry point
langfuse.Langfuse.run_experiment
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
Opik
Pinned version
opik==2.2.7
Executed upstream entry point
opik.evaluation.metrics.Equals.score
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
Evidently
Pinned version
evidently==0.7.21
Executed upstream entry point
evidently.Dataset.from_pandas/Evidently ExactMatch
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained
Upstream evaluator
TruLens
Pinned version
trulens==2.9.0
Executed upstream entry point
trulens.core.Metric.__call__
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained

Table 6: Evaluation and observability platforms. Data table with columns: Upstream evaluator, Pinned version, Executed upstream entry point, Adapter support, Replay authority, Retained signed OCI transactions
Upstream evaluator	Pinned version	Executed upstream entry point	Adapter support	Replay authority	Retained signed OCI transactions
MLflow Model Evaluation	`mlflow==3.14.0`	`mlflow.models.evaluate`	Maintained	Observation-only: aggregate only	None retained
Arize Phoenix Evals	`arize-phoenix-evals==3.3.0`	`phoenix.evals.metrics.exact_match`	Maintained	Independently replayable (102 shared outputs)	None retained
Langfuse	`langfuse==4.14.1`	`langfuse.Langfuse.run_experiment`	Maintained	Independently replayable (102 shared outputs)	None retained
Opik	`opik==2.2.7`	`opik.evaluation.metrics.Equals.score`	Maintained	Independently replayable (102 shared outputs)	None retained
Evidently	`evidently==0.7.21`	`evidently.Dataset.from_pandas/Evidently ExactMatch`	Maintained	Independently replayable (102 shared outputs)	None retained
TruLens	`trulens==2.9.0`	`trulens.core.Metric.__call__`	Maintained	Independently replayable (102 shared outputs)	None retained

General metric libraries

Upstream evaluator
Hugging Face Evaluate
Pinned version
evaluate==0.4.6
Executed upstream entry point
evaluate.load('exact_match').compute
Adapter support
Maintained
Replay authority
Independently replayable (102 shared outputs)
Retained signed OCI transactions
None retained

Table 7: General metric libraries. Data table with columns: Upstream evaluator, Pinned version, Executed upstream entry point, Adapter support, Replay authority, Retained signed OCI transactions
Upstream evaluator	Pinned version	Executed upstream entry point	Adapter support	Replay authority	Retained signed OCI transactions
Hugging Face Evaluate	`evaluate==0.4.6`	`evaluate.load('exact_match').compute`	Maintained	Independently replayable (102 shared outputs)	None retained

Security and red-team evaluators

Upstream evaluator
Garak
Pinned version
garak==0.15.1
Executed upstream entry point
python -m garak
Adapter support
Maintained
Replay authority
Observation-only: unsupported replay semantics
Retained signed OCI transactions
None retained

Table 8: Security and red-team evaluators. Data table with columns: Upstream evaluator, Pinned version, Executed upstream entry point, Adapter support, Replay authority, Retained signed OCI transactions
Upstream evaluator	Pinned version	Executed upstream entry point	Adapter support	Replay authority	Retained signed OCI transactions
Garak	`garak==0.15.1`	`python -m garak`	Maintained	Observation-only: unsupported replay semantics	None retained

Each row under examples/evaluator-qualification/artifacts/ retains:

a profile binding the upstream package, project, runner bundle, dependency declaration, and authority classification;
the normalized upstream execution output, including the resolved Python package inventory where applicable;
an export binding the profile, independent schedule, raw output, runner, and dependency declaration by SHA-256; and
the independently recomputed qualification result.

Native outputs may contain timestamps, temporary paths, run IDs, or large payloads. The example runner retains a deterministic projection of the native result rather than claiming byte-for-byte preservation of those volatile fields. The reviewed runner bundle and its digest define that projection.

Run the retained, network-free verification:

make evaluator-qualification

This verifies the historical matrix profiles and rechecks every retained independently replayable import. The following command reproduces the two historical corpora with their original runner identities and refreshes those retained artifacts:

make evaluator-upstream-qualification

Use the current Inspect, batch or scalar commands above for new evaluations. The historical reproduction command requires uv, Node.js with npx, and network access on a cold cache. The retained model outputs do not require a model during evaluator execution.

To independently regenerate and compare the 102 model outputs, make the pinned model snapshot available locally and run the repository's locked Hugging Face environment:

make evaluator-replayable-corpus

The model corpus binds the immutable model revision, curated snapshot-tree digest, exact generation settings, dataset digest, generation package versions, and every output. This command performs model execution; ordinary offline qualification does not.

Claim boundary

The independently replayable demonstrations prove full per-record exact-match imports for one real, pinned model evaluation. They do not demonstrate every metric, task type, hosted mode, or model-judge feature offered by those evaluators. They also do not turn example runners into engine plugins.

Profiles marked Retained in the matrix separately carry model execution and signed release-assurance evidence with an explicit record count. A replayable row without a retained package begins with the already authenticated model-output corpus and demonstrates evaluator execution, normalization, qualification, and strict runtime-import replay.

Authority rules

deterministic_per_record recomputes the supported metric declared by the profile. The export must cover the independent schedule in exact order, bind each input and output, and carry successful record status. Exact match compares output_sha256 == reference_output_sha256 for every record. Normalized match, numeric tolerance, structured fields and token F1 require independently supplied reference text whose digest agrees with the schedule. Their metric configuration also forms part of the qualified profile. InvarLock rejects aggregate substitution and disagreement between an upstream-reported score and independent replay. The retained evaluator demonstrations continue to use their declared exact-match profiles; support for other metrics does not change those historical semantics.

observation_only requires one explicit reason:

aggregate_only
human_judgment
nondeterministic_judge
unsupported_replay_semantics

An observation-only result contains no import records and cannot be promoted with a CLI flag. --require-verdict-authority instead rejects it.

Proprietary evaluator path

A private SDK, CLI, or HTTP-backed evaluator uses the same boundary:

Execute the proprietary evaluator outside InvarLock.
Retain its response bytes or a reviewed deterministic projection as upstream-output.json.
Create a canonical profile with package ecosystem private, its immutable version, project URI, the runner-bundle digest, and dependency-declaration digest.
Normalize complete ordered records into invarlock/evaluator-qualification-export-v1, or declare the result observation-only when replay is unavailable.
Qualify the four inputs through the CLI or Python SDK.

invarlock-qualify-evaluator qualify \
  profile.json schedule.json export.json upstream-output.json \
  --output qualification-result.json \
  --require-verdict-authority --json

The runner can call a Python SDK, spawn a proprietary CLI, or call an API. That transport is outside the core contract. Do not put credentials, endpoint URLs with secrets, request headers, or private paths in retained public artifacts. The profile and export must bind the immutable evaluator identity and the exact runner/dependency inputs used to produce the normalized evidence.

The JSON schemas are evaluator_qualification_profile.schema.json, evaluator_qualification_schedule.schema.json, evaluator_qualification_export.schema.json, and evaluator_qualification_result.schema.json. Package-owned copies ship with InvarLock and are the validation authority.