Evaluation Records

Reference

Surface: Captured evaluation records and evidence bindings.

Stability: Public core contract for captured evaluation.

Use this page when: Implementing or reviewing captured evidence.

Captured comparisons use neutral evaluation record contracts. A record keeps its stable ID, input, expected value, output, score provenance, metadata, and error state. Baseline and subject records must use the same complete schedule.

The public request contract is invarlock/evaluation-request-v2. Deterministic and recorded-score captured comparisons use directory pack invarlock/evidence-pack-v2 and an external invarlock/evidence-verification-receipt-v3. The pack has one manifest signature, not a second signed record envelope. verify checks recipient-owned policy, complete-run and normalized-request digests, and signer identity. report is a deterministic presentation step and does not confer verification authority.

The captured comparison binds bindings.baseline, bindings.subject, and bindings.policy. Metric results use baseline_mean and subject_mean; absolute policy bounds use subject_minimum and subject_maximum. The captured SDK helper is compare_runs(baseline=..., subject=..., policy=...), with no alternate role keywords or wire-field aliases. These captured fields do not rename native historical reports or third-party exports. Native pack v1 and receipt v1/v2 contracts are unchanged. Selecting comparison.metric: judge instead publishes judge evidence with its own measurement and recipient trust contracts; see judge measurements.

For the exact JSON schemas, see the files under contracts/ and their synced copies under src/invarlock/_data/contracts/.

Record contracts

Scroll horizontally to see every column.

Runs require format, source (name, version), run_id, artifact_digest, source_digest, score_provenance, and records. Every record has id, input, expected, output, scores, metadata, error, and context. Null output or an explicit error is retained, not silently dropped. Likelihood scoring can use a null output when valid reference-continuation facts are present. Matching IDs must bind matching input, reference and metadata. Duplicate IDs or changed paired facts are integration errors. Explicit errors, missing recorded scores and missing required likelihood facts produce insufficient_evidence; cases are not silently discarded.

An incorrect exact-match answer scores zero; other scorers, such as token F1, can award partial credit. Neither is a missing record.

For captured comparisons, exact_match, normalized_match, and token_f1 require string outputs on both sides; null and other non-string outputs are unavailable. Empty strings are present answers and can score zero or one. numeric_tolerance treats null output as unavailable, but a present incorrect or malformed numeric answer still scores zero. JSON scorers accept JSON null as a value: use an explicit row error to distinguish a failed capture from a submitted JSON null. Recorded metrics require their score key, including valid zero scores; likelihood metrics require bound likelihood facts, not answer text. Any unavailable pair makes its metric and slice insufficient, retains its ID in missing_ids, and leaves that metric's means and interval unset. Other complete metrics and slices are still evaluated.

Exact-match capability reporting and comparison share the output-availability rule. The standalone deterministic score function retains its existing zero-for-invalid-answer behavior; native and judge scoring are unchanged. Fresh captured evidence uses the corrected comparison during independent replay. An older pack whose stored comparison counted such answers as usable fails replay consistency (verification exit 6); preserve it and create a new assessment from the retained runs instead of editing signed bytes. Authenticating an older receipt's signature does not rerun that comparison or revise its historical verdict. No format identifiers or scorer descriptors change.

artifact_digest attributes the run to a supplied artifact identity; it does not prove inference. source_digest binds raw imported bytes when applicable and is part of the complete run. Canonical run_digest covers the whole normalized run, including provenance and all records, not just its model or schedule. Do not substitute a raw export checksum for a complete-run pin.

load_run supports invarlock, jsonl, evaluator-native-json, evaluator-json, inspect-json, lm-eval-samples, promptfoo-jsonl, and langfuse-json. These adapters normalize recorded outputs, never register a runtime provider or import execution authority. Original upstream bytes and their source_digest remain unchanged. A canonical invarlock run supplies its own metadata; external adapters require explicit source, run ID and artifact or hosted service identity, with approved score provenance when recorded metrics are used. External captured request sources accept artifact_digest: null with the complete service_identity descriptor. Canonical adapter: invarlock runs carry their own descriptor and reject source identity overrides. The external parser also binds the original file bytes through source_digest.

Dedicated evaluator exports

For an existing pipeline, save its native cases or results as JSON and use adapter: evaluator-native-json in the captured request. Dedicated mappings cover all 19 maintained ecosystems. Your pipeline keeps its current evaluator environment; evaluation, verification and reporting run in a separate core InvarLock environment without the upstream SDK. No common export envelope or SDK-side InvarLock installation is required; the file follows its evaluator's documented JSON profile.

For example, use Ragas's own explicit serializer after retaining your SingleTurnSample objects and their planned IDs:

import json

rows = [{"id": case_id, "sample": sample.model_dump(mode="json")}
        for case_id, sample in captured_samples]
if [row["id"] for row in rows] != planned_ids:
    raise ValueError("capture differs from the complete planned schedule")
with open("subject-native.json", "x", encoding="utf-8") as stream:
    json.dump(rows, stream, ensure_ascii=False, allow_nan=False)

captured_samples and the independently frozen planned_ids belong to your workflow. Export the baseline in the same way, preserving failed cases. Declare the raw file, actual SDK version, run ID and evaluated model identity in each captured request source:

path: subject-native.json
adapter: evaluator-native-json
source: {name: ragas, version: "0.4.3"}
run_id: subject-campaign
artifact_digest: "<actual model artifact digest>"

Use the version actually captured. For hosted models, declare artifact_digest: null and the complete service descriptor. Pin an independently reviewed expected_case_set_digest in policy to reject missing or changed scheduled cases, and an approved expected_run_digest on each source when required. Raw-file digests bind the imported bytes; the source name and version remain explicit attribution rather than proof of execution.

When the evaluator SDK and InvarLock dependency requirements are compatible, the shared Python exporter also accepts native SDK objects directly:

from importlib.metadata import version
from invarlock.engine import export_evaluator_result, evaluator_input_capabilities

run = export_evaluator_result(
    "ragas",
    [{"id": case_id, "sample": sample} for case_id, sample in captured_samples],
    "subject-export.json",
    expected_ids=planned_ids,
    source_version=version("ragas"),
    run_id="subject-campaign",
    artifact_digest=model_artifact_digest,
)
print(evaluator_input_capabilities(run))

This convenience API checks complete case membership before writing, refuses an existing destination, and returns a validated canonical run. Its output is an invarlock/evaluator-export-v1 envelope consumed with adapter: evaluator-json, using the same source/run/model declarations. For hosted models pass artifact_digest=None, service_identity=identity. Both adapters use the same source-specific mapping and downstream scorers.

Keep pinned MLflow 3.14.0 in a separate environment: it requires cryptography<49, while core InvarLock requires cryptography>=50. Export the original prediction table as JSON and use evaluator-native-json; no core dependency change is needed. SDK serializer smoke tests run from source with minimal test dependencies and do not establish SDK/core co-installation.

The native shape recipes cover Harness, Inspect, Promptfoo, DeepEval, Ragas, LightEval, Hugging Face Evaluate, Pydantic Evals, AutoEvals, OpenEvals, MLflow, Garak, OpenAI Evals, Phoenix Evals, Langfuse, Opik, Azure AI Evaluation, Evidently and TruLens. With compatible co-installation, Inspect EvalLog and Langfuse ExperimentResult objects can be passed directly to the Python exporter. The raw-file route uses Inspect's JSON log or bare public Langfuse ExperimentResult fields, including item_results; it does not require a Langfuse-specific InvarLock envelope. Original outputs can be captured before upstream grading. Scalar SDK wrappers accept an optional metric_result; no extra evaluator call is required to use InvarLock's scorers. Reference fields may be omitted for reference-free judge tasks where the native profile permits them; exact match and NLL still require a reference. Structured outputs remain JSON for compatible policies; text-only scorers do not stringify them. Aggregate-only reports still need the original per-case prediction table or attempts.

Per-case string metadata supplies policy slices. Optional metadata.invarlock_scores selects named finite numeric observations; native per-case numeric metrics are also retained where the profile defines them. Recorded-score policies require explicit score_provenance at export/import and matching accepted_provenance in policy. Numeric metrics do not become likelihoods or judge calls. Preserve actual continuation facts under metadata.invarlock_likelihood for NLL, and supply the separate complete judge measurement contract for judging. See scorer selection and policy requirements.

Langfuse experiments

langfuse-json imports the explicit invarlock/langfuse-export-v1 capture produced from a Langfuse Python experiment. It does not parse arbitrary dashboard exports. The maintained exporter uses langfuse==4.14.1; the installed parser, verification and reporting need no Langfuse SDK, account or network access.

Preserve a stable item ID and the complete independently selected case list. The exporter rejects missing or duplicate results instead of accepting only successful tasks. Structured inputs require an explicit text projection for judge scoring. Exact match needs original answers and references; NLL requires explicit continuation measurements; judging requires a recipe and complete judge measurements. Numeric Langfuse scores alone do not establish those facts.

Follow the Langfuse handoff example for capture, source identities, failed records and scorer-specific requirements.

Hosted service identity

A canonical run may include service_identity only with artifact_digest: null. Local artifact runs retain their existing digest contract and need no new field. The service descriptor is closed and requires every field below; unavailable observed model or exposed revision values are explicitly null.

The complete descriptor is limited to 1 MiB. Unknown fields, an inconsistent configuration digest, invalid timestamps or a reversed window are rejected. Identity strings reject ASCII control characters and all-whitespace values. Fractional seconds, when present, contain one through six digits; the start may equal the end. invarlock.engine.make_run and capture_evaluator_run accept artifact_digest=None, service_identity=identity. The configuration digest is computed over the configuration object; the service identity digest is computed over the complete descriptor. run_digest additionally binds all records and run provenance. These are different identities and cannot substitute for one another. The public invarlock.engine.validate_service_identity(identity) validates the descriptor. invarlock.engine.evaluated_subject_digest(run) returns the hosted descriptor digest or, for a local run, its explicit artifact digest; it does not fall back from a malformed hosted identity. Neither helper authorizes the subject or replaces a complete-run pin.

For example, given a reviewed descriptor and explicitly mapped records:

from invarlock.engine import capture_evaluator_run, digest

identity["configuration_digest"] = digest(identity["configuration"])
run = capture_evaluator_run(
    rows,
    source={"name": "service-capture", "version": "1"},
    run_id="subject-campaign",
    artifact_digest=None,
    service_identity=identity,
)

The variables identity and rows must contain the complete descriptor and actual captured facts. Canonical source bytes and complete-run anchors authenticate what was supplied; a service label, revision string or digest of the descriptor does not identify immutable model weights or attest execution. A hosted run uses the captured request and verification path. It does not register a native runtime provider or require provider access during offline replay. See the requalification guide for campaign preparation and interpretation.

Capture and explicit input projections

invarlock.engine.capture_evaluator_run accepts explicitly mapped per-case rows from any evaluator, with its actual source name/version, run ID and supplied artifact digest, or artifact_digest=None with a hosted service_identity. evaluator_input_capabilities reports usable counts and unavailable IDs for exact_match, normalized_nll_per_utf8_byte and judge. Availability describes input facts; it grants neither qualification nor proof of an upstream scorer execution.

For a structured input, input_projection may select an existing string with {"kind": "json-pointer", "pointer": "/input/question"}. Pointers must be rooted at /input or /context and select a string. The capture retains the original input and context plus the exact projection binding. Paired runs must agree on the original input and projection configuration as well as the selected text. Objects and arrays are never implicitly converted to text. External adapter sources may declare the same projection in a captured request. Canonical adapter: invarlock runs already contain their bindings and reject source overrides, including a second projection.

Set comparison.metric to exact_match, normalized_nll_per_utf8_byte or judge to select a scorer explicitly. Exact match and NLL must agree with the single metric in the comparison policy. Omit this selector for an ordinary multi-metric comparison. Judge uses a separate recipe and retained-call path.

Captured reference likelihoods

A record may additionally retain likelihood with these closed fields:

These facts must originate from an actual likelihood measurement. Generated answer probabilities and summary losses have different semantics. With an input projection, input_digest still identifies the preserved original input.

The NLL policy metric uses direction: lower, unit: nats_per_utf8_byte, aggregation: mean and ratio_max in place of maximum_regression. Its configuration requires configuration_digest, baseline_tokenizer_digest and subject_tokenizer_digest. For hosted NLL, the policy has one shared configuration_digest: both service descriptors and each likelihood row must match it. Different service configurations are not admitted by this profile; separate model, observation-window and tokenizer identities may still differ. The score per record is -logprob_sum / utf8_byte_count; the comparison is the ratio of subject and baseline means, with the same paired resampling method as native NLL. Interval width is measured in ratio units. Reference and configuration bindings and the declared tokenizer identities are checked before scoring. Replay does not execute the model or tokenizer, so their measurement claims remain source assertions. The retained Mistral 7B sentinel contains fresh generation and reference-likelihood measurements across all 19 maintained evaluator profiles, both import routes and two model artifacts. The Harness likelihood reference is a smaller six-pair, same-model CPU control. Each reference qualifies its recorded profile; synthetic contract tests and exact-match results alone do not establish likelihood measurement support.

Judge selection uses the separate judge recipe and retained measurement contract through the same captured request. It does not reinterpret a recorded scalar score as a native judge trial. prompt.reference_mode: per_case explicitly adds the frozen string reference to each judge request as a separate field. Omitted or none preserves the reference-free behavior. The reference never becomes part of the evaluated model input. See judge measurements.

Policy and identity

Policies contain 1..16 metrics and up to 16 metadata slices in addition to the overall scope. Each metric selects a unique name, kind, closed configuration, direction (higher/lower), unit, mean aggregation, minimum count, maximum regression and maximum interval width; NLL replaces maximum regression with ratio_max. Optional absolute limits on the subject mean are subject_minimum and subject_maximum.

Built-in captured kinds are exact_match, normalized_match, numeric_tolerance, json_exact, json_fields, token_f1, and normalized_nll_per_utf8_byte. Exact match and normalized NLL share native scorer semantics while retaining captured-input assurance. normalized_match and token_f1 pin unicode_version; a recipient with an incompatible Unicode environment refuses replay without issuing a receipt. recorded instead selects a score_key and exact accepted_provenance (kind, source, version, unit, rubric_digest). The verifier authenticates attributed values and paired arithmetic; it does not reproduce or authenticate the original scoring process.

Recorded scores are converted to binary64 floats before comparison. Large JSON integers can lose distinctions during conversion, and subtracting rounded means can differ from the exact mean of paired differences. Finite inputs can also overflow intermediate sums or differences; these comparisons raise EvaluationRecordsError with a finite-numeric-range message. The captured CLI returns a structured failure without publishing evidence. This refusal does not imply that the exact mathematical mean is infinite.

Normalized NLL uses binary64 arithmetic too: subnormal means can underflow and alter a ratio. Every sampled baseline mean must be positive, so even identical [0, 1] inputs can be refused when a bootstrap draw contains only zeros. Failed draws are not discarded or retried. These comparisons do not promise exact integer or real arithmetic.

An independent planned case set can be pinned in policy as expected_case_set_digest. Use freeze_case_set, case_set_digest, and validate_run_case_set to require the reviewed schedule, not merely agreement between two possibly incomplete captures.

normalize_captured_request strips physical source/output paths and binds each adapter, complete run digest, explicit expected_run_digest when authored, source overrides, and policy digest. captured_request_digest hashes those canonical portable bytes. An explicit run pin is semantic and is not dropped even when equal to the derived run digest. Relocating a request tree changes no portable identity; changing a source, provenance, policy or pin can change it. Use these SDK helpers, not a handwritten YAML/JSON hash recipe.

Scoring equivalence and input identity

A scorer can treat two answers as equivalent while their complete-run pins remain different. Apply each transformation at its own boundary:

• Change to a validated canonical run

  Reorder JSON object keys only

  Scoring effect

  Unchanged

  Complete-run pin

  Unchanged

  Planned case-set pin

  Unchanged

• Change to a validated canonical run

  Reorder records, keeping the same IDs and facts

  Scoring effect

  Captured comparison still pairs by ID

  Complete-run pin

  Changes

  Planned case-set pin

  Unchanged; cases are ordered canonically by ID

• Change to a validated canonical run

  Change an output's letter case or whitespace under normalized_match with case folding enabled

  Scoring effect

  Unchanged when normalized text is identical

  Complete-run pin

  Changes

  Planned case-set pin

  Unchanged; outputs are not case-set inputs

• Change to a validated canonical run

  Make the same equivalent reference-text change on both sides under that scorer

  Scoring effect

  Per-record score unchanged

  Complete-run pin

  Changes

  Planned case-set pin

  Changes; references are case-set inputs

• Change to a validated canonical run

  Change metadata on both sides

  Scoring effect

  Can change slice membership; an unused key does not affect scores

  Complete-run pin

  Changes

  Planned case-set pin

  Changes

• Change to a validated canonical run

  Change input text on both sides without changing retained answers

  Scoring effect

  Scoring retained answers can stay unchanged; this does not establish equivalent model behavior

  Complete-run pin

  Changes

  Planned case-set pin

  Changes

These rows assume all other run fields remain fixed. A raw export's physical checksum can change when its JSON formatting changes, even if the parsed object is equivalent. If an import retains that new checksum in source_digest, the complete-run pin changes too. These captured-run rules do not relax native provider requirements for an identical ordered schedule.

Keep scorer normalization inside scoring. Do not normalize retained answers or references merely to preserve an old pin. A changed complete run requires its own reviewed identity, even if every score and the final verdict stay the same.

Explain an additive score change

The standalone examples/score_contributions.py recipe produces a descriptive breakdown without changing a report, policy, interval or signed evidence. Run its synthetic example from the checkout:

python examples/score_contributions.py

For a complete additive scalar metric, supply one PairedCase per case and the reported delta to mean_contributions. Convert retained captured scores with Fraction.from_float(float(score)). Give each case its own unit for the ordinary equal-case mean. For judge scores, use Fraction(decimal_score_string), average repetitions within each case first, and preserve the plan's declared unit IDs. The recipe gives units equal weight and cases equal weight within their unit; it does not infer independence or validate a judging plan.

Use the same numeric units for case means and reported_delta. For a native exact-match or scorer-extension report, the reported comparison is in percentage points: convert its stored float with Fraction.from_float, then divide by 100 when the supplied case scores remain in [0, 1]. Captured scalar deltas and judge effect means already use score units. Contributions always describe subject minus baseline; a negative contribution can be an improvement for a lower-is-better metric. The recipe does not classify that change.

Each selected case has an exact rational weight and signed contribution. The visible contributions plus the omitted contribution equal the exact delta. The rounding_residual field is the supplied reported delta minus that exact delta; adding it recovers the reported total. This equality holds even if the caller supplies the wrong total, weights or units, so it does not prove rounding caused the gap. Investigate discrepancies against the validated source workflow. All rational values are strings so JSON conversion introduces no further rounding. The output has no verification authority. It does not explain uncertainty, causality, missing observations or a normalized-NLL ratio, and rankings apply only within one additive metric and scope. Validate the complete source workflow before constructing its inputs; do not use a favorable subset to replace it.

Pack boundary

Captured pack v2 has exactly manifest.json, checksums.sha256, request.json, inputs/policy.json, records/baseline.json, records/subject.json, and reports/evaluation.report.json. Signed packs also have manifest.signature.json; authentication: unsigned_local packs have no signature and a null signing-key fingerprint. The signature uses the existing invarlock/evidence-pack-signature-v1 envelope over the canonical manifest. There is no second signed comparison envelope, embedded verification receipt, or implicit legacy migration.

Captured readers check each file against its byte allowance before opening it, then bound the read request by the observed file size plus one byte. They still recheck descriptor and path identity and reject growth, truncation or replacement. This avoids a ceiling-sized read allocation for a small file; it is not a bound on total parsing, validation or report-rendering memory.

The normalized request, policy, records and report are canonical JSON with a final LF. The checksum ledger uses fixed payload paths. comparison_id hashes the canonical object containing kind: captured, the normalized request digest, both run digests and the policy digest; it identifies the comparison inputs. Extra files/directories, path traversal, symlinks, duplicate keys, non-finite numbers and byte-limit violations are rejected. Payloads must be regular files; verification checks a stable snapshot and validates their digests. Publication is atomic and no-clobber. Keep receipts and rendered outputs outside the immutable pack. See capacity, reports and receipt scopes, and the signed trust-profile flow.