Dataset Providers

Overview

Quick Start

dataset:
  provider: wikitext2
  split: validation
  seq_len: 512
  stride: 512
  preview_n: 64
  final_n: 64
  seed: 42

For Compare & evaluate, reuse the same dataset block in baseline and subject runs.

Concepts

• Preview vs final windows: the runner computes the primary metric on two deterministic splits; counts are recorded in run reports and evaluation reports.
• Pairing: invarlock evaluate requires baseline window evidence to pair windows. Missing/invalid evidence fails closed in CI/Release profiles.
• Offline-first: downloads are opt-in. CLI runs use evaluate --allow-network; programmatic callers can set INVARLOCK_ALLOW_NETWORK=1. Cached datasets can be enforced via HF_DATASETS_OFFLINE=1.
• Vision-text manifests: vision_text is local-files-only and expects JSONL records with id, image_path, prompt, and either answer or answers. It is fixed to single-image examples and batch_size=1.
• Tokenizer contract: dataset providers expect either a callable tokenizer that returns input_ids plus optional attention_mask, or an encode(...) method that accepts truncation=True, max_length=..., and padding="max_length".
• Default runtime-container execution: dataset-backed model-loading commands run in the runtime container by default; public host-side execution uses invarlock evaluate --execution-mode host.
• Dedupe & capacity: INVARLOCK_DEDUP_TEXTS=1 removes exact duplicates; INVARLOCK_CAPACITY_FAST=1 speeds up capacity checks for quick runs.
• HF cache fallback: if a local rerun hits a Hugging Face datasets shared-cache lock/permission error, InvarLock retries with its own writable datasets cache. Set INVARLOCK_HF_DATASETS_CACHE to choose that fallback location explicitly.

Pairing invariants (E001)

Counts mismatches are enforced via coverage.preview.used, coverage.final.used, and paired_windows in dataset.windows.stats.

Reference

Provider matrix

Provider field map

Provider-specific config fields (dataset name, paths, fields) are recorded under report.data when available.

Pairing evidence matrix

HF text provider example

dataset:
  provider: hf_text
  dataset_name: wikitext
  config_name: wikitext-2-raw-v1
  text_field: text
  split: validation
  preview_n: 64
  final_n: 64

Local JSONL provider example

dataset:
  provider: local_jsonl
  path: /data/my_corpus
  text_field: text
  preview_n: 64
  final_n: 64

Vision-text provider example

dataset:
  provider:
    kind: vision_text
    path: tests/fixtures/vision_text/demo_manifest.jsonl
  split: validation
  seq_len: 256
  preview_n: 1
  final_n: 1

Seq2seq provider example (HF)

dataset:
  provider: hf_seq2seq
  dataset_name: wmt14
  src_field: translation.en
  tgt_field: translation.de
  preview_n: 32
  final_n: 32

Environment variables

• INVARLOCK_ALLOW_NETWORK=1 — allow dataset downloads.
• HF_DATASETS_OFFLINE=1 — force cached-only datasets.
• INVARLOCK_DEDUP_TEXTS=1 — exact-text dedupe before tokenization.
• INVARLOCK_CAPACITY_FAST=1 — approximate capacity estimation for quick runs.
• INVARLOCK_HF_DATASETS_CACHE=/path/to/cache — override the writable fallback cache used after shared-cache lock/permission failures.

Troubleshooting

• DEPENDENCY-MISSING: datasets: install invarlock[eval] or invarlock[hf].
• NO-SAMPLES / NO-PAIRS errors: verify dataset fields and split names.
• HF cache .lock / permission errors on local reruns: rerun as-is to use the automatic writable-cache fallback, or set INVARLOCK_HF_DATASETS_CACHE to a writable directory you control.
• vision_text image file is missing: ensure manifest image_path values resolve relative to the JSONL file and point to readable local files.
• Pairing failures (E001): ensure baseline report.json contains evaluation_windows and was produced with matching dataset settings.

Observability

• report.data.* stores provider name, split, and window counts.
• report.evaluation_windows stores preview/final token windows.
• reports preserve dataset metadata and window pairing stats under dataset.*.

Related Documentation

• Configuration Schema
• Environment Variables
• CLI Reference
• reports — Schema, telemetry, and HTML export
• Coverage & Pairing — Window requirements and pairing math
• Bring Your Own Data — Custom dataset workflows