Troubleshooting

Overview

Quick Start

# Check environment and configuration
invarlock doctor --config <config.yaml> --profile ci

# Validate a report
invarlock verify reports/eval/evaluation.report.json --profile ci

# Enable debug output for detailed traces
INVARLOCK_DEBUG_TRACE=1 invarlock run -c <config.yaml>

Error Code Reference

Pairing Errors (E001–E006)

These errors relate to window pairing, tokenizer consistency, and evidence integrity.

E001 — Pairing Schedule Mismatch

Triggers:

• PAIRING-EVIDENCE-MISSING: Baseline report path does not exist or cannot be parsed
• PAIRING-SCHEDULE-MISMATCH: Window matching fraction ≠ 1.0, window overlap > 0, or counts diverge after stratification
• PAIRED-WINDOWS-COLLAPSED: paired_windows ≤ 0 under paired baseline

Common causes:

• Missing or corrupted baseline report.json
• Baseline lacks evaluation_windows section
• Dataset settings differ between baseline and subject runs
• Sequence length / stride mismatch

Fixes:

1. Ensure baseline report.json exists and contains evaluation_windows
2. Verify dataset settings match: provider, seq_len, stride, split
3. Regenerate baseline with the same configuration
4. Check that tokenizer hash matches between runs

Example error:

[INVARLOCK:E001] PAIRING-SCHEDULE-MISMATCH: window_match_fraction=0.950

---

E002 — Tokenizer Digest Mismatch

Triggers:

• Subject and baseline tokenizers produce different vocabulary hashes

Common causes:

• Different tokenizer versions or configurations
• Model updated with new vocabulary
• Trust-remote-code flag inconsistency

Fixes:

1. Ensure both runs use the same model checkpoint
2. Pin tokenizer version via revision in config
3. Regenerate baseline with current tokenizer

Example error:

[INVARLOCK:E002] TOKENIZER-DIGEST-MISMATCH: subject and baseline tokenizers differ

---

E003 — Mask Parity Mismatch

Triggers:

• MLM mask positions differ between subject and baseline under identical tokenizers

Common causes:

• Different masking seeds between runs
• Baseline generated with different mask_prob or mask_seed
• Labels in baseline report corrupted or missing

Fixes:

1. Ensure eval.loss.seed matches between runs
2. Regenerate baseline with consistent masking configuration
3. Verify baseline contains MLM labels in evaluation_windows

Example error:

[INVARLOCK:E003] MASK-PARITY-MISMATCH: mask positions differ under matched tokenizers

---

E004 — Provider Digest Missing

Triggers:

• Subject or baseline is missing the provider digest (ids/tokenizer hash)

Common causes:

• Old baseline report generated before digest tracking was added
• Report truncated or missing provenance section
• Windows not materialized due to INVARLOCK_STORE_EVAL_WINDOWS=0

Fixes:

1. Regenerate baseline with current InvarLock version
2. Ensure INVARLOCK_STORE_EVAL_WINDOWS=1 (default)
3. Check report.provenance.provider_digest exists

Example error:

[INVARLOCK:E004] PROVIDER-DIGEST-MISSING: subject or baseline missing ids/tokenizer digest

---

E006 — Window IDs Digest Mismatch

Triggers:

• Subject and baseline window IDs differ (different windows selected)

Common causes:

• Different dataset splits or stratification seeds
• Capacity constraints forced window reduction
• Baseline generated with different preview_n/final_n

Fixes:

1. Use identical dataset.seed and window counts
2. Verify capacity allows requested windows
3. Regenerate baseline with matching configuration

Example error:

[INVARLOCK:E006] IDS-DIGEST-MISMATCH: subject and baseline window IDs differ

---

Primary Metric Errors (E111)

E111 — Primary Metric Degraded

Triggers:

• Primary metric (perplexity/accuracy) is non-finite (NaN, Inf)
• Primary metric degraded beyond acceptable ratio

Common causes:

• Numerical instability (overflow/underflow)
• Model weights corrupted by edit
• Insufficient precision (try float32)
• Empty or malformed evaluation windows

Fixes:

1. Force dtype: float32 in model config
2. Reduce batch size if memory-constrained
3. Use an accelerator (CUDA/MPS) for better precision
4. Lower plan.max_modules to reduce edit scope
5. Check that evaluation windows contain valid tokens

Example error:

[INVARLOCK:E111] Primary metric degraded or non-finite (preview=inf, final=nan)

Notes:

• invarlock evaluate always emits a report before exiting on E111
• invarlock run logs a warning for non-finite PM but does not raise E111

---

Verification Errors (E601)

E601 — report Verification Failed

Triggers:

• report fails schema validation
• Pairing math recomputation fails
• Gate checks fail in CI/Release profile

Common causes:

• report JSON corrupted or hand-edited
• Schema version mismatch (schema_version ≠ v1)
• Missing required fields (run_id, meta, dataset, etc.)

Fixes:

1. Regenerate report via invarlock report --format report
2. Ensure report is unmodified from generation
3. Check schema_version is "v1"

Example error:

[INVARLOCK:E601] report schema validation failed

---

Exit Code Summary

Common Issues

Network Blocked

Symptom: Downloads fail silently or model loading hangs.

Fix:

INVARLOCK_ALLOW_NETWORK=1 invarlock evaluate --baseline gpt2 --subject gpt2

Dependency Missing

Symptom: ModuleNotFoundError for torch, transformers, etc.

Fix:

pip install "invarlock[hf]"      # HF adapters + eval
pip install "invarlock[guards]"  # Guard math
pip install "invarlock[adapters]" # All adapters

Calibration Data Not Indexable

Symptom: Runner fails with "calibration data must be indexable."

Fix:

INVARLOCK_ALLOW_CALIBRATION_MATERIALIZE=1 invarlock run -c <config.yaml>

Guard Prepare Failures

Symptom: Guard initialization fails in strict mode.

Fix (local debugging only):

context:
  run:
    strict_guard_prepare: false

Non-Finite Metrics on CPU

Symptom: Perplexity returns inf or nan on CPU runs.

Fixes:

1. Use an accelerator when available: --device cuda or --device mps
2. Force higher precision: dtype: float32 in config
3. Reduce edit scope or batch size

Debug Tools

Doctor Command

Run comprehensive environment checks:

invarlock doctor --config <config.yaml> --profile ci --strict

Debug Trace

Enable detailed logging:

INVARLOCK_DEBUG_TRACE=1 invarlock run -c <config.yaml>

Plugins Inspection

List available adapters/guards/edits:

invarlock plugins list --verbose
invarlock plugins adapters --explain hf_causal

Related Documentation

• CLI Reference
• Configuration Schema
• Dataset Providers
• Environment Variables
• reports