How often do our checks catch what they claim to catch?
A verdict is only as trustworthy as the scanner behind it, so we validate every check against synthetic fixtures with known ground truth (injected glitch tokens, fabricated lineage claims, mangled metadata) plus matched clean controls. The tables show each check's measured true-positive and false-positive rate, regenerated with npm run validate; a check that breaches its declared floor fails the build. Fixtures are synthetic and seeded, so these numbers measure the decision logic; a clean row is a validated check, not a certification.
CPU weights battery (embedding-geometry checks)
Synthetic known-positive/known-negative fixtures with seeded deterministic RNG, run through the exact production decision cores (analyzeNorms, lineageVerdict). TPR = flagged positives / positives; FPR = flagged negatives / negatives. Findings with severity 'info' do not count as flags. PRNG seed 20260820, so the run is deterministic.
| check | TPR | FPR | fixtures (pos / neg) | status |
|---|---|---|---|---|
| glitch-norm-scan | 100% | 0% | 200 / 400 | validated |
| lineage-row-cosine | 100% | 0% | 200 / 400 | validated |
| gguf-metadata | 100% | 0% | 7 / 13 | validated |
| zero-template-token | 100% | 0% | 2 / 5 | validated |
Fixture construction
| glitch-norm-scan | positives: 200 synthetic vocabs (8192 tokens) with 5–40 injected undertrained tokens (norm 0.01–0.25× median), half ASCII-surfaced negatives: 400 clean vocabs (lognormal σ=0.1 norms + 32 excluded special tokens), half with a legit low-norm tail at 0.4–0.6× median threshold: norm < 0.3× vocab median |
| lineage-row-cosine | positives: 200 independently-sampled child embeddings claiming a finetune relation (mislabeled lineage), plus 200 hidden-dim mismatches negatives: 200 true finetunes (row perturbation ε ≤ 0.1) and 200 5-bit quantizations of the parent threshold: consistent ≥ 0.8 mean row cosine, inconsistent < 0.3 |
| gguf-metadata | positives: 7 GGUF metadata fixtures modeled on real incidents (phi-4 quant EOS resurrection, Yi wrong-EOS quants, pre-May-2024 missing pre-tokenizer, template drift/missing vs source) negatives: 13 clean/benign fixtures (matching metadata, sentencepiece without pre, dynamic eos_token templates, unknown sources) threshold: deterministic metadata comparisons (exact) |
| zero-template-token | positives: 2 vocabs with zero-row tokens the template uses (Llama-3 base incident shape) negatives: 4 vocabs: low-but-functional control tokens (~4–6% of median, the Llama-3-Instruct pattern), unused zero-row reserved tokens, healthy specials, a no-template repo, and a token at 1e-5 (just above the epsilon) threshold: embedding row norm < ZERO_NORM_EPS (1e-6) AND token string appears in the chat template — untrained rows measure ~2e-21 on the Llama-3 base incident, functional control tokens ~4e-2 |
Static battery (metadata checks)
Deterministic known-positive/known-negative metadata fixtures per check, run through the exact production decision core (staticChecks). Floors: TPR = 1.0, FPR = 0.0 — the checks are metadata comparisons, so anything less is a logic regression.
| check | TPR | FPR | fixtures (pos / neg) | status |
|---|---|---|---|---|
| pickle-no-safetensors | 100% | 0% | 3 / 2 | validated |
| custom-code | 100% | 0% | 2 / 1 | validated |
| gated-repo | 100% | 0% | 2 / 1 | validated |
| no-license | 100% | 0% | 1 / 1 | validated |
| gated-files-unreadable | 100% | 0% | 1 / 3 | validated |
| license-drift | 100% | 0% | 1 / 3 | validated |
| template-dropped | 100% | 0% | 2 / 4 | validated |
| template-drift | 100% | 0% | 3 / 3 | validated |
| architecture-mismatch | 100% | 0% | 2 / 3 | validated |
| vocab-drift | 100% | 0% | 1 / 2 | validated |
| parent-unreadable | 100% | 0% | 1 / 2 | validated |
| eos-disjoint | 100% | 0% | 1 / 3 | validated |
| token-id-out-of-range | 100% | 0% | 2 / 3 | validated |
| pad-is-eos | 100% | 0% | 2 / 3 | validated |
| terminator-not-stoppable | 100% | 0% | 2 / 5 | validated |
| stop-token-dropped | 100% | 0% | 1 / 3 | validated |
What this does and doesn't claim
Findings are measured differentials, never certifications; this page just measures the instruments themselves. Known limits: fixtures are synthetic, the GPU battery's behavioral probes are validated separately at run time, and a validated check can still miss defect classes it was never designed to detect.