How often do our checks catch what they claim to catch?
A verdict is only as trustworthy as the scanner behind it, so every check is validated against synthetic fixtures with known ground truth — injected glitch tokens, fabricated lineage claims, mangled metadata — plus matched clean controls. The tables show each check's measured true-positive and false-positive rate, regenerated with npm run validate; a check that breaches its declared floor fails the build. Fixtures are synthetic and seeded, so these numbers measure the decision logic — a clean row is a validated check, not a certification.
CPU weights battery (embedding-geometry checks)
Synthetic known-positive/known-negative fixtures with seeded deterministic RNG, run through the exact production decision cores (analyzeNorms, lineageVerdict). TPR = flagged positives / positives; FPR = flagged negatives / negatives. Findings with severity 'info' do not count as flags. PRNG seed 20260820 — the run is deterministic.
| check | TPR | FPR | fixtures (pos / neg) | status |
|---|---|---|---|---|
| glitch-norm-scan | 100% | 0% | 200 / 400 | validated |
| lineage-row-cosine | 100% | 0% | 200 / 400 | validated |
Fixture construction
| glitch-norm-scan | positives: 200 synthetic vocabs (8192 tokens) with 5–40 injected undertrained tokens (norm 0.01–0.25× median), half ASCII-surfaced negatives: 400 clean vocabs (lognormal σ=0.1 norms + 32 excluded special tokens), half with a legit low-norm tail at 0.4–0.6× median threshold: norm < 0.3× vocab median |
| lineage-row-cosine | positives: 200 independently-sampled child embeddings claiming a finetune relation (mislabeled lineage), plus 200 hidden-dim mismatches negatives: 200 true finetunes (row perturbation ε ≤ 0.1) and 200 5-bit quantizations of the parent threshold: consistent ≥ 0.8 mean row cosine, inconsistent < 0.3 |
Static battery (metadata checks)
Deterministic known-positive/known-negative metadata fixtures per check, run through the exact production decision core (staticChecks). Floors: TPR = 1.0, FPR = 0.0 — the checks are metadata comparisons, so anything less is a logic regression.
| check | TPR | FPR | fixtures (pos / neg) | status |
|---|---|---|---|---|
| pickle-no-safetensors | 100% | 0% | 3 / 2 | validated |
| custom-code | 100% | 0% | 2 / 1 | validated |
| gated-repo | 100% | 0% | 2 / 1 | validated |
| no-license | 100% | 0% | 1 / 1 | validated |
| gated-files-unreadable | 100% | 0% | 1 / 3 | validated |
| license-drift | 100% | 0% | 1 / 3 | validated |
| template-dropped | 100% | 0% | 2 / 3 | validated |
| template-drift | 100% | 0% | 3 / 3 | validated |
| architecture-mismatch | 100% | 0% | 2 / 3 | validated |
| vocab-drift | 100% | 0% | 1 / 2 | validated |
| parent-unreadable | 100% | 0% | 1 / 2 | validated |
What this does and doesn't claim
Findings are measured differentials, never certifications — this page just measures the instruments themselves. Known limits: fixtures are synthetic, the GPU battery's behavioral probes are validated separately at run time, and a validated check can still miss defect classes it was never designed to detect.