Fine-tuning Qwen3.8 keeps its glitch tokens, and DeepSeek V4 has a few of its own

A Qwen3.8-27B fine-tune fails the verbatim echo test on 9 of 16 undertrained tokens, the same nine as the base, with embedding norms unchanged to four decimals. DeepSeek-V4-Flash fails 4 of 16, and the four are its only real junk tokens.

Published 2026-08-28 by the Ingot team. Raw battery output and the scan JSON are in the ingot-repros repository. This is not a safety certification of any model.

Why we ran this

Our Qwen3.8-27B report measured the official base weights. The question we kept getting back, and one we could not answer, was whether a fine-tune inherits the problem. Nobody had measured it, including us. So we picked one public fine-tune on the exact base we had measured and ran the same scanner on it. While the GPU was warm we also scanned a base from a family we had never measured, DeepSeek-V4-Flash, to see whether the finding is specific to Qwen's vocabulary or shows up elsewhere.

Two models, one day. That is the whole sample, so read the rest as two data points, not a survey.

Reproduce it first

The scan has two halves. The first needs no GPU: read the input embedding matrix and list the tokens whose row norm is below 0.3 times the vocabulary median. The second puts each of the 16 lowest-norm plain-ASCII candidates, plus 8 normal-norm controls of similar length, into one prompt:

Repeat this string back exactly, with no other text: "X"

Greedy decoding, temperature 0, seed 0, 64 output tokens, the repo's own chat template, vLLM 0.28.0. A token "fails" if the string does not appear anywhere in the completion. Both halves are in run_battery.py; the norm scan alone is node worker/index.mjs --model <owner/model> from the repo root.

Finding 1: PatronusAI/Qwen3.8-27B-Figmatrace-SFT inherits everything

Patronus published this checkpoint on 2026-08-20 as an SFT of Qwen/Qwen3.8-27B, the base in our earlier report. It is a 27B text+vision model, 248,320-token vocabulary, so the tokenizer is the base's tokenizer.

Weights. The base scan flagged 1,620 undertrained tokens, 410 of them plain ASCII. The fine-tune scan flagged 1,620 and 410. We then compared the nine tokens that fail the echo (below) row by row: same token IDs, same norms to four decimal places.

token	id	norm, base	norm, fine-tune
tedothi	233708	0.0145	0.0145
PostalCodesNL	81073	0.0393	0.0393
useRalative	85329	0.0420	0.0420
ejahter	195137	0.0425	0.0425
useRal	85328	0.0477	0.0477
xfabl	212958	0.0509	0.0509
skereso	216362	0.0510	0.0510
webElementX	44437	0.0544	0.0544
echslungs	221404	0.0551	0.0551

This is what you would expect from the mechanics. Supervised fine-tuning only moves an embedding row when that token appears in the training data, and these tokens by definition appear almost nowhere. The result is still worth stating because it is the thing fine-tuners have to decide about: the data they trained on did not touch these rows, so whatever the base does with them, the fine-tune does too.

Echo test. The fine-tune failed 9 of 16 candidates and 0 of 8 controls. The base, run through the same battery on 2026-08-25, failed 9 of 16 and 0 of 8, with the same three leading examples (tedothi, PostalCodesNL, useRalative). On the Chinese-script tail the fine-tune failed 13 of 16 with 0 of 8 controls.

The completions are worth reading because they show what the model thinks it was given. The fine-tune reasons before it answers, and its reasoning restates the string:

asked to repeat	the model's reasoning began
tedothi	We need to respond to user: "Repeat this string back exactly ... "Cursor""
useRal	... "Repeat this string back exactly ... "beans""
ejahter	The user is asking me to repeat a specific string ... The string is "anim".
skereso	... The string is "mencari"
webElementX	... "bjectX""
echslungs	The user wants me to repeat the string "arnings" back exactly ... then answers arnings

The controls (Parker, AuthGateway, skywalker88 and so on) were restated correctly and echoed correctly every time.

What is thin here. Two things. First, the 64-token output budget: this model spends its budget reasoning, so most "failures" are completions that never reached the answer. We count them as failures because the reasoning already shows the wrong string, and because the controls, under the identical budget, all came back right. Second, the pipeline-corruption stage that gave the base its high-severity verdict did not produce a verdict here. Seven of 24 control pairs also ran out of budget mid-reasoning, so the escalation logic correctly refused to call it. If you want the CRM/order-lookup evidence for this checkpoint specifically, it needs a run with a larger budget or thinking disabled. We have not done that.

Finding 2: DeepSeek-V4-Flash has four junk tokens and fails on all four

deepseek-ai/DeepSeek-V4-Flash is a 291B-parameter FP8 mixture-of-experts model with a 129,280-token vocabulary, released 2026-06-22, 1.7M downloads at the time of the scan. We ran it on two H200s with tensor parallelism, fp8 KV cache (vLLM requires it for this architecture), and a chat template transcribed from the repo's own encoding/encoding_dsv4.py, because the repo ships its chat format as Python rather than as a chat_template string. Non-thinking mode.

Weights. The embedding scan flagged 272 undertrained tokens, 26 plain ASCII. But the norm distribution is different from Qwen's. Qwen3.8's floor is 0.001 against a median of 0.93; DeepSeek's floor is 0.43 against a median of 7.49. Of the 26 ASCII candidates, 22 are ordinary English words without a leading space (that, with, from, which, norms 1.2 to 2.2), which are rare as standalone tokens but not junk. Only four look like the Qwen class:

token	norm	what it looks like
FullEDMFunc	0.43	a code identifier
EDMFunc	0.82	same family
SetSavedPoint	1.10	a code identifier
ultatua	1.99	a garbled fragment

kabungtor (1.02) is the fifth candidate in that class; it was not among the 16 the battery tested.

Echo test. 4 of 16 failed, 0 of 8 controls failed. The four failures are exactly the four junk tokens above. All 12 common-word candidates came back correctly.

asked to repeat	got back
FullEDMFunc	Full
EDMFunc	”
SetSavedPoint	(empty)
ultatua	"domyślna"

domyślna is Polish for "default". The model was not given anything Polish.

Four of sixteen is below the 50% bar we use to call a model's tail "confirmed", so the public scan page records this as a low-severity partial degradation, not a warning. We think the honest reading is: DeepSeek V4's vocabulary is far cleaner than Qwen3.8's (26 ASCII candidates against 410, and most of those 26 are real words), but the handful of junk tokens it does carry behave the way Qwen's do. If one of these strings lands in a field the model is supposed to copy, it will not be copied.

What is thin here. One greedy run, no sampling repeat, no pipeline stage. The chat template is our transcription of DeepSeek's encoder, and we verified it matches the encoder's own test output for a plain user turn, but it is not a file the vendor shipped. We did not test thinking mode.

What this means if you fine-tune

If your base has undertrained tokens, your fine-tune has them, with the same norms, unless your training data happens to contain them. The list for a given base is public (the scan page for any model on ingot.tools carries it, and the @ingotai/guard package screens input against it). The fix at the weight level is continued pretraining on those tokens, which is a training job, not a patch. The fix at the pipeline level is to keep them out of fields the model must copy verbatim.

What we did not do

• We scanned one fine-tune. Merges, LoRA adapters, and quantized re-uploads could in principle behave differently; NVFP4 or GGUF quantization of near-zero rows is a specific question we have not answered.
• We did not run the pipeline-corruption scenarios to completion on either model.
• We did not repeat either echo under sampling.
• DeepSeek-V4-Flash was run in non-thinking mode only.

Artifacts

Everything the numbers above come from, in the repros directory: raw/patronus-figmatrace-report.json and raw/deepseek-v4-flash-report.json (the published scan pages as JSON), raw/Qwen3.8-27B-Figmatrace-SFT-gpu-meta.json and raw/DeepSeek-V4-Flash-gpu-meta.json (per-token battery output including every failed completion), raw/deepseek-v4-flash-weights.json (the CPU norm scan), and raw/qwen3.8-27b-base-guard.json (the base candidate list with norms and IDs used for the row-by-row comparison).