tejeshbhalla/phi-3-vision warn

Convert the weights to safetensors before loading them anywhere that matters.

1. Do not load the pickle files in-process — pickle deserialization executes arbitrary code from the file.
2. Convert locally in a sandbox: `pip install safetensors` and use `safetensors.torch.save_file` on a state dict loaded with `torch.load(..., weights_only=True)` (refuses most code-execution payloads), or use Hugging Face's `convert.py` space/script.
3. Pin the exact revision hash you converted from, and load only your converted safetensors artifact from then on.

Review and pin the custom code; never float on `main` with trust_remote_code=True.

1. Read every `.py` file in the repo before first load — this code runs in your process.
2. Pin the revision: `from_pretrained(model_id, revision="<commit sha>", trust_remote_code=True)` so a later push can't swap the code under you.
3. Prefer a version of the architecture already in `transformers` if one exists, which removes the remote-code requirement entirely.

Get a license from the owner or pick a licensed alternative — this is a legal gap, not a technical one.

1. With no declared license you have no usage rights by default; treat the weights as all-rights-reserved.
2. Open an issue or discussion on the repo asking the owner to declare a license, or use the licensed upstream/parent model instead.

Align the stop-token declarations — a pure metadata fix to `generation_config.json` (and `config.json`).

1. Identify the token the chat template actually ends assistant turns with (e.g. `<|eot_id|>`, `<end_of_turn>`, `<|im_end|>`) and make sure its id is in `generation_config.json`'s `eos_token_id` list.
2. Keep `config.json`'s `eos_token_id` consistent with (or a subset of) `generation_config.json`'s — runtimes differ in which file they read.
3. For the pad-equals-EOS hazard: repoint `pad_token` at a dedicated padding token before fine-tuning; serving is unaffected.
4. Until the repo is fixed, pass explicit stop tokens to your serving stack (e.g. vLLM `stop_token_ids`, llama.cpp `--override-kv tokenizer.ggml.eos_token_id`).

Keep the affected token strings out of the model's input — the scan-derived runtime guard carries this model's exact blocklist.

1. Fetch this model's guard artifact (`/api/v1/guard/<owner>/<model>`): the confirmed corrupting tokens and the low-norm candidate list, derived from the published scan.
2. Screen inbound text with it (the `@ingotai/guard` package is a reference implementation) and route flagged records to a different model or human review — verbatim-copy tasks on flagged strings are the failure mode.
3. The underlying cause is undertrained embeddings in the weights; a true fix is weight-level (continued pretraining on the affected tokens) — that is not a patch, it's a training job.