3 commits

Author	SHA1	Message	Date
Simone Iacomino	2ce20d22fd	unicode : reject overlong UTF-8 and out-of-range codepoints (#30485) ... * unicode : reject overlong UTF-8 and out-of-range codepoints * Update tests/test-unicode.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * tests: document byte-wise UTF-8 error recovery * break out of first test on failure --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>	2026-10-15 10:12:56 +03:00
Sandro Hanea	10da8c463f	unicode : add opt-in NFC normalization for Unigram tokenizers (#30390) ... * unicode : add opt-in NFC normalization for Unigram tokenizers Note: used help from Codex, Sol 6.1 Medium * Fixing initial bot comments + linter error (using existing out instead of print) * removed test_nfc_tokenizer	2026-10-14 08:33:24 +03:00
Thiago Padilha	afd439df1f	unicode : include '~' in collapsed symbol class (#26972) ... The collapsed \p{S} class was missing '~', which split " ~" into separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4. This caused re-tokenized prompts to diverge from sampled tokens and broke KV cache reuse. Assisted-by: Codex	2026-08-18 15:15:22 +02:00