https://www.pistack.xyz/tags/tokenization/ Recent content in Tokenization on Pi Stack Hugo en-us Sat, 26 Sep 2026 00:00:00 +0000 https://www.pistack.xyz/posts/2026-09-26-word-segmentation-libraries-jieba-kagome-kuromoji/ Sat, 26 Sep 2026 00:00:00 +0000 https://www.pistack.xyz/posts/2026-09-26-word-segmentation-libraries-jieba-kagome-kuromoji/ <p>Paste <code>南京市长江大桥</code> into any system that splits on spaces and you get exactly one token: a meaningless eleven-character blob. Chinese, Japanese and Thai do not put spaces between words, so before a search index, a log pipeline, a content filter or a keyword extractor can do anything useful with them, some component has to decide where the word boundaries are. That component is a <strong>word segmenter</strong>, and the wrong one quietly poisons everything downstream — broken search recall, useless word clouds, and keyword rules that never fire.</p>