分词器

The component that splits text into token sequences—the first and last stop for model I/O.

Tokenizers decide how text is digitized: common schemes like BPE and WordPiece split words into subwords, balancing vocabulary size and coverage. Tokenization quality affects spelling, numbers and multilingual handling.