sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
About this project
SentencePiece SentencePiece is a fast, lightweight, and unsupervised text tokenizer and detokenizer designed for neural network-based text generation systems (such as Large Language Models) where the vocabulary size is fixed prior to training. It implements subword units—including Byte-Pair-Encoding (BPE) [Sennrich et al.] and the unigram language model [Kudo.]—with the ability to train directly from raw sentences. By treating input text as a raw sequence of Unicode characters, SentencePiece enables a purely end-to-end, language-independent pipeline that completely eliminates the need for language-specific pre- or post-processing. This is not an official Google…
Technologies
Project health
GitHub
Reviews
Built by
Maintain google/sentencepiece? Claiming verifies admin access through your GitHub account and gives you control of this listing.
