Data.TokenizerType Module Reference

Enumerations
enum class	Mila::Data::TokenizerType : uint8_t {   Unknown = 0 , Char , Bpe , SentencePiece ,   Word , Unigram }
	Tokenizer type discriminator used across tokenizer and vocabulary types. More...

Functions
TokenizerType	Mila::Data::stringToTokenizerType (std::string_view s) noexcept
	Parse a string into TokenizerType.
std::string_view	Mila::Data::tokenizerTypeToString (TokenizerType t) noexcept
	Convert TokenizerType to a stable string representation.

Files
file	/__w/Mila/Mila/Mila/Src/Data/Tokenizers/TokenizerType.ixx