SpecialTokens.ixx File Reference

Configuration for special tokens used across all tokenizer types. More...

#include <string>
#include <string_view>
#include <vector>
#include <unordered_map>
#include <cstdint>

Classes
struct	Mila::Data::SpecialTokens
	Configuration for special tokens across all tokenizer types. More...

Namespaces
namespace	Mila
	Mila main API namespace.
namespace	Mila::Data

Enumerations
enum class	Mila::Data::SpecialToken : uint8_t {   Mila::Data::PAD = 0 , Mila::Data::UNK , Mila::Data::BOS , Mila::Data::EOS ,   Mila::Data::MASK , Mila::Data::SEP , Mila::Data::CLS }
	Special token types for tokenization. More...

Detailed Description

Configuration for special tokens used across all tokenizer types.