CharTokenizer.ixx File Reference

Character-level tokenizer implementing the Tokenizer API. More...

#include <string>
#include <string_view>
#include <vector>
#include <span>
#include <memory>
#include <optional>
#include <filesystem>
import Data.TokenizerVocabulary;
import Data.Tokenizer;
import Data.CharVocabulary;

Classes
class	Mila::Data::CharTokenizer
	Character-level tokenizer. More...

Namespaces
namespace	Mila
	Mila main API namespace.
namespace	Mila::Data

Typedefs
using	Mila::Data::TokenId

Detailed Description

Character-level tokenizer implementing the Tokenizer API.

Provides a simple byte/char tokenizer that maps single-byte characters to token ids via a TokenizerVocabulary.