CharVocabulary.ixx File Reference

Character vocabulary with factory-based construction. More...

#include <vector>
#include <unordered_map>
#include <string>
#include <fstream>
#include <algorithm>
#include <stdexcept>
#include <cstdint>
#include <filesystem>
#include <optional>
#include <cctype>
import Data.Tokenizer;
import Serialization.Metadata;
import Data.FileHeader;
import Data.CharVocabularyConfig;
import Data.TokenizerVocabulary;
import Data.SpecialTokens;

Classes
class	Mila::Data::CharVocabulary
	Character vocabulary for tokenization. More...

Namespaces
namespace	Mila
	Mila main API namespace.
namespace	Mila::Data

Typedefs
using	Mila::Data::TokenId

Detailed Description

Character vocabulary with factory-based construction.

Vocabularies are immutable after construction and store their configuration for full provenance tracking. Use static factory methods to create instances.