TokenizerVocabulary.ixx File Reference

Abstract interface for tokenizer vocabularies used by data pipelines. More...

#include <string>
#include <vector>
#include <unordered_map>
#include <optional>
#include <filesystem>
import Data.Tokenizer;

Classes
class	Mila::Data::TokenizerVocabulary
	Generic tokenizer vocabulary interface. More...

Namespaces
namespace	Mila
	Mila main API namespace.
namespace	Mila::Data

Detailed Description

Abstract interface for tokenizer vocabularies used by data pipelines.

Defines the minimal API for mapping between token strings and numeric ids, and for persisting/loading vocabulary contents.