BpeTrainer.ixx File Reference

BPE vocabulary trainer with incremental corpus accumulation. More...

#include <string>
#include <stdexcept>
#include <istream>
#include <fstream>
#include <filesystem>
import Data.TokenizerTrainer;
import Data.BpeVocabulary;
import Data.BpeVocabularyConfig;

Classes
class	Mila::Data::BpeTrainer
	Corpus accumulator and trainer for BPE vocabularies. More...

Namespaces
namespace	Mila
	Mila main API namespace.
namespace	Mila::Data

Detailed Description

BPE vocabulary trainer with incremental corpus accumulation.

Delegates vocabulary construction to BpeVocabulary::train(). Retained as a separate class for future extensibility: progress callbacks, streaming corpus processing, and training checkpointing.