Configuration for StreamingSequenceLoader behavior. More...

Public Attributes
uint32_t	batch_timeout_ms = 10000
	Timeout for subsequent batch preparation (milliseconds).
uint32_t	initialization_timeout_ms = 5000
	Timeout for initial batch preparation (milliseconds).
size_t	max_queue_size = 2
	DEPRECATED: No longer used in refactored implementation.
size_t	token_window_size = 0
	Size of token window to load from disk (in tokens).
bool	verbose_logging = false
	Enable verbose logging during initialization and operation.

Detailed Description

Configuration for StreamingSequenceLoader behavior.

Member Data Documentation

uint32_t Mila::Data::TokenSequenceLoaderConfig::batch_timeout_ms = 10000

Timeout for subsequent batch preparation (milliseconds).

How long to wait for each batch in nextBatch(). Should be generous enough to account for disk I/O variance.

Default: 10000ms (10 seconds)

uint32_t Mila::Data::TokenSequenceLoaderConfig::initialization_timeout_ms = 5000

Timeout for initial batch preparation (milliseconds).

How long to wait for the first batch during construction/reset.

Default: 5000ms (5 seconds)

size_t Mila::Data::TokenSequenceLoaderConfig::max_queue_size = 2

DEPRECATED: No longer used in refactored implementation.

The new architecture uses double buffering instead of a queue, so this parameter has no effect.

size_t Mila::Data::TokenSequenceLoaderConfig::token_window_size = 0

Size of token window to load from disk (in tokens).

Set to 0 for automatic sizing based on memory constraints. Larger windows reduce I/O frequency but increase memory usage.

Default: 0 (automatic, typically ~25M tokens)

bool Mila::Data::TokenSequenceLoaderConfig::verbose_logging = false

Enable verbose logging during initialization and operation.

When true, prints dataset statistics, window sizes, and batch counts.

Default: false

The documentation for this struct was generated from the following file: