Dropout regularization module for neural networks. More...

Inheritance diagram for Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >:

Collaboration diagram for Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >:

[legend]

Public Types
using	ComponentBase = Component<TDeviceType, TInput, TOutput>
	Alias for base Component type.
using	MR = typename DeviceTypeTraits<TDeviceType>::memory_resource
	Memory resource type used for tensors, selected based on device type.

Public Member Functions
	Dropout (const std::string &device_name, const DropoutConfig &config)
	Constructs a new Dropout module with a device name.
	Dropout (std::shared_ptr< DeviceContext > device_context, const DropoutConfig &config)
	Constructs a new Dropout module with a provided device context.
void	backward (const Tensor< TInput, MR > &input, const Tensor< TOutput, MR > &output_grad, Tensor< TInput, MR > &input_grad)
	Performs the backward pass of the Dropout operation.
void	forward (const Tensor< TInput, MR > &input, Tensor< TOutput, MR > &output)
	Performs the forward pass of the Dropout operation.
float	getProbability () const
	Gets the dropout probability used by this module.
unsigned int	getSeed () const
	Gets the current random seed.
const ComponentType	getType () const override
	Get the component type identifier.
void	load (ModelArchive &archive) override
	Deserializes the module state from a ZIP archive.
size_t	parameterCount () const override
	Gets the number of trainable parameters in this module.
void	save (ModelArchive &archive) const override
	Serializes the module state to a ZIP archive.
void	setSeed (unsigned int seed)
	Sets the random seed for dropout mask generation.
std::string	toString () const override
	Generates a string representation of this module's configuration.
Public Member Functions inherited from Mila::Dnn::Component< DeviceType::Cuda, float, float >
	Component (const std::string &name)
	Construct component with required name identifier.
virtual	~Component ()=default
virtual void	build (const BuildContext &context) final
	Build the component with the provided BuildContext (canonical overload).
virtual DeviceId	getDeviceId () const=0
	Get the compute device id associated with this component.
virtual std::vector< ITensor * >	getGradients () const=0
	Return non-owning pointers to parameter gradient tensors.
virtual MemoryStats	getMemoryStats () const=0
	Return the current memory allocation breakdown for this component.
const std::string	getName () const
	Get the component's name identifier.
virtual std::vector< std::string >	getParameterNames () const
	List all available parameter names for this component.
virtual std::vector< ITensor * >	getParameters () const=0
	Return non-owning pointers to parameter tensors.
RuntimeMode	getRuntimeMode () const noexcept
	Convenience accessor — true if currently in Eval mode.
TrainingMode	getTrainingMode () const noexcept
	The current runtime behavioral mode of this Component.
virtual bool	isBuilt () const final
	Returns true if build() has completed successfully.
bool	isInferenceMode () const noexcept
bool	isTrainingMode () const noexcept
virtual void	loadParameter (const std::string &name, const Serialization::ITensorBlob &blob)
	Load a parameter from serialized tensor data.
virtual void	save_ (ModelArchive &archive, SerializationMode mode) const=0
void	setTrainingMode (TrainingMode mode)
	Set the runtime behavioral mode for this Component.
virtual void	synchronize ()=0
	Wait for outstanding device work submitted by this component.
virtual void	zeroGradients ()
	Clear all model-owned gradients for this component.

Private Member Functions
void	createOperation ()
	Creates the appropriate Dropout operation based on the current device context.
void	generateMask (Tensor< TOutput, MR > &mask, const std::vector< int64_t > &shape)
	Generates a new dropout mask for the given shape.

Private Attributes
DropoutConfig	config_
	Configuration for the Dropout module.
std::shared_ptr< Tensor< TOutput, MR > >	mask_ { nullptr }
	The binary mask tensor for element selection.
std::shared_ptr< UnaryOperation< TDeviceType, TInput, TOutput > >	operation_ { nullptr }
	The operation that implements the dropout calculation.
std::vector< std::shared_ptr< Tensor< TOutput, MR > > >	output_state_ = {}
	Collection of output state tensors for caching.
std::vector< std::shared_ptr< ITensor > >	parameters_ = {}
	Collection of parameters for this module (empty for Dropout).
std::mt19937	rng_
	Random number generator for mask generation.
unsigned int	seed_ { 0 }
	Random seed for reproducible dropout patterns.

Additional Inherited Members
Static Public Member Functions inherited from Mila::Dnn::Component< DeviceType::Cuda, float, float >
static constexpr DeviceType	getDeviceType ()
	Compile-time device type for this component instance.
static constexpr TensorDataType	getPrecision () noexcept
	Compile-time tensor precision for this component instance.
Protected Member Functions inherited from Mila::Dnn::Component< DeviceType::Cuda, float, float >
IExecutionContext *	getExecutionContext () const
	Get the shared execution context.
bool	hasExecutionContext () const noexcept
	Check if execution context has been set.
void	loadParameterFromBlob (const std::string &param_name, const Serialization::ITensorBlob &blob, Tensor< TParameterPrecision, TMemoryResource > &target, const shape_t &expected_shape)
	Load a tensor blob into a parameter tensor with validation.
virtual void	onBuilding (const BuildContext &config)
	Hook invoked by build() to allocate component buffers.
virtual void	onExecutionContextSet ()
	Lifecycle hook: Called immediately after ExecutionContext is set.
virtual void	onTrainingModeChanging (TrainingMode mode)
	Hook called before TrainingMode transitions.
void	setExecutionContext (IExecutionContext *context)
	Set the execution context for this component.
Protected Attributes inherited from Mila::Dnn::Component< DeviceType::Cuda, float, float >
BuildContext	build_context_
	The BuildContext stored at build time.

Detailed Description

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>
requires ValidFloatTensorTypes<TInput, TOutput>
class Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >

Dropout regularization module for neural networks.

Dropout is a regularization technique that randomly zeroes elements of an input tensor during training with probability p, and optionally scales the remaining elements by 1/(1-p). During inference, dropout is typically disabled.

Dropout helps prevent overfitting by preventing co-adaptation of feature detectors. The technique effectively trains an ensemble of multiple networks sharing parameters, which improves generalization.

Template Parameters

TDeviceType	The device type (CPU or CUDA) on which the module will operate.
TInput	The data type of the input tensor elements.
TOutput	The data type of the output tensor elements, defaults to TInput.

Constructor & Destructor Documentation

◆ Dropout() [1/2]

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::Dropout	(	const std::string &	device_name,
		const DropoutConfig &	config )

inlineexplicitexport

Constructs a new Dropout module with a device name.

Creates a new DeviceContext internally using the provided device name. This constructor is useful for creating standalone modules without pre-existing device contexts.

Parameters

device_name	The name of the device to use (e.g., Device::Cpu(), "CUDA:0").
config	Configuration parameters for the Dropout module.

Exceptions

std::invalid_argument	If the device name is invalid or the configuration is invalid
std::runtime_error	If device type doesn't match template parameter TDeviceType

◆ Dropout() [2/2]

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::Dropout	(	std::shared_ptr< DeviceContext >	device_context,
		const DropoutConfig &	config )

inlineexplicitexport

Constructs a new Dropout module with a provided device context.

Uses a pre-existing DeviceContext instance. This constructor is useful when integrating the module into a larger network that shares device contexts across modules.

Parameters

device_context	The device context to use for this module.
config	Configuration parameters for the Dropout module.

Exceptions

std::invalid_argument	If device_context is null or configuration is invalid
std::runtime_error	If device context type doesn't match template parameter TDeviceType

Member Function Documentation

◆ backward()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

void Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::backward	(	const Tensor< TInput, MR > &	input,
		const Tensor< TOutput, MR > &	output_grad,
		Tensor< TInput, MR > &	input_grad )

inlineexport

Performs the backward pass of the Dropout operation.

Computes the gradient of the Dropout function with respect to its input. During training, gradients are only propagated for non-zeroed elements. The gradient computation is straightforward: multiply the output gradient by the same mask used in the forward pass.

Parameters

input	The input tensor from the forward pass.
output_grad	The gradient of loss with respect to the output.
input_grad	The tensor to store gradients with respect to input.

◆ createOperation()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

void Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::createOperation ( )

inlineexportprivate

Creates the appropriate Dropout operation based on the current device context.

This method initializes the operation_ member with the appropriate implementation of the Dropout operation for either CPU or CUDA, based on the current device context. It also passes the config object to the operation.

Here is the caller graph for this function:

◆ forward()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

void Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::forward	(	const Tensor< TInput, MR > &	input,
		Tensor< TOutput, MR > &	output )

inlineexport

Performs the forward pass of the Dropout operation.

During training, randomly zeroes elements with probability p and scales remaining elements by 1/(1-p). During inference, performs identity operation or scaling based on configuration.

Parameters

input	The input tensor.
output	The output tensor where the results will be stored.

◆ generateMask()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

void Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::generateMask	(	Tensor< TOutput, MR > &	mask,
		const std::vector< int64_t > &	shape )

inlineexportprivate

Generates a new dropout mask for the given shape.

Creates a binary mask with elements set to 0 (drop) with probability p, and scale factor 1/(1-p) (keep) with probability (1-p).

Parameters

mask	The tensor to populate with the mask values
shape	The shape of the input/mask tensor

Here is the caller graph for this function:

◆ getProbability()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

float Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::getProbability ( ) const

inlineexport

Gets the dropout probability used by this module.

Returns: float The probability of zeroing elements

◆ getSeed()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

unsigned int Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::getSeed ( ) const

inlineexport

Gets the current random seed.

Returns: unsigned int The current random seed

◆ getType()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

const ComponentType Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::getType ( ) const

inlineoverrideexportvirtual

Get the component type identifier.

Used for serialization and runtime type identification.

Returns: Component type enum value.

Implements Mila::Dnn::Component< DeviceType::Cuda, float, float >.

◆ load()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

void Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::load ( ModelArchive & archive )

inlineoverrideexport

Deserializes the module state from a ZIP archive.

Implementation of the Module interface for deserialization. Since Dropout has no learnable parameters, this is a no-op implementation.

Parameters

zip	ZIP archive for deserialization

◆ parameterCount()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

size_t Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::parameterCount ( ) const

inlineoverrideexportvirtual

Gets the number of trainable parameters in this module.

The Dropout module has no trainable parameters.

Returns: size_t Always returns 0 for Dropout.

Implements Mila::Dnn::Component< DeviceType::Cuda, float, float >.

◆ save()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

void Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::save ( ModelArchive & archive ) const

inlineoverrideexport

Serializes the module state to a ZIP archive.

Implementation of the Module interface for serialization. Since Dropout has no learnable parameters, this is a no-op implementation.

Parameters

zip	ZIP archive for serialization

◆ setSeed()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

void Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::setSeed ( unsigned int seed )

inlineexport

Sets the random seed for dropout mask generation.

This allows for reproducible dropout patterns when needed.

Parameters

seed	The random seed value

◆ toString()

template<DeviceType TDeviceType = DeviceType::Cuda, typename TInput = float, typename TOutput = TInput>

std::string Mila::Dnn::Dropout< TDeviceType, TInput, TOutput >::toString ( ) const

inlineoverrideexportvirtual

Generates a string representation of this module's configuration.

Returns: std::string A formatted string with module information

Implements Mila::Dnn::Component< DeviceType::Cuda, float, float >.

The documentation for this class was generated from the following file:

/__w/Mila/Mila/Mila/Src/Dnn/Components/Regularization/Dropout.ixx

Public Types

Public Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ Dropout() [1/2]

◆ Dropout() [2/2]

Member Function Documentation

◆ backward()

◆ createOperation()

◆ forward()

◆ generateMask()

◆ getProbability()

◆ getSeed()

◆ getType()

◆ load()

◆ parameterCount()

◆ save()

◆ setSeed()

◆ toString()