CUDA implementation of the residual operation for neural networks. More...

Inheritance diagram for Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >:

[legend]

Collaboration diagram for Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >:

[legend]

Public Types
using	BinaryOperationBase = BinaryOperation< DeviceType::Cuda, TInput, TInput, TOutput >

using	MR = typename CudaDevice::MR

Public Types inherited from Mila::Dnn::Compute::BinaryOperation< TDeviceType, TInput1, TInput2, TOutput >
using	MR = std::conditional_t< TDeviceType==DeviceType::Cuda, CudaMemoryResource, HostMemoryResource >
	Memory resource type based on device type.

Public Member Functions
	CudaResidualOp (const ResidualConfig &config)
	Constructs a new CUDA Residual operation with the default device context.

	CudaResidualOp (std::shared_ptr< DeviceContext > context, const ResidualConfig &config)
	Constructs a new CUDA Residual operation with a specific device context.

void	backward (const Tensor< TInput, MR > &input1, const Tensor< TInput, MR > &input2, const Tensor< TOutput, MR > &output, const Tensor< TOutput, MR > &output_gradient, const std::vector< std::shared_ptr< Tensor< TInput, MR > > > &parameters, std::vector< std::shared_ptr< Tensor< TOutput, MR > > > &parameter_gradients, Tensor< TInput, MR > &input1_gradient, Tensor< TInput, MR > &input2_gradient, const OperationAttributes &properties, const std::vector< std::shared_ptr< Tensor< TOutput, MR > > > &output_state) const
	Performs the backward pass of the residual operation.

void	forward (const Tensor< TInput, MR > &input1, const Tensor< TInput, MR > &input2, const std::vector< std::shared_ptr< Tensor< TInput, MR > > > &parameters, const OperationAttributes &properties, Tensor< TOutput, MR > &output, std::vector< std::shared_ptr< Tensor< TOutput, MR > > > &output_state) const override
	Performs the forward pass of the residual operation on CUDA.

std::string	getName () const override
	Gets the name of this operation.

Public Member Functions inherited from Mila::Dnn::Compute::BinaryOperation< TDeviceType, TInput1, TInput2, TOutput >
	BinaryOperation (OperationType operation_type)
	Constructs a BinaryOperation with the specified operation type and precision policy.

	BinaryOperation (OperationType operation_type, std::shared_ptr< DeviceContext > context)
	Constructs a BinaryOperation with the specified operation type, device context, and precision policy.

virtual	~BinaryOperation ()=default
	Virtual destructor for proper cleanup of derived classes.

virtual void	backward (const Tensor< TInput1, MR > &input1, const Tensor< TInput2, MR > &input2, const Tensor< TOutput, MR > &output, const Tensor< TOutput, MR > &output_gradient, const std::vector< std::shared_ptr< Tensor< TInput1, MR > > > &parameters, std::vector< std::shared_ptr< Tensor< TOutput, MR > > > &parameter_gradients, Tensor< TInput1, MR > &input1_gradient, Tensor< TInput2, MR > &input2_gradient, const OperationAttributes &attributes, const std::vector< std::shared_ptr< Tensor< TOutput, MR > > > &output_state) const
	Executes the backward pass of a binary operation.

virtual void	forward (const Tensor< TInput1, MR > &input1, const Tensor< TInput2, MR > &input2, const std::vector< std::shared_ptr< Tensor< TInput1, MR > > > &parameters, const OperationAttributes &attributes, Tensor< TOutput, MR > &output, std::vector< std::shared_ptr< Tensor< TOutput, MR > > > &output_state) const =0
	Executes the forward pass of a binary operation.

Public Member Functions inherited from Mila::Dnn::Compute::OperationBase< TDeviceType, TInput1, TInput2, TOutput >
	OperationBase (OperationType operation_type, std::shared_ptr< DeviceContext > context)
	Constructs an OperationBase object with a specific device context and compute precision.

virtual	~OperationBase ()=default
	Virtual destructor for the OperationBase class.

std::shared_ptr< DeviceContext >	getDeviceContext () const
	Gets the device context associated with this operation.

DeviceType	getDeviceType () const
	Gets the device type for this operation.

OperationType	getOperationType () const
	Gets the operation type enumeration value.

Private Attributes
ResidualConfig	config_
	Configuration for the residual operation.

Detailed Description

template<typename TInput, typename TOutput = TInput>
requires ValidFloatTensorTypes<TInput, TOutput>
class Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >

CUDA implementation of the residual operation for neural networks.

This class provides a CUDA-based implementation of the residual operation, which performs element-wise addition of two input tensors. It is commonly used in residual connections in neural network architectures such as ResNet and Transformers to help with gradient flow and mitigate the vanishing gradient problem. The implementation is optimized for NVIDIA GPUs.

The implementation leverages CUDA for GPU acceleration, providing efficient computation for large neural network models. It also supports different precision modes via the ComputePrecision policy.

Template Parameters

TInput	The data type of both input tensor elements.
TOutput	The data type of the output tensor elements (defaults to TInput).

Member Typedef Documentation

◆ BinaryOperationBase

template<typename TInput , typename TOutput = TInput>

using Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >::BinaryOperationBase = BinaryOperation<DeviceType::Cuda, TInput, TInput, TOutput>

◆ MR

template<typename TInput , typename TOutput = TInput>

using Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >::MR = typename CudaDevice::MR

Constructor & Destructor Documentation

◆ CudaResidualOp() [1/2]

template<typename TInput , typename TOutput = TInput>

Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >::CudaResidualOp ( const ResidualConfig & config )

inline

Constructs a new CUDA Residual operation with the default device context.

Initializes the operation with a CUDA device context (defaults to CUDA:0).

Parameters

precision_policy The precision policy to use for mixed precision computation.

◆ CudaResidualOp() [2/2]

template<typename TInput , typename TOutput = TInput>

Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >::CudaResidualOp	(	std::shared_ptr< DeviceContext >	context,
		const ResidualConfig &	config
	)

inline

Constructs a new CUDA Residual operation with a specific device context.

Parameters

context	The device context to use for this operation.
precision_policy	The precision policy to use for mixed precision computation.

Exceptions

std::runtime_error If the context is not for a CUDA device.

Member Function Documentation

◆ backward()

template<typename TInput , typename TOutput = TInput>

void Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >::backward	(	const Tensor< TInput, MR > &	input1,
		const Tensor< TInput, MR > &	input2,
		const Tensor< TOutput, MR > &	output,
		const Tensor< TOutput, MR > &	output_gradient,
		const std::vector< std::shared_ptr< Tensor< TInput, MR > > > &	parameters,
		std::vector< std::shared_ptr< Tensor< TOutput, MR > > > &	parameter_gradients,
		Tensor< TInput, MR > &	input1_gradient,
		Tensor< TInput, MR > &	input2_gradient,
		const OperationAttributes &	properties,
		const std::vector< std::shared_ptr< Tensor< TOutput, MR > > > &	output_state
	)		const

inline

Performs the backward pass of the residual operation.

Computes gradients with respect to both inputs by propagating the output gradient to each input.

The precision policy affects the computation in the same way as the forward pass.

Parameters

input1	First input tensor from the forward pass.
input2	Second input tensor from the forward pass.
output	Output tensor from the forward pass.
output_gradient	Gradient of the loss with respect to the output.
parameters	Parameters tensor from forward pass (not used in this operation).
parameter_gradients	Gradients for parameters (not used in this operation).
input1_gradient	Gradient of the loss with respect to input1.
input2_gradient	Gradient of the loss with respect to input2.
properties	Additional attributes for the operation.
output_state	Cache tensors from forward pass (not used in this operation).

Here is the call graph for this function:

◆ forward()

template<typename TInput , typename TOutput = TInput>

void Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >::forward	(	const Tensor< TInput, MR > &	input1,
		const Tensor< TInput, MR > &	input2,
		const std::vector< std::shared_ptr< Tensor< TInput, MR > > > &	parameters,
		const OperationAttributes &	properties,
		Tensor< TOutput, MR > &	output,
		std::vector< std::shared_ptr< Tensor< TOutput, MR > > > &	output_state
	)		const

inlineoverride

Performs the forward pass of the residual operation on CUDA.

Adds two input tensors element-wise and stores the result in the output tensor. The computation is performed on the GPU using CUDA kernels for optimal performance.

The precision policy affects how the computation is performed:

Performance: May use faster but less precise algorithms
Accuracy: Will use the most accurate algorithm available
Auto: Will select an appropriate balance based on the hardware
Disabled: Will use the standard precision of the input/output types

Parameters

input1	The first input tensor to be added.
input2	The second input tensor to be added.
parameters	Additional parameters (not used in this operation).
properties	Additional attributes for the operation.
output	The output tensor where the results will be stored.
output_state	Cache for intermediate results (not used in this operation).

Here is the call graph for this function:

◆ getName()

template<typename TInput , typename TOutput = TInput>

std::string Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >::getName ( ) const

inlineoverridevirtual

Gets the name of this operation.

Returns: std::string The name of the operation ("Cuda::ResidualOp").

Implements Mila::Dnn::Compute::OperationBase< TDeviceType, TInput1, TInput2, TOutput >.

Member Data Documentation

◆ config_

template<typename TInput , typename TOutput = TInput>

ResidualConfig Mila::Dnn::Compute::CudaResidualOp< TInput, TOutput >::config_

private

Configuration for the residual operation.

The documentation for this class was generated from the following file:

/home/runner/work/Mila/Mila/Mila/Src/Dnn/Compute/Operations/Cuda/CudaResidualOp.ixx

Public Types

Public Member Functions

Private Attributes

Detailed Description

Member Typedef Documentation

◆ BinaryOperationBase

◆ MR

Constructor & Destructor Documentation

◆ CudaResidualOp() [1/2]

◆ CudaResidualOp() [2/2]

Member Function Documentation

◆ backward()

◆ forward()

◆ getName()

Member Data Documentation

◆ config_