CPU implementation of the residual operation for neural networks. More...

Inheritance diagram for Mila::Dnn::Compute::CpuResidualOp:

[legend]

Collaboration diagram for Mila::Dnn::Compute::CpuResidualOp:

[legend]

Public Types
using	MR = typename CpuDevice::MR

using	OperationBase = BinaryOperation< DeviceType::Cpu, float, float, float >

Public Types inherited from Mila::Dnn::Compute::BinaryOperation< DeviceType::Cpu, float, float, float >
using	MR = std::conditional_t< TDeviceType==DeviceType::Cuda, CudaMemoryResource, HostMemoryResource >
	Memory resource type based on device type.

Public Member Functions
	CpuResidualOp (const ResidualConfig &config)
	Constructs a new CPU Residual operation with the default device context.

	CpuResidualOp (std::shared_ptr< DeviceContext > context, const ResidualConfig &config)
	Constructs a new CPU Residual operation with a specific device context.

void	backward (const Tensor< float, MR > &input_a, const Tensor< float, MR > &input_b, const Tensor< float, MR > &output, const Tensor< float, MR > &output_gradient, const std::vector< std::shared_ptr< Tensor< float, MR > > > &parameters, std::vector< std::shared_ptr< Tensor< float, MR > > > &parameter_gradients, Tensor< float, MR > &input_a_gradient, Tensor< float, MR > &input_b_gradient, const OperationAttributes &attributes, const std::vector< std::shared_ptr< Tensor< float, MR > > > &output_state) const
	Performs the backward pass of the residual operation.

void	backward_impl (float dinp1, float dinp2, const float *dout, int N) const
	Helper method for backward pass implementation.

void	forward (const Tensor< float, MR > &input_a, const Tensor< float, MR > &input_b, const std::vector< std::shared_ptr< Tensor< float, MR > > > &parameters, const OperationAttributes &attributes, Tensor< float, MR > &output, std::vector< std::shared_ptr< Tensor< float, MR > > > &output_state) const override
	Performs the forward pass of the residual operation.

std::string	getName () const override
	Gets the name of this operation.

Public Member Functions inherited from Mila::Dnn::Compute::BinaryOperation< DeviceType::Cpu, float, float, float >
	BinaryOperation (OperationType operation_type)
	Constructs a BinaryOperation with the specified operation type and precision policy.

	BinaryOperation (OperationType operation_type, std::shared_ptr< DeviceContext > context)
	Constructs a BinaryOperation with the specified operation type, device context, and precision policy.

virtual	~BinaryOperation ()=default
	Virtual destructor for proper cleanup of derived classes.

virtual void	backward (const Tensor< float, MR > &input1, const Tensor< float, MR > &input2, const Tensor< float, MR > &output, const Tensor< float, MR > &output_gradient, const std::vector< std::shared_ptr< Tensor< float, MR > > > &parameters, std::vector< std::shared_ptr< Tensor< float, MR > > > &parameter_gradients, Tensor< float, MR > &input1_gradient, Tensor< float, MR > &input2_gradient, const OperationAttributes &attributes, const std::vector< std::shared_ptr< Tensor< float, MR > > > &output_state) const
	Executes the backward pass of a binary operation.

virtual void	forward (const Tensor< float, MR > &input1, const Tensor< float, MR > &input2, const std::vector< std::shared_ptr< Tensor< float, MR > > > &parameters, const OperationAttributes &attributes, Tensor< float, MR > &output, std::vector< std::shared_ptr< Tensor< float, MR > > > &output_state) const=0
	Executes the forward pass of a binary operation.

Public Member Functions inherited from Mila::Dnn::Compute::OperationBase< TDeviceType, TInput1, TInput2, TOutput >
	OperationBase (OperationType operation_type, std::shared_ptr< DeviceContext > context)
	Constructs an OperationBase object with a specific device context and compute precision.

virtual	~OperationBase ()=default
	Virtual destructor for the OperationBase class.

std::shared_ptr< DeviceContext >	getDeviceContext () const
	Gets the device context associated with this operation.

DeviceType	getDeviceType () const
	Gets the device type for this operation.

OperationType	getOperationType () const
	Gets the operation type enumeration value.

Static Public Member Functions
static const std::string &	className ()
	Gets the class name of this operation.

Detailed Description

CPU implementation of the residual operation for neural networks.

This class provides a CPU-based implementation of the residual operation, which performs element-wise addition of two input tensors. It is commonly used in residual connections in neural network architectures such as ResNet and Transformers to help with gradient flow.

Template Parameters

TInput	The data type of the input tensor elements.
TDataType	The data type used for computation and output (defaults to the input type).

Member Typedef Documentation

◆ MR

using Mila::Dnn::Compute::CpuResidualOp::MR = typename CpuDevice::MR

◆ OperationBase

using Mila::Dnn::Compute::CpuResidualOp::OperationBase = BinaryOperation<DeviceType::Cpu, float, float, float>

Constructor & Destructor Documentation

◆ CpuResidualOp() [1/2]

Mila::Dnn::Compute::CpuResidualOp::CpuResidualOp ( const ResidualConfig & config )

inline

Constructs a new CPU Residual operation with the default device context.

Initializes the operation with a CPU device context.

◆ CpuResidualOp() [2/2]

Mila::Dnn::Compute::CpuResidualOp::CpuResidualOp	(	std::shared_ptr< DeviceContext >	context,
		const ResidualConfig &	config
	)

inline

Constructs a new CPU Residual operation with a specific device context.

Parameters

context The device context to use for this operation.

Exceptions

std::runtime_error If the context is not for a CPU device.

Member Function Documentation

◆ backward()

void Mila::Dnn::Compute::CpuResidualOp::backward	(	const Tensor< float, MR > &	input_a,
		const Tensor< float, MR > &	input_b,
		const Tensor< float, MR > &	output,
		const Tensor< float, MR > &	output_gradient,
		const std::vector< std::shared_ptr< Tensor< float, MR > > > &	parameters,
		std::vector< std::shared_ptr< Tensor< float, MR > > > &	parameter_gradients,
		Tensor< float, MR > &	input_a_gradient,
		Tensor< float, MR > &	input_b_gradient,
		const OperationAttributes &	attributes,
		const std::vector< std::shared_ptr< Tensor< float, MR > > > &	output_state
	)		const

inline

Performs the backward pass of the residual operation.

Computes the gradient with respect to each input by adding the output gradient to each input gradient.

Parameters

input_a	First input tensor from the forward pass.
input_b	Second input tensor from the forward pass.
output	Output tensor from the forward pass.
output_gradient	Gradient of the loss with respect to the output.
parameters	Parameters used in forward pass (not used in this operation).
parameter_gradients	Gradients for parameters (not used in this operation).
input_a_gradient	Gradient of the loss with respect to input_a.
input_b_gradient	Gradient of the loss with respect to input_b.
attributes	Additional attributes for the operation.
output_state	Cache tensors from forward pass.

Here is the call graph for this function:

◆ backward_impl()

void Mila::Dnn::Compute::CpuResidualOp::backward_impl	(	float *	dinp1,
		float *	dinp2,
		const float *	dout,
		int	N
	)		const

inline

Helper method for backward pass implementation.

Parameters

dinp1	Pointer to the gradient buffer for the first input.
dinp2	Pointer to the gradient buffer for the second input.
dout	Pointer to the gradient buffer for the output.
N	Size of the gradient buffers.

Here is the caller graph for this function:

◆ className()

static const std::string & Mila::Dnn::Compute::CpuResidualOp::className ( )

inlinestatic

Gets the class name of this operation.

Returns: const std::string& The class name of the operation.

◆ forward()

void Mila::Dnn::Compute::CpuResidualOp::forward	(	const Tensor< float, MR > &	input_a,
		const Tensor< float, MR > &	input_b,
		const std::vector< std::shared_ptr< Tensor< float, MR > > > &	parameters,
		const OperationAttributes &	attributes,
		Tensor< float, MR > &	output,
		std::vector< std::shared_ptr< Tensor< float, MR > > > &	output_state
	)		const

inlineoverride

Performs the forward pass of the residual operation.

Adds two input tensors element-wise and stores the result in the output tensor.

Parameters

input_a	The first input tensor.
input_b	The second input tensor.
parameters	Additional parameters (not used in this operation).
attributes	Additional attributes for the operation.
output	The output tensor where the results will be stored.
output_state	Cache for storing intermediate results (used in backward pass).

Here is the call graph for this function:

◆ getName()

std::string Mila::Dnn::Compute::CpuResidualOp::getName ( ) const

inlineoverridevirtual

Gets the name of this operation.

Returns: std::string The name of the operation ("Cpu::ResidualOp").

Implements Mila::Dnn::Compute::OperationBase< TDeviceType, TInput1, TInput2, TOutput >.

The documentation for this class was generated from the following file:

/home/runner/work/Mila/Mila/Mila/Src/Dnn/Compute/Operations/Cpu/CpuResidualOp.ixx

Public Types

Public Member Functions

Static Public Member Functions

Detailed Description

Member Typedef Documentation

◆ MR

◆ OperationBase

Constructor & Destructor Documentation

◆ CpuResidualOp() [1/2]

◆ CpuResidualOp() [2/2]

Member Function Documentation

◆ backward()

◆ backward_impl()

◆ className()

◆ forward()

◆ getName()