CUDA implementation of the GELU activation function for neural networks. More...

Inheritance diagram for Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >:

[legend]

Collaboration diagram for Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >:

[legend]

Public Types
using	CudaExecutionContext = ExecutionContext<DeviceType::Cuda>
using	MR = CudaDeviceMemoryResource
using	NativeType = typename Mila::Dnn::Compute::Cuda::TensorDataTypeMap<TPrecision>::device_type
using	TensorType = Tensor<TPrecision, MR>
using	UnaryOperationBase = UnaryOperation<DeviceType::Cuda, TPrecision>
Public Types inherited from Mila::Dnn::Compute::UnaryOperation< DeviceType::Cuda, TPrecision >
using	MR
using	TensorInputType
using	TensorOutputType
Public Types inherited from Mila::Dnn::Compute::Operation< TDeviceType, TInput >
using	DataTypeTraits

Public Member Functions
	CudaGeluOp (IExecutionContext *context, const GeluConfig &config)
void	backward (const ITensor &input, const ITensor &output_gradient, ITensor &input_gradient) const override
	Performs the backward pass of the GELU activation function.
void	forward (const ITensor &input, ITensor &output) const override
	Performs the forward pass of the GELU activation function on CUDA.
std::string	getName () const override
	Gets the name of this operation.
OperationType	getOperationType () const override
	Operation type identifier.
Public Member Functions inherited from Mila::Dnn::Compute::UnaryOperation< DeviceType::Cuda, TPrecision >
virtual	~UnaryOperation ()=default
Public Member Functions inherited from Mila::Dnn::Compute::Operation< TDeviceType, TInput >
virtual	~Operation ()=default
virtual void	build (const BuildContext &build_context)
	Prepare the operation for a concrete input shape.
virtual void	clearGradients () noexcept
	Clear any cached gradient pointers held by the operation.
virtual TensorDataType	getDataType () const
	Tensor data type for this operation.
virtual DeviceType	getDeviceType () const
	Device type for this operation.
virtual std::size_t	getStateMemorySize () const
	Returns the number of bytes of state memory allocated by this operation.
virtual bool	isBuilt () const
	Whether build() completed successfully for a concrete input shape.
virtual bool	isEvalMode () const
	Query whether operation is configured for training.
virtual void	setGradients (ITensor weight_grad, ITensor bias_grad)
	Bind module-owned gradient tensors to the operation.
virtual void	setParameters (ITensor weight, ITensor bias)
	Bind module-owned parameter tensors to the operation.
virtual void	setTrainingMode (TrainingMode training_mode)
	Configure operation training-mode behavior.

Private Attributes
GeluConfig	config_
	Configuration for the GELU operation.
CudaExecutionContext *	context_
	Execution context for CUDA resources.
Detail::cuda_gelu_impl< NativeType >	impl_
	Implementation details for the GELU operation.

Additional Inherited Members
Static Public Attributes inherited from Mila::Dnn::Compute::Operation< TDeviceType, TInput >
static constexpr TensorDataType	data_type
static constexpr DeviceType	device_type
Static Protected Member Functions inherited from Mila::Dnn::Compute::UnaryOperation< DeviceType::Cuda, TPrecision >
static const TensorInputType &	asInputTensor (const ITensor &t)
static TensorOutputType &	asOutputTensor (ITensor &t)
Protected Attributes inherited from Mila::Dnn::Compute::Operation< TDeviceType, TInput >
bool	is_built_
TrainingMode	training_mode_

Detailed Description

template<TensorDataType TPrecision>
requires ValidFloatTensorDataType<TPrecision>
class Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >

CUDA implementation of the GELU activation function for neural networks.

This class provides a CUDA-based implementation of the Gaussian Error Linear Unit (GELU) activation function, which is commonly used in transformer architectures. GELU is a smooth approximation of the ReLU function that applies a non-linear transformation to its input.

The implementation leverages CUDA for GPU acceleration, providing efficient computation for large neural network models. It also supports different precision modes via the ComputePrecision policy.

Template Parameters

TPrecision Abstract TensorDataType for the operation (FP32, FP16, BF16, etc.).

Member Typedef Documentation

◆ CudaExecutionContext

template<TensorDataType TPrecision>

using Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::CudaExecutionContext = ExecutionContext<DeviceType::Cuda>

◆ MR

template<TensorDataType TPrecision>

using Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::MR = CudaDeviceMemoryResource

◆ NativeType

template<TensorDataType TPrecision>

using Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::NativeType = typename Mila::Dnn::Compute::Cuda::TensorDataTypeMap<TPrecision>::device_type

◆ TensorType

template<TensorDataType TPrecision>

using Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::TensorType = Tensor<TPrecision, MR>

◆ UnaryOperationBase

template<TensorDataType TPrecision>

using Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::UnaryOperationBase = UnaryOperation<DeviceType::Cuda, TPrecision>

Constructor & Destructor Documentation

◆ CudaGeluOp()

template<TensorDataType TPrecision>

Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::CudaGeluOp	(	IExecutionContext *	context,
		const GeluConfig &	config )

inline

Member Function Documentation

◆ backward()

template<TensorDataType TPrecision>

void Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::backward	(	const ITensor &	input,
		const ITensor &	output_gradient,
		ITensor &	input_gradient ) const

inlineoverridevirtual

Performs the backward pass of the GELU activation function.

Computes gradients with respect to inputs for the GELU function. The precision policy affects the computation in the same way as the forward pass.

Parameters

input	Input tensor from the forward pass.
output	Output tensor from the forward pass.
output_gradient	Gradient of the loss with respect to the output.
parameters	Parameters tensor from forward pass (not used).
parameter_gradients	Gradients for parameters (not used).
input_gradient	Gradient of the loss with respect to the input.
output_state	Cache tensors from forward pass.

Implements Mila::Dnn::Compute::UnaryOperation< DeviceType::Cuda, TPrecision >.

◆ forward()

template<TensorDataType TPrecision>

void Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::forward	(	const ITensor &	input,
		ITensor &	output ) const

inlineoverridevirtual

Performs the forward pass of the GELU activation function on CUDA.

Computes the GELU transformation of the input elements: GELU(x) = 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3)))

The precision policy affects how the computation is performed:

Performance: May use faster but less precise algorithms
Accuracy: Will use the most accurate algorithm available
Auto: Will select an appropriate balance based on the hardware
Native: Use native precision of the tensor type

Parameters

input	Input tensor containing the values to transform.
parameters	Additional parameters (not used in this operation).
output	Output tensor to store the transformed values.
output_state	Cache for intermediate results (not used in this operation).

Implements Mila::Dnn::Compute::UnaryOperation< DeviceType::Cuda, TPrecision >.

◆ getName()

template<TensorDataType TPrecision>

std::string Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::getName ( ) const

inlineoverridevirtual

Gets the name of this operation.

Returns: std::string The name of the operation ("Cuda::GeluOp").

Implements Mila::Dnn::Compute::Operation< TDeviceType, TInput >.

◆ getOperationType()

template<TensorDataType TPrecision>

OperationType Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::getOperationType ( ) const

inlineoverridevirtual

Operation type identifier.

Implements Mila::Dnn::Compute::Operation< TDeviceType, TInput >.

Member Data Documentation

◆ config_

template<TensorDataType TPrecision>

GeluConfig Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::config_

private

Configuration for the GELU operation.

◆ context_

template<TensorDataType TPrecision>

CudaExecutionContext* Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::context_

private

Execution context for CUDA resources.

◆ impl_

template<TensorDataType TPrecision>

Detail::cuda_gelu_impl<NativeType> Mila::Dnn::Compute::Cuda::Gelu::CudaGeluOp< TPrecision >::impl_

private

Implementation details for the GELU operation.

The documentation for this class was generated from the following file:

/__w/Mila/Mila/Mila/Src/Dnn/Compute/Devices/Cuda/Operations/Activations/Gelu/CudaGeluOp.ixx

Public Types

Public Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Member Typedef Documentation

◆ CudaExecutionContext

◆ MR

◆ NativeType

◆ TensorType

◆ UnaryOperationBase

Constructor & Destructor Documentation

◆ CudaGeluOp()

Member Function Documentation

◆ backward()

◆ forward()

◆ getName()

◆ getOperationType()

Member Data Documentation

◆ config_

◆ context_

◆ impl_