RmsNorm.Config.ixx File Reference

Configuration for RMS Normalization component. More...

#include <stdexcept>
#include <string>
#include <sstream>
#include <optional>
#include <utility>
import Serialization.Metadata;
import Dnn.TensorTypes;
import Dnn.ComponentConfig;
import Dnn.Component;

Classes
class	Mila::Dnn::RmsNormConfig

Namespaces
namespace	Mila
	Mila main API namespace.
namespace	Mila::Dnn

Detailed Description

Configuration for RMS Normalization component.

Design principle (Mila-wide):

• Constructor parameters are structurally required — no sensible default exists.
• Fluent setters are reserved for optional behavioural parameters that have well-known defaults. There are no fluent overrides for constructor parameters.

RmsNormConfig supports two mutually exclusive normalization modes selected by constructor overload:

• Shape mode: RmsNormConfig( shape_t ) — normalize over a trailing shape.
• Axis mode: RmsNormConfig( int64_t ) — normalize over a single axis.

The overloads are unambiguous — shape_t and int64_t cannot collide.

Typical usage:

// Shape mode (most common for transformers)
auto cfg = RmsNormConfig( shape_t{ model_dim } )
    .withEpsilon( config_.getRMSNormEpsilon() )
    .withBias( false );
 
// Axis mode
auto cfg = RmsNormConfig( int64_t{ -1 } )
    .withBias( false );