pub struct RmsNorm {
pub weight: Param,
pub eps: f32,
pub kind: NormKind,
}Expand description
Differentiable RMSNorm layer using candle_nn::ops::rms_norm_slow so QAT backprop works.
Fields§
§weight: ParamScale weight (dim,).
eps: f32Numerical stability epsilon.
kind: NormKindStandard vs. +1.0 weight offset.
Implementations§
Source§impl RmsNorm
impl RmsNorm
Sourcepub fn from_tensors(
prefix: &str,
tensors: &HashMap<String, Tensor>,
eps: f64,
kind: NormKind,
dtype: DType,
device: &Device,
) -> Result<Self>
pub fn from_tensors( prefix: &str, tensors: &HashMap<String, Tensor>, eps: f64, kind: NormKind, dtype: DType, device: &Device, ) -> Result<Self>
Sourcepub fn new_init(
dim: usize,
eps: f64,
kind: NormKind,
dtype: DType,
device: &Device,
) -> Result<Self>
pub fn new_init( dim: usize, eps: f64, kind: NormKind, dtype: DType, device: &Device, ) -> Result<Self>
Construct unit-initialized RMSNorm (zero-initialized for NormKind::PlusOne).
§Errors
Returns an error if tensor creation fails.
Sourcepub fn forward(&self, x: &Tensor) -> Result<Tensor>
pub fn forward(&self, x: &Tensor) -> Result<Tensor>
Forward pass using differentiable rms_norm_slow.
§Errors
Returns an error if norm computation fails.
Sourcepub fn trainable_vars(&self) -> Vec<Var>
pub fn trainable_vars(&self) -> Vec<Var>
Collect trainable Var if unfrozen.
Sourcepub fn make_trainable(&mut self) -> Result<()>
pub fn make_trainable(&mut self) -> Result<()>
Sourcepub fn pack_state_dict(&self, prefix: &str, out: &mut HashMap<String, Tensor>)
pub fn pack_state_dict(&self, prefix: &str, out: &mut HashMap<String, Tensor>)
Export {prefix}.weight.
Trait Implementations§
Auto Trait Implementations§
impl !RefUnwindSafe for RmsNorm
impl !UnwindSafe for RmsNorm
impl Freeze for RmsNorm
impl Send for RmsNorm
impl Sync for RmsNorm
impl Unpin for RmsNorm
impl UnsafeUnpin for RmsNorm
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> ErasedDestructor for Twhere
T: 'static,
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
impl<T, U> Imply<T> for U
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more