Skip to main content

Module layer

Module layer 

Source
Expand description

Differentiable neural network building blocks for quantized and dense LLMs.

Structs§

AttentionParams
Parameters controlling scaled dot-product attention inside decoder layers.
Embedding
Token or positional embedding table (vocab_size, embed_dim).
KvCache
Layer-indexed key/value cache for autoregressive generation.
LayerNorm
Differentiable LayerNorm using candle_nn::ops::layer_norm_slow (for OPT).
LinearLoadSpec
Bundle of parameters specifying how to load a LinearLayer from a state dict.
RmsNorm
Differentiable RMSNorm layer using candle_nn::ops::rms_norm_slow so QAT backprop works.
RmsNormNoScale
Weightless RMSNorm used for Gemma 4 value normalization (v_norm) and router input norm.
RotaryEmbedding
Precomputed Rotary Position Embedding (cos, sin) supporting differentiable rope_slow as well as partial and proportional RoPE (p-RoPE).

Enums§

LinearLayer
Unified linear projection supporting dense weights, differentiable LittleBit QAT layers, and compact 1-bit packed inference layers.
NormKind
RMSNorm variant (standard vs. Gemma +1.0 offset).
RopeMode
Rotary Position Embedding frequency parameterization mode.

Functions§

apply_activation
Apply activation function matching Hugging Face hidden_act.
causal_mask
Build a causal additive attention mask (1, 1, seq_len, total_len) with -1e9 on masked positions.
repeat_kv
Repeat key/value heads n_rep times along dimension 1 for Grouped-Query Attention (GQA).