Expand description
Differentiable neural network building blocks for quantized and dense LLMs.
Structs§
- Attention
Params - Parameters controlling scaled dot-product attention inside decoder layers.
- Embedding
- Token or positional embedding table
(vocab_size, embed_dim). - KvCache
- Layer-indexed key/value cache for autoregressive generation.
- Layer
Norm - Differentiable LayerNorm using
candle_nn::ops::layer_norm_slow(for OPT). - Linear
Load Spec - Bundle of parameters specifying how to load a
LinearLayerfrom a state dict. - RmsNorm
- Differentiable RMSNorm layer using
candle_nn::ops::rms_norm_slowso QAT backprop works. - RmsNorm
NoScale - Weightless RMSNorm used for Gemma 4 value normalization (
v_norm) and router input norm. - Rotary
Embedding - Precomputed Rotary Position Embedding (
cos,sin) supporting differentiablerope_slowas well as partial and proportional RoPE (p-RoPE).
Enums§
- Linear
Layer - Unified linear projection supporting dense weights, differentiable LittleBit QAT layers, and compact 1-bit packed inference layers.
- Norm
Kind - RMSNorm variant (standard vs. Gemma
+1.0offset). - Rope
Mode - Rotary Position Embedding frequency parameterization mode.
Functions§
- apply_
activation - Apply activation function matching Hugging Face
hidden_act. - causal_
mask - Build a causal additive attention mask
(1, 1, seq_len, total_len)with-1e9on masked positions. - repeat_
kv - Repeat key/value heads
n_reptimes along dimension 1 for Grouped-Query Attention (GQA).