Expand description
Gemma 4 Dense and Mixture-of-Experts (MoE) decoder architectures.
Implements Hugging Face Gemma4TextModel / Gemma4ForConditionalGeneration text backbone:
- Hybrid per-layer
sliding_attentionandfull_attentionschedules - Heterogeneous sliding vs. global head dimensions and KV head counts
- Optional
attention_k_eq_von global (full_attention) layers (sharingk_projforVprior tok_norm/RoPE + scalelessv_norm) - Standard (non-offset)
Gemma4RMSNorm(NormKind::Standard) plus weightlessRmsNormNoScaleforv_normand router input norm - Unit attention scaling (
scale = 1.0) and Proportional RoPE (p-RoPE) - Optional dual-path FFN combining a shared dense MLP with sparse top-k routed MoE (
Gemma4Router+Gemma4Experts) - Per-layer residual
layer_scalarmultiplier
Structsยง
- Gemma4
Expert - Single routed SwiGLU / GeGLU expert MLP inside
Gemma4Experts. - Gemma4
Experts - Collection of routed Mixture-of-Experts feed-forward networks with sparse dispatch.
- Gemma4
MoE - Complete Mixture-of-Experts sub-block inside a
Gemma4Modeldecoder layer. - Gemma4
Model - Causal language model supporting Gemma 4 Dense and Mixture-of-Experts checkpoints.
- Gemma4
Router - Top-k softmax router for Gemma 4 Mixture-of-Experts blocks.
- Router
Dispatch - Sparse expert routing decision emitted by
Gemma4Router.