Skip to main content

Module gemma4

Module gemma4 

Source
Expand description

Gemma 4 Dense and Mixture-of-Experts (MoE) decoder architectures.

Implements Hugging Face Gemma4TextModel / Gemma4ForConditionalGeneration text backbone:

  • Hybrid per-layer sliding_attention and full_attention schedules
  • Heterogeneous sliding vs. global head dimensions and KV head counts
  • Optional attention_k_eq_v on global (full_attention) layers (sharing k_proj for V prior to k_norm/RoPE + scaleless v_norm)
  • Standard (non-offset) Gemma4RMSNorm (NormKind::Standard) plus weightless RmsNormNoScale for v_norm and router input norm
  • Unit attention scaling (scale = 1.0) and Proportional RoPE (p-RoPE)
  • Optional dual-path FFN combining a shared dense MLP with sparse top-k routed MoE (Gemma4Router + Gemma4Experts)
  • Per-layer residual layer_scalar multiplier

Structsยง

Gemma4Expert
Single routed SwiGLU / GeGLU expert MLP inside Gemma4Experts.
Gemma4Experts
Collection of routed Mixture-of-Experts feed-forward networks with sparse dispatch.
Gemma4MoE
Complete Mixture-of-Experts sub-block inside a Gemma4Model decoder layer.
Gemma4Model
Causal language model supporting Gemma 4 Dense and Mixture-of-Experts checkpoints.
Gemma4Router
Top-k softmax router for Gemma 4 Mixture-of-Experts blocks.
RouterDispatch
Sparse expert routing decision emitted by Gemma4Router.