Skip to main content

Module loss

Module loss 

Source
Expand description

Knowledge distillation KL and layer-to-layer hidden-state MSE losses (utils/kd_utils.py).

Structs§

KdLossBreakdown
Scalar + differentiable breakdown returned by compute_kd_loss.

Functions§

compute_kd_loss
Compute combined Knowledge Distillation loss (KD + L2L MSE) from student and teacher outputs.
kl_divergence_batchmean
Compute F.kl_div(F.log_softmax(student, -1), F.softmax(teacher, -1), reduction="batchmean") in F32 for numerical stability.
l2l_mse_loss
Compute layer-to-layer MSE loss summed over aligned transformer hidden states and scaled by scale.