Expand description
Knowledge distillation KL and layer-to-layer hidden-state MSE losses (utils/kd_utils.py).
Structs§
- KdLoss
Breakdown - Scalar + differentiable breakdown returned by
compute_kd_loss.
Functions§
- compute_
kd_ loss - Compute combined Knowledge Distillation loss (
KD + L2L MSE) from student and teacher outputs. - kl_
divergence_ batchmean - Compute
F.kl_div(F.log_softmax(student, -1), F.softmax(teacher, -1), reduction="batchmean")inF32for numerical stability. - l2l_
mse_ loss - Compute layer-to-layer MSE loss summed over aligned transformer hidden states and scaled by
scale.