pub fn evaluate_perplexity(
model: &CausalLm,
token_ids: &[u32],
seqlen: usize,
dataset_name: &str,
device: &Device,
) -> Result<PplResult>Expand description
Evaluate autoregressive perplexity on a flat token stream matching eval.py.
ยงErrors
Returns an error if seqlen < 2, token_ids.len() < seqlen, or model forward passes fail.