pub fn load_eval_tokens(
kind: &DatasetKind,
tokenizer: &Tokenizer,
seqlen: usize,
cache_dir: Option<&Path>,
token: Option<&str>,
) -> Result<Vec<u32>>Expand description
Build a single contiguous evaluation token vector for perplexity measurement (get_eval_loaders).
ยงErrors
Returns an error if loading or tokenizing fails.