Skip to main content

load_eval_tokens

Function load_eval_tokens 

Source
pub fn load_eval_tokens(
    kind: &DatasetKind,
    tokenizer: &Tokenizer,
    seqlen: usize,
    cache_dir: Option<&Path>,
    token: Option<&str>,
) -> Result<Vec<u32>>
Expand description

Build a single contiguous evaluation token vector for perplexity measurement (get_eval_loaders).

ยงErrors

Returns an error if loading or tokenizing fails.