Skip to main content

load_tokenizer

Function load_tokenizer 

Source
pub fn load_tokenizer(
    model_id_or_path: &str,
    cache_dir: Option<&Path>,
    token: Option<&str>,
    local_files_only: bool,
) -> Result<Tokenizer>
Expand description

Load a Hugging Face tokenizer.json from a local checkpoint directory or Hub model repo ID.

ยงErrors

Returns TrainError::Tokenizer or TrainError::Hub if tokenizer.json cannot be resolved.