pub fn load_tokenizer(
model_id_or_path: &str,
cache_dir: Option<&Path>,
token: Option<&str>,
local_files_only: bool,
) -> Result<Tokenizer>Expand description
Load a Hugging Face tokenizer.json from a local checkpoint directory or Hub model repo ID.
ยงErrors
Returns TrainError::Tokenizer or TrainError::Hub if tokenizer.json cannot be resolved.