pub fn read_tokens_be16(
path: impl AsRef<Path>,
) -> Result<Vec<u16>, PreprocessError>Expand description
Reads big-endian u16 token IDs from path.
ยงErrors
Returns PreprocessError::OddTokenStreamLength or PreprocessError::Io.
pub fn read_tokens_be16(
path: impl AsRef<Path>,
) -> Result<Vec<u16>, PreprocessError>Reads big-endian u16 token IDs from path.
Returns PreprocessError::OddTokenStreamLength or PreprocessError::Io.