pub fn encode_bytes(
input: &[u8],
config: &EncodeConfig,
) -> Result<(Dictionary, Vec<u16>, EncodeReport), PreprocessError>Expand description
Runs Fabrice Bellard’s bigram dictionary induction and encodes input into token IDs.
§Errors
Returns PreprocessError::InvalidMaxWords if config.max_words < 256 or > 65536,
or PreprocessError::Io if config.debug_dict_path cannot be written.