Skip to main content

encode_bytes

Function encode_bytes 

Source
pub fn encode_bytes(
    input: &[u8],
    config: &EncodeConfig,
) -> Result<(Dictionary, Vec<u16>, EncodeReport), PreprocessError>
Expand description

Runs Fabrice Bellard’s bigram dictionary induction and encodes input into token IDs.

§Errors

Returns PreprocessError::InvalidMaxWords if config.max_words < 256 or > 65536, or PreprocessError::Io if config.debug_dict_path cannot be written.