Skip to main content

decode_tokens

Function decode_tokens 

Source
pub fn decode_tokens(
    tokens: &[u16],
    dict: &Dictionary,
) -> Result<Vec<u8>, PreprocessError>
Expand description

Decodes a sequence of token IDs back into the exact original source bytes.

ยงErrors

Returns PreprocessError::InvalidSymbol if any token ID is >= dict.len().