pub fn decode_tokens(
tokens: &[u16],
dict: &Dictionary,
) -> Result<Vec<u8>, PreprocessError>Expand description
Decodes a sequence of token IDs back into the exact original source bytes.
ยงErrors
Returns PreprocessError::InvalidSymbol if any token ID is >= dict.len().