Skip to main content

Crate tufazip_preprocess

Crate tufazip_preprocess 

Source
Expand description

Pure-Rust port of Fabrice Bellard’s NNCP dictionary preprocessor (preprocess.c).

Performs reversible case/space encoding, iterative bigram dictionary induction, big-endian u16 token serialization, and streaming decoding back to exact source bytes.

Structs§

CaseSpaceDecoder
Stateful decoder inverting case_space_encode across streamed dictionary entries.
Dictionary
NNCP token dictionary mapping 16-bit symbol IDs (0..vocab_size) to raw word byte slices.
EncodeConfig
Parameters controlling dictionary induction during preprocessing.
EncodeReport
Summary statistics from a preprocessing run.

Enums§

PreprocessError
Errors returned by dictionary preprocessing, loading, encoding, or decoding.
SortOrder
Ordering applied to the final dictionary entries.

Constants§

CH_FIRST_UPPER
Control byte capitalizing the first ASCII letter of the next word.
CH_NO_SPACE
Control byte suppressing the implicit leading space before the next word.
CH_TO_UPPER
Control byte marking the next word as all uppercase.

Functions§

case_space_decode
Decodes a byte buffer produced by case_space_encode back to original bytes.
case_space_encode
Applies Fabrice Bellard’s reversible case-and-space encoding to raw input bytes.
decode_file
Streams a big-endian u16 token file back to its original raw file using dict_path.
decode_tokens
Decodes a sequence of token IDs back into the exact original source bytes.
encode_bytes
Runs Fabrice Bellard’s bigram dictionary induction and encodes input into token IDs.
encode_file
Preprocesses in_path, writing the learned dictionary to dict_path and big-endian tokens to out_path.
encode_with_dictionary
Encodes input greedily (longest match first) using an existing Dictionary.
read_tokens_be16
Reads big-endian u16 token IDs from path.
write_tokens_be16
Writes tokens as big-endian u16 pairs to path.