Expand description
Pure-Rust port of Fabrice Bellard’s NNCP dictionary preprocessor (preprocess.c).
Performs reversible case/space encoding, iterative bigram dictionary induction,
big-endian u16 token serialization, and streaming decoding back to exact source bytes.
Structs§
- Case
Space Decoder - Stateful decoder inverting
case_space_encodeacross streamed dictionary entries. - Dictionary
- NNCP token dictionary mapping 16-bit symbol IDs (
0..vocab_size) to raw word byte slices. - Encode
Config - Parameters controlling dictionary induction during preprocessing.
- Encode
Report - Summary statistics from a preprocessing run.
Enums§
- Preprocess
Error - Errors returned by dictionary preprocessing, loading, encoding, or decoding.
- Sort
Order - Ordering applied to the final dictionary entries.
Constants§
- CH_
FIRST_ UPPER - Control byte capitalizing the first ASCII letter of the next word.
- CH_
NO_ SPACE - Control byte suppressing the implicit leading space before the next word.
- CH_
TO_ UPPER - Control byte marking the next word as all uppercase.
Functions§
- case_
space_ decode - Decodes a byte buffer produced by
case_space_encodeback to original bytes. - case_
space_ encode - Applies Fabrice Bellard’s reversible case-and-space encoding to raw input bytes.
- decode_
file - Streams a big-endian
u16token file back to its original raw file usingdict_path. - decode_
tokens - Decodes a sequence of token IDs back into the exact original source bytes.
- encode_
bytes - Runs Fabrice Bellard’s bigram dictionary induction and encodes
inputinto token IDs. - encode_
file - Preprocesses
in_path, writing the learned dictionary todict_pathand big-endian tokens toout_path. - encode_
with_ dictionary - Encodes
inputgreedily (longest match first) using an existingDictionary. - read_
tokens_ be16 - Reads big-endian
u16token IDs frompath. - write_
tokens_ be16 - Writes
tokensas big-endianu16pairs topath.