Build Release Binary
git clone https://github.com/bhubbard/tufazip-rs.git
cd tufazip-rs
cargo build --release --bin tufazip
Pure-Rust Port · Tufalabs/tufazip v2
A pure-Rust workspace porting Tufalabs/tufazip into a memory-safe neural lossless compressor and decompressor: Fabrice Bellard’s NNCP dictionary preprocessor, a Candle-backed Peri-RMSNorm SwiGLU Transformer initialized from quantized QPI1 loop-untying priors, online Adam adaptation with prefix replay, and an exact 32-bit resumable arithmetic coder.
unsafe Blocks (Forbidden)
Both compression and decompression run identical online training steps at every coding block so decoder logits match encoder logits without storing per-token side information beyond the binary .voc table and optional prior.
unsafe Rust across five focused crates.
tufazip-preprocess)
Pure-Rust port of Fabrice Bellard’s preprocess.c encoding Unicode whitespace/capitalization escapes and inducing up to 65,536 bigram dictionary tokens (.voc + big-endian u16 .pp streams).
tufazip-prior)
Parses and verifies QPI1, QPR1, LRA1, and QLA1 weight archives at 1/2/4/8/16-bit widths, expanding 2-layer loop-untying or depth-growth priors into 12-layer targets without wasting PRNG draws.
tufazip-codec)
Runs a Candle-backed Transformer (Peri/Post RMSNorm, RoPE or learned-relative attention, QK-Norm, SwiGLU) updated online with deterministic Adam, gradient norm clipping, prefix replay, and atomic safetensors + JSON checkpoints.
tufazip-arith)
Converts logits to deterministic integer cumulative frequencies (FREQ_TOTAL = 10,000,000, round-half-to-even) and writes or decodes resumable big-endian bitstreams with byte-exact checkpoint restoration.
Every workspace library crate compiles cleanly under RUSTDOCFLAGS="-D warnings" and forbids unsafe_code at the workspace level.
| Crate | Purpose | Upstream Mapping | API Reference |
|---|---|---|---|
tufazip-arith |
32-bit range encoder/decoder, big-endian checkpointed bit streams, and FREQ_TOTAL CDF table builder |
arithmetic_coder.py, make_freq_table |
Docs for tufazip-arith |
tufazip-preprocess |
Case/space reversible transform, iterative bigram induction, and streaming .voc/.pp codec |
preprocess.c (Bellard NNCP) |
Docs for tufazip-preprocess |
tufazip-prior |
Binary QPI1/QPR1/LRA1/QLA1 reader/writer and loop-untying / depth-growth planner |
read_tensor_archive, load_depth_prior |
Docs for tufazip-prior |
tufazip-codec |
Candle 0.11 Transformer, online Adam + prefix replay, ChaCha8 RNG, and atomic checkpoint resume | nn.py, checkpoint_utils.py |
Docs for tufazip-codec |
tufazip-cli |
Unified tufazip binary (compress, decompress, codec, preprocess, v2-decode, inspect-prior) |
tufazip, nn.py, preprocess.c |
Docs for tufazip-cli binary |
Configure subcommand, Candle compute backend feature flag, model preset, and build profile to generate the exact cargo run invocation.
cargo run --release --bin tufazip -- compress input.txt archive.tz --vocab out.voc --preset tiny --device cpu
End-to-end compression induces a vocabulary into out.voc, tokenizes input.txt, and writes the arithmetic-coded artifact.
git clone https://github.com/bhubbard/tufazip-rs.git
cd tufazip-rs
cargo build --release --bin tufazip
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
make pages
Repository assets/ includes the bit-identical dictionary and quantized prior distributed by Tufalabs/tufazip.
| Artifact | Size | SHA-256 Digest | Structure & Notes |
|---|---|---|---|
assets/enwik9.voc |
34,421 B | a63e517783e84a778f3f08c1f2321f3e2035e64d34c8e9206f03591460e44629 |
NNCP dictionary mapping 256 byte bases + learned bigrams for enwik9 tokenization |
assets/weights.bin |
4,334,741 B | e94c089f8eb985a87307dca3a7702385fcaccd14d95743811f8788f7da24448d |
QPI1 loop-untying prior (2 → 12 layers via schedule [0,1]×6, 26 tensors, 162,585,600 total params / 154,194,944 non-embedding) |
Measured via paired back-to-back runs against baseline fdf2b21 using examples/benchmark.rs on Apple M3 Pro (11-core, 18 GiB, macOS 27.2.1, lowpowermode 1, load avg 16.89–17.41, rustc 1.100.0, release LTO) over all 21 sorted crates/*/src/*.rs files (300,119 B, SHA-256 5d65828e...36bb44bd). See full methodology and host-load notes in BENCHMARKS.md.
fdf2b21 vs After)| Workload | Before (fdf2b21) |
After (Optimized) | Speedup |
|---|---|---|---|
tufazip-preprocess Encode |
213.09 ms (1.19 MiB/s) | 93.74 ms (3.05 MiB/s) | 2.56× (& 1.53× faster than C 1.99 MiB/s) |
tufazip-arith CDF Table (20k) |
517.30 ms (38,662/s) | 310.56 ms (64,401/s) | 1.67× (single-pass exp() bit reuse) |
tufazip-arith Encode / Decode (500k) |
20.96 ms / 46.69 ms | 19.43 ms / 40.86 ms | 1.08× / 1.14× (u64 path + partition_point) |
Online tiny Compress / Decompress (CPU) |
0.599 s / 0.675 s | 0.411 s / 0.411 s | 1.46× / 1.64× (fused Adam & SIMD RMSNorm) |
Online tiny Compress / Decompress (Accelerate) |
0.425 s / 0.377 s | 0.322 s / 0.302 s | 1.32× / 1.25× (fastest backend: 3,396 tok/s) |
Online medium 4L Compress (CPU / Accelerate / Metal) |
5.254 s / 2.382 s / 5.129 s | 3.790 s / 1.953 s / 3.646 s | 1.39× / 1.22× / 1.41× (Metal now beats CPU) |
Online tiny Decompress (Metal GPU) |
12.683 s (80.7 tok/s) | 5.486 s (186.7 tok/s) | 2.31× (single-sync Tensor::stack grad norm) |
v2 12L 162.6M Fwd b=4×192 (CPU / Accel / Metal) |
1,855 / 1,082 / 266.5 ms | 1,642 / 764.9 / 230.1 ms | 1.13× / 1.41× / 1.16× (Metal 7.13× over CPU) |
Track 5 End-to-End 16 KiB (tiny / medium 4L) |
2.282 s / 7.541 s | 1.547 s / 5.978 s | 1.48× / 1.26× (exact SHA-256 verified) |
| Benchmark | Measured Time | Throughput | Parity & Notes |
|---|---|---|---|
tufazip-arith CDF Table |
310.56 ms (20k tables) | 64,401 tables/s | 4,097-symbol softmax → integer CDF (FREQ_TOTAL=10M) |
tufazip-arith Encode |
19.43 ms (500k sym) | 25.74 Msym/s | 954,384 B output; exact 32-bit range stream |
tufazip-arith Decode |
40.86 ms (500k sym) | 12.24 Msym/s | 500,000 / 500,000 symbols lossless verified |
preprocess Encode (Rust) |
93.74 ms | 3.05 MiB/s | 1.53× faster than C; cycle-safe against upstream loop |
preprocess.c Encode (Clang -O2) |
143.75 ms | 1.99 MiB/s | 10,290 B .voc / 303,962 B .pp; C⇋Rust lossless |
preprocess Decode (Rust) |
1.89 ms | 151.52 MiB/s | 11.84× faster than C CLI (22.36 ms); bidirectional C⇋Rust verified |
tufazip-prior read_archive |
126.00 ms | 1,032 MiB/s f32 |
4.33 MB weights.bin (26 tensors → 130 MiB f32) |
Transformer::from_prior (v2) |
2,883.65 ms | 56.4M params/s | Expands 2 → 12 layers (162,585,600 params) |
| Neural Workload | Pure-Rust CPU | Apple Accelerate BLAS | Apple Metal GPU |
|---|---|---|---|
Online tiny Compress (1,024 tok) |
0.411 s (2,490.9 tok/s) | 0.322 s (3,178.0 tok/s · 1.28×) | 5.366 s (190.8 tok/s · per-step sync) |
Online tiny Decompress (1,024 tok) |
0.411 s (2,488.9 tok/s) | 0.302 s (3,395.6 tok/s · 1.36×) | 5.486 s (186.7 tok/s · per-step sync) |
Online medium 4L/256d Compress |
3.790 s (270.2 tok/s) | 1.953 s (524.3 tok/s · 1.94×) | 3.646 s (280.8 tok/s · per-step sync) |
v2 12L 162.6M Fwd (F32, b=1×192) |
476.41 ms (403.0 tok/s) | 239.02 ms (803.3 tok/s · 1.99×) | 51.31 ms (3,742.0 tok/s · 9.29×) |
v2 12L 162.6M Fwd (BF16, b=1×192) |
— | — | 62.52 ms (3,071.2 tok/s · 7.62×) |
v2 12L 162.6M Fwd (F32, b=4×192) |
1,641.78 ms (467.8 tok/s) | 764.94 ms (1,004.0 tok/s · 2.15×) | 230.12 ms (3,337.3 tok/s · 7.13×) |
16,384 B Rust Source Prefix)| Compressor | Payload Bytes (bpb) |
Total Archive (bpb) |
Time & Verification |
|---|---|---|---|
zstd -19 |
3,482 B (1.700 bpb) | 3,482 B (1.700 bpb) | 0.018 s · SHA-256 verified |
bzip2 -9 |
3,549 B (1.733 bpb) | 3,549 B (1.733 bpb) | 0.006 s · SHA-256 verified |
xz -9 |
3,568 B (1.742 bpb) | 3,568 B (1.742 bpb) | 0.012 s · SHA-256 verified |
gzip -9 |
3,574 B (1.745 bpb) | 3,574 B (1.745 bpb) | 0.005 s · SHA-256 verified |
tufazip-rs (medium 4L, cold-start) |
3,832 B (1.871 bpb) | 7,249 B (+3,417 B .voc, 3.540 bpb) |
5.978 s · SHA-256 verified |
tufazip-rs (tiny, cold-start) |
3,900 B (1.904 bpb) | 7,317 B (+3,417 B .voc, 3.573 bpb) |
1.547 s · SHA-256 verified |
In addition to unit roundtrips, tufazip-rs includes ports of Project Nayuki’s reference arithmetic coding suite and a live C differential harness compiled against unmodified upstream preprocess.c.
| Crate | Harness / Suite | What It Verifies |
|---|---|---|
tufazip-arith |
Nayuki Reference Tests | Ports Project Nayuki’s ArithmeticCodingTest, ArithmeticCompressTest, and AdaptiveArithmeticCompressTest (empty, single-symbol, underflow, 1,000 random distributions) plus EOF and 16/24/32/48-bit state sweeps |
tufazip-preprocess |
Live C Differential Harness | Compiles unmodified upstream preprocess.c via cc and checks byte-for-byte identical .voc/.pp outputs, bidirectional C⇋Rust decoding, -s sorted dictionaries, and assets/enwik9.voc parity |
tufazip-prior |
Binary Validation Matrix | Roundtrips 1/2/4/8/16-bit QPI1/QPR1/LRA1/QLA1 archives, validates depth-growth and loop-untying schedules, and rejects malformed headers, shapes, and trailing bytes |
tufazip-codec |
Model & Checkpoint Edge Cases | Verifies Transformer::from_prior real weights.bin forward pass, PiecewiseLrSchedule, LearnedRelative + Post norms, pause/resume flags, checkpoint corruption checks, and zero-token streams |
tufazip-cli |
CLI Integration Roundtrips | Exercises end-to-end compress/decompress, preprocess, inspect-prior --json, error paths, and --stop-after-blocks 1 exit code 75 checkpoint requeue & resume |