Pure-Rust Port · Tufalabs/tufazip v2

tufazip-rs

A pure-Rust workspace porting Tufalabs/tufazip into a memory-safe neural lossless compressor and decompressor: Fabrice Bellard’s NNCP dictionary preprocessor, a Candle-backed Peri-RMSNorm SwiGLU Transformer initialized from quantized QPI1 loop-untying priors, online Adam adaptation with prefix replay, and an exact 32-bit resumable arithmetic coder.

  • 5 Workspace Crates
  • 0 unsafe Blocks (Forbidden)
  • 48 Workspace Tests Passing
  • 4.33 MB Verified QPI1 Prior
  • 162.6M v2 Total Parameters

Compression & Decompression Pipeline

Both compression and decompression run identical online training steps at every coding block so decoder logits match encoder logits without storing per-token side information beyond the binary .voc table and optional prior.

tufazip-rs neural lossless compression and decompression pipeline 01 · INPUT Raw Bytes Arbitrary byte stream (e.g. text) 02 · TUFAZIP-PREPROCESS NNCP Dictionary Case/space escape + .voc / .pp u16 tokens 03 · TUFAZIP-CODEC + PRIOR Transformer + Online Adam Peri-RMSNorm · SwiGLU · RoPE QPI1 loop-untying initialization Prefix replay + safetensors ckpt 04 · TUFAZIP-ARITH 32-bit Range Coder Softmax-to-CDF (10M) Exact bitstream + resume Lossless invertibility: Decompression mirrors steps 03 → 04 (bitstream → .pp tokens) and 02 (tokens → exact raw bytes)
Four-stage lossless neural coding architecture implemented with zero unsafe Rust across five focused crates.
  1. 1. NNCP Dictionary Preprocessing (tufazip-preprocess)

    Pure-Rust port of Fabrice Bellard’s preprocess.c encoding Unicode whitespace/capitalization escapes and inducing up to 65,536 bigram dictionary tokens (.voc + big-endian u16 .pp streams).

  2. 2. Quantized Prior Loading (tufazip-prior)

    Parses and verifies QPI1, QPR1, LRA1, and QLA1 weight archives at 1/2/4/8/16-bit widths, expanding 2-layer loop-untying or depth-growth priors into 12-layer targets without wasting PRNG draws.

  3. 3. Neural Context Modeling & Online Adaptation (tufazip-codec)

    Runs a Candle-backed Transformer (Peri/Post RMSNorm, RoPE or learned-relative attention, QK-Norm, SwiGLU) updated online with deterministic Adam, gradient norm clipping, prefix replay, and atomic safetensors + JSON checkpoints.

  4. 4. 32-bit Arithmetic Coding (tufazip-arith)

    Converts logits to deterministic integer cumulative frequencies (FREQ_TOTAL = 10,000,000, round-half-to-even) and writes or decodes resumable big-endian bitstreams with byte-exact checkpoint restoration.

Workspace Crates & Rustdoc API

Every workspace library crate compiles cleanly under RUSTDOCFLAGS="-D warnings" and forbids unsafe_code at the workspace level.

Workspace crates, architectural responsibilities, upstream C/Python equivalents, and API documentation links
Crate Purpose Upstream Mapping API Reference
tufazip-arith 32-bit range encoder/decoder, big-endian checkpointed bit streams, and FREQ_TOTAL CDF table builder arithmetic_coder.py, make_freq_table Docs for tufazip-arith
tufazip-preprocess Case/space reversible transform, iterative bigram induction, and streaming .voc/.pp codec preprocess.c (Bellard NNCP) Docs for tufazip-preprocess
tufazip-prior Binary QPI1/QPR1/LRA1/QLA1 reader/writer and loop-untying / depth-growth planner read_tensor_archive, load_depth_prior Docs for tufazip-prior
tufazip-codec Candle 0.11 Transformer, online Adam + prefix replay, ChaCha8 RNG, and atomic checkpoint resume nn.py, checkpoint_utils.py Docs for tufazip-codec
tufazip-cli Unified tufazip binary (compress, decompress, codec, preprocess, v2-decode, inspect-prior) tufazip, nn.py, preprocess.c Docs for tufazip-cli binary

Interactive CLI Command Builder

Configure subcommand, Candle compute backend feature flag, model preset, and build profile to generate the exact cargo run invocation.

Select an end-to-end, codec-only, preprocessing, prior inspection, or v2 verification workflow.
Select optional Cargo feature flags and device flags matched to your hardware.
tiny runs fast locally; v2 sets full 12-layer enwik9 geometry.
Enables thin LTO and single-codegen-unit release optimization configured in root Cargo.toml.
Generated Invocation
cargo run --release --bin tufazip -- compress input.txt archive.tz --vocab out.voc --preset tiny --device cpu

End-to-end compression induces a vocabulary into out.voc, tokenizes input.txt, and writes the arithmetic-coded artifact.

Build Release Binary

git clone https://github.com/bhubbard/tufazip-rs.git
cd tufazip-rs
cargo build --release --bin tufazip

Run Workspace Checks

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
make pages

Verified Upstream Assets & Determinism

Repository assets/ includes the bit-identical dictionary and quantized prior distributed by Tufalabs/tufazip.

Verified upstream binary assets, exact byte sizes, SHA-256 checksums, and inspected v2 structure
Artifact Size SHA-256 Digest Structure & Notes
assets/enwik9.voc 34,421 B a63e517783e84a778f3f08c1f2321f3e2035e64d34c8e9206f03591460e44629 NNCP dictionary mapping 256 byte bases + learned bigrams for enwik9 tokenization
assets/weights.bin 4,334,741 B e94c089f8eb985a87307dca3a7702385fcaccd14d95743811f8788f7da24448d QPI1 loop-untying prior (2 → 12 layers via schedule [0,1]×6, 26 tensors, 162,585,600 total params / 154,194,944 non-embedding)

Measured Benchmarks (Apple M3 Pro · Release)

Measured via paired back-to-back runs against baseline fdf2b21 using examples/benchmark.rs on Apple M3 Pro (11-core, 18 GiB, macOS 27.2.1, lowpowermode 1, load avg 16.89–17.41, rustc 1.100.0, release LTO) over all 21 sorted crates/*/src/*.rs files (300,119 B, SHA-256 5d65828e...36bb44bd). See full methodology and host-load notes in BENCHMARKS.md.

Optimization Gains (Paired fdf2b21 vs After)

Paired same-machine Before and After benchmark speedups across arithmetic coding, dictionary preprocessing, neural backends, and end-to-end compression
Workload Before (fdf2b21) After (Optimized) Speedup
tufazip-preprocess Encode 213.09 ms (1.19 MiB/s) 93.74 ms (3.05 MiB/s) 2.56× (& 1.53× faster than C 1.99 MiB/s)
tufazip-arith CDF Table (20k) 517.30 ms (38,662/s) 310.56 ms (64,401/s) 1.67× (single-pass exp() bit reuse)
tufazip-arith Encode / Decode (500k) 20.96 ms / 46.69 ms 19.43 ms / 40.86 ms 1.08× / 1.14× (u64 path + partition_point)
Online tiny Compress / Decompress (CPU) 0.599 s / 0.675 s 0.411 s / 0.411 s 1.46× / 1.64× (fused Adam & SIMD RMSNorm)
Online tiny Compress / Decompress (Accelerate) 0.425 s / 0.377 s 0.322 s / 0.302 s 1.32× / 1.25× (fastest backend: 3,396 tok/s)
Online medium 4L Compress (CPU / Accelerate / Metal) 5.254 s / 2.382 s / 5.129 s 3.790 s / 1.953 s / 3.646 s 1.39× / 1.22× / 1.41× (Metal now beats CPU)
Online tiny Decompress (Metal GPU) 12.683 s (80.7 tok/s) 5.486 s (186.7 tok/s) 2.31× (single-sync Tensor::stack grad norm)
v2 12L 162.6M Fwd b=4×192 (CPU / Accel / Metal) 1,855 / 1,082 / 266.5 ms 1,642 / 764.9 / 230.1 ms 1.13× / 1.41× / 1.16× (Metal 7.13× over CPU)
Track 5 End-to-End 16 KiB (tiny / medium 4L) 2.282 s / 7.541 s 1.547 s / 5.978 s 1.48× / 1.26× (exact SHA-256 verified)

Core Primitives & C Differential Parity

Arithmetic coder, NNCP preprocessor Rust vs C reference, and quantized prior loading benchmarks
Benchmark Measured Time Throughput Parity & Notes
tufazip-arith CDF Table 310.56 ms (20k tables) 64,401 tables/s 4,097-symbol softmax → integer CDF (FREQ_TOTAL=10M)
tufazip-arith Encode 19.43 ms (500k sym) 25.74 Msym/s 954,384 B output; exact 32-bit range stream
tufazip-arith Decode 40.86 ms (500k sym) 12.24 Msym/s 500,000 / 500,000 symbols lossless verified
preprocess Encode (Rust) 93.74 ms 3.05 MiB/s 1.53× faster than C; cycle-safe against upstream loop
preprocess.c Encode (Clang -O2) 143.75 ms 1.99 MiB/s 10,290 B .voc / 303,962 B .pp; C⇋Rust lossless
preprocess Decode (Rust) 1.89 ms 151.52 MiB/s 11.84× faster than C CLI (22.36 ms); bidirectional C⇋Rust verified
tufazip-prior read_archive 126.00 ms 1,032 MiB/s f32 4.33 MB weights.bin (26 tensors → 130 MiB f32)
Transformer::from_prior (v2) 2,883.65 ms 56.4M params/s Expands 2 → 12 layers (162,585,600 params)

Neural Backend Scaling (CPU vs Accelerate vs Metal)

Throughput comparison across pure-Rust CPU, Apple Accelerate BLAS, and Apple Metal GPU backends
Neural Workload Pure-Rust CPU Apple Accelerate BLAS Apple Metal GPU
Online tiny Compress (1,024 tok) 0.411 s (2,490.9 tok/s) 0.322 s (3,178.0 tok/s · 1.28×) 5.366 s (190.8 tok/s · per-step sync)
Online tiny Decompress (1,024 tok) 0.411 s (2,488.9 tok/s) 0.302 s (3,395.6 tok/s · 1.36×) 5.486 s (186.7 tok/s · per-step sync)
Online medium 4L/256d Compress 3.790 s (270.2 tok/s) 1.953 s (524.3 tok/s · 1.94×) 3.646 s (280.8 tok/s · per-step sync)
v2 12L 162.6M Fwd (F32, b=1×192) 476.41 ms (403.0 tok/s) 239.02 ms (803.3 tok/s · 1.99×) 51.31 ms (3,742.0 tok/s · 9.29×)
v2 12L 162.6M Fwd (BF16, b=1×192) — — 62.52 ms (3,071.2 tok/s · 7.62×)
v2 12L 162.6M Fwd (F32, b=4×192) 1,641.78 ms (467.8 tok/s) 764.94 ms (1,004.0 tok/s · 2.15×) 230.12 ms (3,337.3 tok/s · 7.13×)

Lossless Compression Ratio (16,384 B Rust Source Prefix)

Lossless compression ratio in bits per byte compared against standard compressors on 16,384 bytes
Compressor Payload Bytes (bpb) Total Archive (bpb) Time & Verification
zstd -19 3,482 B (1.700 bpb) 3,482 B (1.700 bpb) 0.018 s · SHA-256 verified
bzip2 -9 3,549 B (1.733 bpb) 3,549 B (1.733 bpb) 0.006 s · SHA-256 verified
xz -9 3,568 B (1.742 bpb) 3,568 B (1.742 bpb) 0.012 s · SHA-256 verified
gzip -9 3,574 B (1.745 bpb) 3,574 B (1.745 bpb) 0.005 s · SHA-256 verified
tufazip-rs (medium 4L, cold-start) 3,832 B (1.871 bpb) 7,249 B (+3,417 B .voc, 3.540 bpb) 5.978 s · SHA-256 verified
tufazip-rs (tiny, cold-start) 3,900 B (1.904 bpb) 7,317 B (+3,417 B .voc, 3.573 bpb) 1.547 s · SHA-256 verified

Differential & Regression Test Suite

In addition to unit roundtrips, tufazip-rs includes ports of Project Nayuki’s reference arithmetic coding suite and a live C differential harness compiled against unmodified upstream preprocess.c.

Verification coverage across all five workspace crates
Crate Harness / Suite What It Verifies
tufazip-arith Nayuki Reference Tests Ports Project Nayuki’s ArithmeticCodingTest, ArithmeticCompressTest, and AdaptiveArithmeticCompressTest (empty, single-symbol, underflow, 1,000 random distributions) plus EOF and 16/24/32/48-bit state sweeps
tufazip-preprocess Live C Differential Harness Compiles unmodified upstream preprocess.c via cc and checks byte-for-byte identical .voc/.pp outputs, bidirectional C⇋Rust decoding, -s sorted dictionaries, and assets/enwik9.voc parity
tufazip-prior Binary Validation Matrix Roundtrips 1/2/4/8/16-bit QPI1/QPR1/LRA1/QLA1 archives, validates depth-growth and loop-untying schedules, and rejects malformed headers, shapes, and trailing bytes
tufazip-codec Model & Checkpoint Edge Cases Verifies Transformer::from_prior real weights.bin forward pass, PiecewiseLrSchedule, LearnedRelative + Post norms, pause/resume flags, checkpoint corruption checks, and zero-token streams
tufazip-cli CLI Integration Roundtrips Exercises end-to-end compress/decompress, preprocess, inspect-prior --json, error paths, and --stop-after-blocks 1 exit code 75 checkpoint requeue & resume