littlebit-rs is a pure-Rust implementation of
SamsungLabs/LittleBit and
LittleBit-2 Joint-ITQ built on Candle and faer — covering
low-rank SVD + Joint-ITQ initialization, Knowledge Distillation QAT, upstream bit-identical
i32 safetensors packing, and direct 1-bit packed inference.
# Header-only layer & compression ratio inspection (no full weight load)
cargo run --release -p littlebit-cli -- inspect --model-id ./outputs/Llama-2-7b-LittleBit-2
# Packed 1-bit perplexity evaluation
cargo run --release -p littlebit-cli -- eval \
--model-id ./outputs/Llama-2-7b-LittleBit-2 \
--packed-inference --seqlen 2048 --ppl-task wikitext2
# Autoregressive generation
cargo run --release -p littlebit-cli -- generate \
--model-id ./outputs/Llama-2-7b-LittleBit-2 \
--packed-inference --prompt "Sub-1-bit LLM compression works by" --max-tokens 64
Benchmarks: Rust vs. Original Python / PyTorch
Head-to-head measurements collected back-to-back on an Apple M3 Pro
(arm64, 11 CPU cores, 18 GiB RAM) comparing littlebit-rs
(rustc 1.100.0, --release) against unmodified upstream
SamsungLabs/LittleBit
(commit 42d658b, Python 3.15.2, PyTorch 2.15.0 CPU, 5 threads) using
identical seeded synthetic tensors. See
BENCHMARKS.md
for full CPU default vs. Apple Accelerate tables and batched prefill (b=16) breakdowns.
Bit-Exact 1-Bit Unpacking (Issue #21 / PR #22 & Deployment Audit):
Uses the exact LSB-first right-shift decoder ((word >> bit) & 1) and
validates all 2,010 deterministic test cases across widths
[1, 7, 8, 31, 32, 33, 64] against an independent shift-free byte-table
reference decoder, plus full SmoothSign + residual checkpoint round-trips
with bitwise-equal F32 outputs.
FP16 Overflow Protection (Issue #24 & Volta Bench):
Guards intermediate binary matmuls via exact power-of-two scaling
(pow2_at_least) on GPU paths and F32 accumulation on CPU,
keeping long-summation projections (e.g., in_features = 2048) free of NaN/Inf.
Mixed-Bit Patching & Warm-Start:
Supports --eff-bit-layers (e.g. "0-7,27:2.0") persisted in
littlebit_config.json and config.json, plus
warm_start_from_binarized(latent_mag) for resuming QAT from finished packed models.
Zero Unsafe & Clean Tooling:
Workspace enforces unsafe_code = "forbid", passes cargo fmt --check,
cargo clippy --all-targets -- -D warnings, 59 tests across all
crates, and RUSTDOCFLAGS="-D warnings" cargo doc.
Citations
@inproceedings{lee2026littlebit2,
title={LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment},
author={Lee, Banseok and Kim, Youngmin},
booktitle={Proceedings of the 43rd International Conference on Machine Learning},
year={2026}
}
@inproceedings{lee2025littlebit,
title={LittleBit: Ultra Low-Bit Quantization via Latent Factorization},
author={Lee, Banseok and Kim, Dongkyu and You, Youngcheon and Kim, Youngmin},
booktitle={Advances in Neural Information Processing Systems},
year={2025}
}