NeurIPS 2025 & ICML 2026 · Pure Rust Port

Sub-1-Bit LLM Compression Without Python

littlebit-rs is a pure-Rust implementation of SamsungLabs/LittleBit and LittleBit-2 Joint-ITQ built on Candle and faer — covering low-rank SVD + Joint-ITQ initialization, Knowledge Distillation QAT, upstream bit-identical i32 safetensors packing, and direct 1-bit packed inference.

Workspace Crates

Organized into four focused crates with zero Python runtime or build dependencies. Select any library crate to open its generated rustdoc reference.

Workspace Crates
Crate Role API Docs
littlebit-core STEBinary & SmoothSign autograd ops, LSB-first 32-bit binary_packer / binary_unpacker, PackedBinaryMatrix matmul, faer thin & randomized SVD, Joint-ITQ rotation, LittleBitLinear / PackedLittleBitLinear, safeload/inspection, and per-block eff_bit_layers parsing littlebit_core docs
littlebit-models Differentiable decoders for Llama, Qwen2/2.5/QwQ, Qwen3, Phi-3/4 (PhiQKVSplitAttention), Gemma 2/3, and OPT; dynamic apply_littlebit_patch; KV-cached generation; Hugging Face Hub upload & reload littlebit_models docs
littlebit-train Knowledge Distillation QAT trainer (KdTrainer) with KL-divergence logit loss and scaled layer-to-layer hidden-state MSE, cosine warmup scheduler, SHA-256[:7] dataset caching, perplexity & zero-shot evaluation, and configs/zero*.json parsing littlebit_train docs
littlebit-cli Unified littlebit binary exposing quantize, train, eval, generate, inspect, pack, unpack, and push subcommands CLI Quickstart

Supported Model Families

Handles both legacy unpacked checkpoints and modern bit-packed checkpoints with automatic architecture detection from Hugging Face config.json.

Supported Causal LM Architectures
Family model_type Notable Support
Llama / Llama 2 / Llama 3 "llama" RoPE, GQA/MQA, SwiGLU MLP, kv_factor rank boost on K/V projections
Qwen2 / Qwen2.5 / QwQ "qwen2" Attention QKV bias support, RMSNorm, tied or untied LM head exclusion
Qwen3 "qwen3" Per-head q_norm / k_norm, FP16 overflow-safe scaling, mixed-bit --eff-bit-layers
Phi-3 / Phi-4 "phi3" PhiQKVSplitAttention + split_qkv() for exact kv_factor scaling on fused QKV
Gemma 2 / Gemma 3 "gemma2", "gemma3", "gemma3_text" Pre/post feedforward norms, softcapping & attention logit scaling parity
Meta OPT "opt" Learned positional embeddings, project_in/project_out, ReLU FFN

CLI Quickstart

Accepts both Rust-native kebab-case (--use-itq True) and upstream underscored flags (--use_itq True).

Build (CPU, Accelerate, Metal, or CUDA)

# Default CPU build (faer multi-threaded BLAS/SVD + rayon)
cargo build --release

# Apple Silicon Accelerate / Metal GPU / NVIDIA CUDA
cargo build --release --features accelerate
cargo build --release --features metal
cargo build --release --features cuda

Post-Training Quantization with LittleBit-2 Joint-ITQ & Mixed Bits

cargo run --release -p littlebit-cli -- quantize \
  --model-id meta-llama/Llama-2-7b-hf \
  --save-dir ./outputs/Llama-2-7b-LittleBit-2 \
  --quant-func SmoothSign \
  --eff-bit 1.0 \
  --eff-bit-layers "0-3:2.0" \
  --residual True \
  --kv-factor 1.0 \
  --min-split-dim 8 \
  --use-itq True \
  --itq-n-iter 50

Knowledge Distillation QAT Training

cargo run --release -p littlebit-cli -- train \
  --model-id meta-llama/Llama-2-7b-hf \
  --dataset c4_wiki \
  --save-dir ./outputs/Llama-2-7b-LittleBit-2 \
  --num-train-epochs 5.0 \
  --per-device-train-batch-size 4 \
  --lr 4e-5 \
  --warmup-ratio 0.02 \
  --quant-func SmoothSign \
  --residual True \
  --eff-bit 1.0 \
  --l2l-loss-scale 10.0 \
  --use-itq True

Inspect, Evaluate (1-Bit Packed), & Generate

# Header-only layer & compression ratio inspection (no full weight load)
cargo run --release -p littlebit-cli -- inspect --model-id ./outputs/Llama-2-7b-LittleBit-2

# Packed 1-bit perplexity evaluation
cargo run --release -p littlebit-cli -- eval \
  --model-id ./outputs/Llama-2-7b-LittleBit-2 \
  --packed-inference --seqlen 2048 --ppl-task wikitext2

# Autoregressive generation
cargo run --release -p littlebit-cli -- generate \
  --model-id ./outputs/Llama-2-7b-LittleBit-2 \
  --packed-inference --prompt "Sub-1-bit LLM compression works by" --max-tokens 64

Benchmarks: Rust vs. Original Python / PyTorch

Head-to-head measurements collected back-to-back on an Apple M3 Pro (arm64, 11 CPU cores, 18 GiB RAM) comparing littlebit-rs (rustc 1.100.0, --release) against unmodified upstream SamsungLabs/LittleBit (commit 42d658b, Python 3.15.2, PyTorch 2.15.0 CPU, 5 threads) using identical seeded synthetic tensors. See BENCHMARKS.md for full CPU default vs. Apple Accelerate tables and batched prefill (b=16) breakdowns.

Deployment Footprint, Cold Startup & Runtime Capabilities

Standalone Rust Binary vs. Upstream Python / PyTorch Environment
Metric / Capability littlebit-rs (Rust) Original Python (42d658b) Comparison
Install / Binary Size 18.59 MiB (Accel) / 18.67 MiB (default) 766.79 MiB (.venv w/ torch only) 41.2× smaller binary
Cold Startup (5-run median) 9.80 ms (default) / 10.32 ms (Accel) 3,854.33 ms (import torch) 373×–393× faster
1-Bit Packed Runtime Matmul Built-in (PackedLittleBitLinear) None (unpacks to dense float RAM) 1-bit resident inference
FP16 Overflow & Mixed-Bit Config Issue #24 fix + --eff-bit-layers Open issue / requires patch Integrated natively
Test Suite 59 Rust tests (0 unsafe) 0 unit tests in repo Full regression parity

Packer & Unpacker Speed: Rust vs. PyTorch

LSB-First i8 → i32 Binary Packing & Unpacking (Apple M3 Pro)
Shape Packed Size Rust Unpack (Accel) PyTorch Unpack Unpack Speedup Rust Pack (Accel) PyTorch Pack
1024 × 4096 0.50 MiB 3.01 ms (1,394.3 MW/s) 14.47 ms (289.8 MW/s) 4.81× faster 18.23 ms 36.15 ms
4096 × 4096 2.00 MiB 11.59 ms (1,447.1 MW/s) 203.77 ms (82.3 MW/s) 17.58× faster 72.17 ms 58.01 ms
4096 × 11008 5.38 MiB 35.61 ms (1,266.2 MW/s) 147.12 ms (306.5 MW/s) 4.13× faster 196.81 ms 43.68 ms

Resident Weight RAM & Projection Decode: Rust 1-Bit vs. PyTorch LittleBitLinear

1-Bit Packed Resident Inference (Rust) vs. Unpacked Float Resident Inference (PyTorch)
Projection Rust 1-Bit RAM PyTorch RAM (F32 / BF16) RAM Savings b=1 Rust Packed (Accel) b=1 PyTorch LB b=16 Packed vs PyTorch
4096×4096 @ 0.1 (single, s=184) 222.2 KiB 5.78 MiB / 2.89 MiB 26.65× / 13.33× 0.35 ms 0.06 ms 7.02 ms vs 1.69 ms
4096×4096 @ 0.5 (residual, s=488) 1.05 MiB 30.57 MiB / 15.28 MiB 29.11× / 14.55× 1.42 ms 0.88 ms 25.75 ms vs 5.68 ms
4096×4096 @ 1.0 (residual, s=1000) 2.06 MiB 62.58 MiB / 31.29 MiB 30.35× / 15.17× 2.83 ms 2.06 ms 45.70 ms vs 9.47 ms
4096×11008 @ 0.5 (residual, s=728) 2.77 MiB 84.02 MiB / 42.01 MiB 30.28× / 15.14× 4.74 ms 3.36 ms 56.82 ms vs 12.51 ms

SVD & Joint-ITQ Initialization: Rust (faer) vs. PyTorch

Zero-Shot Factorization Speed & Relative Frobenius Error (eff_bit = 0.5, Residual)
Shape & Method split_dim Rust Init (Accel) PyTorch Init Rust Rel. Error PyTorch Rel. Error
1024×1024 SVD-only 104 107.70 ms 86.51 ms 0.88363 0.88563
1024×1024 Joint-ITQ (20 iters) 104 226.42 ms 126.62 ms 0.86192 0.86314
2048×2048 SVD-only 232 557.67 ms 293.77 ms 0.87422 0.87471
2048×2048 Joint-ITQ (20 iters) 232 1,093.93 ms 700.80 ms 0.84829 0.84890

Reproduce Rust Benchmark Suite

# Default CPU (faer + rayon)
cargo run --release -p littlebit-cli --example bench

# macOS Apple Accelerate BLAS
cargo run --release -p littlebit-cli --features accelerate --example bench

Rust Projection Modes: Dense vs. Unpacked vs. 1-Bit Packed (batch = 1)

littlebit-rs 1-Token Decode & Footprint (Apple M3 Pro)
Projection eff_bit Packed Size vs FP16 Dense F32 (Accel) Unpacked LB (Accel) Packed 1-Bit (Accel)
Llama-7B Attn 4096×4096 (single) 0.098 222.2 KiB 147.50× 1.66 ms 4.69 ms 0.35 ms
Llama-7B Attn 4096×4096 (residual) 0.493 1.05 MiB 30.47× 1.09 ms 25.67 ms 1.42 ms
Llama-7B Attn 4096×4096 (residual) 0.994 2.06 MiB 15.52× 1.36 ms 54.69 ms 2.83 ms
Llama-7B MLP 4096×11008 (residual) 0.499 2.77 MiB 31.00× 3.64 ms 73.65 ms 4.74 ms

4-Layer Causal LM Storage & KV-Cached Decode Throughput

32-Token Greedy Decode (16-Token Prompt, 0.469 avg bpw)
Model Layers Size vs FP16 Dense (Accel) Unpacked (Accel) Packed 1-Bit (Accel)
Llama-4L (h=512, i=1376, v=4096) 0.76 MiB 29.19× 23.3 tok/s 48.3 tok/s 51.6 tok/s (58.1 CPU default)
Qwen3-4L (h=512, i=1376, v=4096) 0.76 MiB 29.19× 21.8 tok/s 50.2 tok/s 58.0 tok/s

Upstream Parity & Community Audits

Citations

@inproceedings{lee2026littlebit2,
  title={LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment},
  author={Lee, Banseok and Kim, Youngmin},
  booktitle={Proceedings of the 43rd International Conference on Machine Learning},
  year={2026}
}

@inproceedings{lee2025littlebit,
  title={LittleBit: Ultra Low-Bit Quantization via Latent Factorization},
  author={Lee, Banseok and Kim, Dongkyu and You, Youngcheon and Kim, Youngmin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2025}
}