Expand description
Streaming layer-by-layer quantization for large checkpoints under constrained RAM.
Reads one transformer block (or one MoE expert slice) at a time directly from .safetensors
shards via [SafetensorsReader], runs SVD + Joint-ITQ factorization and i32 binary packing,
streams the packed outputs to sharded safetensors via [ShardedSafetensorsWriter], and
immediately drops dense weights between blocks.
Structs§
- Streaming
Quantize Options - Configuration options for
stream_quantize_checkpoint. - Streaming
Quantize Report - Summary report returned by
stream_quantize_checkpoint.
Functions§
- stream_
quantize_ checkpoint - Quantize a dense local checkpoint in streaming layer-by-layer fashion without loading the full model weights into RAM.