Skip to main content

Module streaming

Module streaming 

Source
Expand description

Streaming layer-by-layer quantization for large checkpoints under constrained RAM.

Reads one transformer block (or one MoE expert slice) at a time directly from .safetensors shards via [SafetensorsReader], runs SVD + Joint-ITQ factorization and i32 binary packing, streams the packed outputs to sharded safetensors via [ShardedSafetensorsWriter], and immediately drops dense weights between blocks.

Structs§

StreamingQuantizeOptions
Configuration options for stream_quantize_checkpoint.
StreamingQuantizeReport
Summary report returned by stream_quantize_checkpoint.

Functions§

stream_quantize_checkpoint
Quantize a dense local checkpoint in streaming layer-by-layer fashion without loading the full model weights into RAM.