Skip to main content

Crate tufazip_codec

Crate tufazip_codec 

Source
Expand description

Neural Transformer compression and decompression codec for tufazip backed by Candle.

Ported from nn.py and checkpoint_utils.py: includes the Peri-RMSNorm / Post-RMSNorm SwiGLU Transformer with RoPE or learned-relative attention, online Adam updates, periodic prefix retraining, QPI1 prior loading, and atomic checkpoint resume.

Structs§

Adam
Pure-Rust PyTorch-compatible Adam optimizer (without decoupled weight decay) with state export/import.
CheckpointMetadata
Serializable metadata written alongside latest.safetensors for resumable execution.
CodecConfig
Complete configuration for running the neural compressor or decompressor.
DeterministicRng
Deterministic, checkpointable PRNG for weight initialization and replay dropout.
LoadedCheckpoint
Loaded checkpoint metadata and tensor map.
ModelConfig
Transformer architectural configuration.
OptimConfig
Online Adam optimizer hyperparameters.
PiecewiseLrSchedule
Linearly interpolated piecewise learning-rate schedule (lr0[,step1,lr1,...]).
PriorLoadInfo
Summary of loading a QPI1 prior archive into a Transformer.
RetrainConfig
Periodic prefix retraining (replay) hyperparameters.
RngState
Exact serializable state of DeterministicRng.
RunSummary
Metrics reported upon completion of a neural coding or estimation run.
Transformer
Neural Transformer language model matching nn.py.

Enums§

AutocastPolicy
Controls whether ComputeDType::Bf16 activates on CPU or only GPU accelerators.
CodecError
Errors returned by the neural codec, model initialization, optimizer, or checkpoints.
CodecMode
Operating mode for crate::run_codec.
ComputeDType
Model activation precision during forward/backward passes.
DeviceChoice
Hardware execution backend preference.
ForwardMode
Forward pass execution mode.
NormStyle
Residual block normalization placement.
OnlineLoss
Online supervision target mode.
OutputSlice
Selects which sequence positions are projected through the vocabulary head.
PositionType
Positional attention mechanism.
RunOutcome
Outcome returned by run_codec.

Constants§

CHECKPOINT_VERSION
Checkpoint schema version.
DONE_NAME
Sentinel filename marking successful codec completion.
REQUEUE_EXIT_CODE
Exit code returned when stopping cleanly at a block boundary for requeue/resume.
STOP_NAME
Sentinel filename requesting clean checkpoint-and-stop at the next block boundary.

Functions§

clear_stop
Removes STOP_REQUESTED in dir if present.
clip_grad_norm
Computes the global L2 norm across vars’ gradients and rescales grads in place when above max_norm.
is_done
Returns true if dir/DONE exists.
load_checkpoint
Loads an existing checkpoint from dir, verifying latest.safetensors against tensors_sha256.
lr_at
Linear warmup followed by cosine decay to min_lr.
mark_done
Atomically writes the DONE completion marker in dir.
request_stop
Writes STOP_REQUESTED in dir.
resolve_device
Resolves a Candle [Device] matching the requested DeviceChoice.
run_codec
Runs online neural compression, decompression, or entropy estimation.
save_checkpoint
Saves a resumable checkpoint (latest.safetensors + latest.json) atomically under dir.
stop_requested
Returns true if dir/STOP_REQUESTED exists.