pub struct CodecConfig {Show 21 fields
pub model: ModelConfig,
pub optim: OptimConfig,
pub retrain: RetrainConfig,
pub batch_size: usize,
pub block_len: usize,
pub online_loss: OnlineLoss,
pub project_only_targets: bool,
pub seed: u64,
pub dtype: ComputeDType,
pub autocast_policy: AutocastPolicy,
pub device: DeviceChoice,
pub max_tokens: Option<usize>,
pub raw_bytes: Option<u64>,
pub vocab_bytes: u64,
pub vocab_file: Option<PathBuf>,
pub initial_prior: Option<PathBuf>,
pub log_interval: usize,
pub checkpoint_dir: Option<PathBuf>,
pub checkpoint_every_blocks: usize,
pub stop_after_blocks: Option<usize>,
pub run_dir: Option<PathBuf>,
}Expand description
Complete configuration for running the neural compressor or decompressor.
Fields§
§model: ModelConfigTransformer architecture configuration.
optim: OptimConfigOnline Adam configuration.
retrain: RetrainConfigPrefix replay configuration.
batch_size: usizeNumber of parallel streams per block.
block_len: usizeTarget token count per block (rounded down to multiple of batch_size).
online_loss: OnlineLossSupervision loss mode (Code vs Window).
project_only_targets: boolProject only supervised positions through head.
seed: u64Deterministic PRNG seed.
dtype: ComputeDTypeActivation precision (Fp32 or Bf16).
autocast_policy: AutocastPolicyControls when Bf16 autocast is active.
device: DeviceChoiceExecution device choice.
max_tokens: Option<usize>Optional cap on input tokens during compression.
raw_bytes: Option<u64>Original byte length before preprocessing (for bits/byte metrics).
vocab_bytes: u64Vocabulary file byte size (for benchmark accounting).
vocab_file: Option<PathBuf>Path to vocabulary file required when initial_prior is set.
initial_prior: Option<PathBuf>Path to QPI1 quantized prior archive (weights.bin).
log_interval: usizeProgress logging interval in symbols.
checkpoint_dir: Option<PathBuf>Optional directory for durable resumable checkpoints.
checkpoint_every_blocks: usizePeriodic checkpoint interval in completed blocks.
stop_after_blocks: Option<usize>Optional block limit to trigger clean checkpoint & requeue exit (for testing).
run_dir: Option<PathBuf>Optional directory where log.txt is appended.
Implementations§
Source§impl CodecConfig
impl CodecConfig
Sourcepub fn tiny(vocab_size: usize) -> Self
pub fn tiny(vocab_size: usize) -> Self
Returns a fast, tiny preset suitable for unit/integration tests and quick local runs.
Sourcepub fn validate(&self, mode: CodecMode) -> Result<(), CodecError>
pub fn validate(&self, mode: CodecMode) -> Result<(), CodecError>
Validates all invariants before starting compression or decompression.
§Errors
Returns CodecError::InvalidConfig on invalid settings.
Sourcepub fn checkpoint_signature(&self) -> String
pub fn checkpoint_signature(&self) -> String
Computes a deterministic JSON configuration signature used to reject incompatible resumes.
Sourcepub fn effective_compute_dtype(&self, device: &Device) -> DType
pub fn effective_compute_dtype(&self, device: &Device) -> DType
Resolves the effective activation DType for device.
Trait Implementations§
Source§impl Clone for CodecConfig
impl Clone for CodecConfig
Source§impl Debug for CodecConfig
impl Debug for CodecConfig
Source§impl<'de> Deserialize<'de> for CodecConfig
impl<'de> Deserialize<'de> for CodecConfig
Source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Source§impl PartialEq for CodecConfig
impl PartialEq for CodecConfig
Source§impl Serialize for CodecConfig
impl Serialize for CodecConfig
impl StructuralPartialEq for CodecConfig
Auto Trait Implementations§
impl Freeze for CodecConfig
impl RefUnwindSafe for CodecConfig
impl Send for CodecConfig
impl Sync for CodecConfig
impl Unpin for CodecConfig
impl UnsafeUnpin for CodecConfig
impl UnwindSafe for CodecConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> DeserializeOwned for Twhere
T: for<'de> Deserialize<'de>,
impl<T> ErasedDestructor for Twhere
T: 'static,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more