pub struct TokenBlockDataset {
pub blocks: Vec<Vec<u32>>,
pub block_size: usize,
pub cache_dir: Option<PathBuf>,
}Expand description
Persisted token-block dataset with fast binary cache round-trip (blocks.bin + metadata.json).
Fields§
§blocks: Vec<Vec<u32>>Fixed-length token sequences (num_blocks, block_size).
block_size: usizeBlock sequence length.
cache_dir: Option<PathBuf>Cache directory if persisted.
Implementations§
Source§impl TokenBlockDataset
impl TokenBlockDataset
Sourcepub fn from_blocks(blocks: Vec<Vec<u32>>, block_size: usize) -> Self
pub fn from_blocks(blocks: Vec<Vec<u32>>, block_size: usize) -> Self
Construct in-memory from raw token blocks.
Sourcepub fn save_to_disk(
&self,
dir: &Path,
dataset_name: &str,
tokenizer_info: &str,
) -> Result<()>
pub fn save_to_disk( &self, dir: &Path, dataset_name: &str, tokenizer_info: &str, ) -> Result<()>
Save token blocks as little-endian u32 chunks alongside metadata.json and tokenizer_info.
§Errors
Returns an I/O or serialization error if writing fails.
Sourcepub fn load_from_disk(dir: &Path) -> Result<Self>
pub fn load_from_disk(dir: &Path) -> Result<Self>
Load previously cached token blocks from dir.
§Errors
Returns an error if metadata or blocks.bin is missing or truncated.
Trait Implementations§
Source§impl Clone for TokenBlockDataset
impl Clone for TokenBlockDataset
Auto Trait Implementations§
impl Freeze for TokenBlockDataset
impl RefUnwindSafe for TokenBlockDataset
impl Send for TokenBlockDataset
impl Sync for TokenBlockDataset
impl Unpin for TokenBlockDataset
impl UnsafeUnpin for TokenBlockDataset
impl UnwindSafe for TokenBlockDataset
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> ErasedDestructor for Twhere
T: 'static,
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
impl<T, U> Imply<T> for U
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more