Skip to main content

Module data

Module data 

Source
Expand description

Dataset loading, tokenization, SHA-256 caching, and block chunking (utils/datautils.py).

Structs§

TokenBlockDataset
Persisted token-block dataset with fast binary cache round-trip (blocks.bin + metadata.json).

Enums§

DatasetKind
Supported training and perplexity evaluation datasets.

Constants§

DEFAULT_BLOCK_SIZE
Default sequence block length matching utils/datautils.py.

Functions§

group_token_blocks
Concatenate tokenized streams into contiguous block_size chunks, dropping any short tail remainder (exact match to group_texts in utils/datautils.py).
load_eval_tokens
Build a single contiguous evaluation token vector for perplexity measurement (get_eval_loaders).
load_local_texts
Extract text documents from a local .txt, .jsonl, or .jsonl.gz file.
load_tokenizer
Load a Hugging Face tokenizer.json from a local checkpoint directory or Hub model repo ID.
prepare_dataset
Prepare or load cached training token blocks matching utils/datautils.py::prepare_dataset.
tokenizer_cache_key
Compute 7-character SHA-256 hex cache key from tokenizer config metadata, matching hashlib.sha256(hash_key.encode()).hexdigest()[:7].