Expand description
Dataset loading, tokenization, SHA-256 caching, and block chunking (utils/datautils.py).
Structs§
- Token
Block Dataset - Persisted token-block dataset with fast binary cache round-trip (
blocks.bin+metadata.json).
Enums§
- Dataset
Kind - Supported training and perplexity evaluation datasets.
Constants§
- DEFAULT_
BLOCK_ SIZE - Default sequence block length matching
utils/datautils.py.
Functions§
- group_
token_ blocks - Concatenate tokenized streams into contiguous
block_sizechunks, dropping any short tail remainder (exact match togroup_textsinutils/datautils.py). - load_
eval_ tokens - Build a single contiguous evaluation token vector for perplexity measurement (
get_eval_loaders). - load_
local_ texts - Extract text documents from a local
.txt,.jsonl, or.jsonl.gzfile. - load_
tokenizer - Load a Hugging Face
tokenizer.jsonfrom a local checkpoint directory or Hub model repo ID. - prepare_
dataset - Prepare or load cached training token blocks matching
utils/datautils.py::prepare_dataset. - tokenizer_
cache_ key - Compute 7-character SHA-256 hex cache key from tokenizer config metadata,
matching
hashlib.sha256(hash_key.encode()).hexdigest()[:7].