PYTHON · LLM TRAINING PIPELINE

minimind — Train a 64M-Parameter Transformer LLM from Scratch in Under 2 Hours

minimind is an end-to-end educational and production LLM training pipeline that builds a compact 64-million-parameter Transformer from raw text to chat deployment. It covers BPE tokenizer training, RoPE/SwiGLU pretraining, supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and Mixture-of-Experts routing.

$ python train_pretrain.py

GitHub ↗

Model Parameters 25.8M – 64.2M

Training Wall Time ~1h 54m

Architecture RoPE + GQA + SwiGLU

VRAM Footprint < 6.4 GB FP16

Interactive Transformer Architecture, VRAM & Chinchilla Scaling Calculator Model Hyperparameters (dim, n_layers, n_heads, vocab_size) Training Stage Pipeline

Deterministic Execution Output & Telemetry STATUS: READY // 0 ERRORS

Python (PyTorch RoPE + GQA Block)

class MiniMindBlock(nn.Module):
    def __init__(self, args: LMConfig):
        super().__init__()
        self.attn_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.attn = GroupedQueryAttention(args)
        self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.ffn = SwiGLUFFN(args.dim, args.hidden_dim)

    def forward(self, x, freqs_cos, freqs_sin, past_kv=None):
        h, kv = self.attn(self.attn_norm(x), freqs_cos, freqs_sin, past_kv)
        out = x + h
        return out + self.ffn(self.ffn_norm(out)), kv

Production Engineering Specifications

Repository:    https://github.com/bhubbard/minimind
Live Studio:   https://code.brandonhubbard.com/minimind/
Portal:        https://code.brandonhubbard.com/
Install:       git clone https://github.com/bhubbard/minimind && pip install -r requirements.txt
Design System: Brandon Hubbard Swiss Modernist (#0A0A0C / 1px Exposed Structural Grid)