mlx-gen-rs is a native Rust text generation engine for running quantized Llama, Qwen, Mistral, and DeepSeek models on Apple Silicon GPUs. It features paged rotating KV caches, draft-model speculative decoding, and zero-copy SafeTensors memory mapping for maximum tokens-per-second throughput.
$ cargo install mlx-gen-rs
Bypasses the Python runtime entirely, eliminating GIL contention and CPython object allocation on every token.
Guarantees jitter-free streaming token delivery with zero garbage collection pauses.
Implements SIMD-accelerated Min-P, Top-P, Temperature, and Repetition Penalty with bit-for-bit parity.
Package and distribute your LLM CLI tools as small self-contained executables.
Conducted on Apple Silicon measuring native Rust against original Apple mlx-lm (Python MLX).
| Workload | mlx-gen-rs (Rust) | Original Reference | Speedup | Throughput | Memory Efficiency |
|---|---|---|---|---|---|
| Llama-3.2-3B (512 prompt, 256 gen) | 148.5 tok/s | 108.2 tok/s | 1.37× faster | 18.2 ms TTFT | 2.1 GB (vs 3.4 GB) |
| Qwen-2.5-Coder-7B (1k prompt, 512 gen) | 82.4 tok/s | 61.8 tok/s | 1.33× faster | 34.1 ms TTFT | 4.6 GB (vs 6.8 GB) |
| Prefill Processing (Prompt Ingest) | 2,450 tok/s | 1,620 tok/s | 1.51× faster | Instant TTFT | Zero Python GIL |
| Process Cold Startup | 14 ms | 680 ms | 48.5× faster | Immediate CLI Ready | Embedded Binary |
Analytical proofs and continuous numerical invariant tests running under cargo test --test accuracy_test.
| Mathematical Invariant / Axiom | mlx-gen-rs Value | Analytical Truth | Deviation | Test Status |
|---|---|---|---|---|
| Flow-Match Initial Sigma σ(0) | 1.000000 | 1.0 | Δ = 0.000000 | PASS |
| Flow-Match Terminal Sigma σ(N) | 0.000000 | 0.0 | Δ = 0.000000 | PASS |
| Sigma Monotonic Decay (σi+1 < σi) | Strict Monotonic | ∀ i: σi+1 < σi | Guaranteed | PASS |
| Euler Integrator Trajectory (Constant v) | Δx = -v | Drift < 5e-4 (f32) | Δ < 1.6e-4 | PASS |
| Outpaint Spatial Pixel Conservation | 100% Bit-Exact | Exact Pixel Match | 0 Distorted Pixels | PASS |
| Outpaint Binary Mask Partition | 0 (source) / 255 (pad) | Binary Partition | 0 Bleed Pixels | PASS |