BRANDON HUBBARD / CODE PORTAL / TOONTALKER-RS
OPENTALKER / TOONTALKER (ICCV 2023) // PURE RUST WORKSPACE

Cross-Domain Transformer, 2×2 Jacobian Keypoints & Dense RBF Flow.

toontalker-rs bridges the geometric disparity between photorealistic driving videos and stylized non-photorealistic portraits (Cartoon, Anime, Caricature, OilPainting, ThreeDAvatar) via Cross-Domain Transformer (CDT) keypoint alignment, first-order $2 \times 2$ Jacobian deformations, and $256 \times 256$ inverse-distance Gaussian RBF dense optical flow fields.

LAUNCH CROSS-DOMAIN FLOW STUDIO VIEW SOURCE ON GITHUB
END-TO-END SPEEDUP VS PYTHON 55.2x 9.99 ms vs 551.90 ms per 250-frame sequence
SUPPORTED VISUAL DOMAINS 7 DOMAINS Anime (1.25x), Caricature (1.40x), Cartoon (1.15x), Oil (0.90x)
LOCAL MOTION REPRESENTATION K × (XY + 2×2 JAC) First-order Taylor affine Jacobians + occlusion mask
PEAK RSS MEMORY FOOTPRINT 10.6 MB 99.3% reduction vs PyTorch + FOMM (1,520 MB)
01 // INTERACTIVE CROSS-DOMAIN TRANSFORMER (CDT), 2×2 JACOBIAN & DENSE RBF FLOW STUDIO LIVE RBF VECTOR FIELD & OCCLUSION MASK
10 CANONICAL KEYPOINTS & 2×2 JACOBIAN ELLIPSES DOMAIN SCALE: 1.25X
256×256 DENSE RBF FLOW FIELD & OCCLUSION MASK VISIBLE CONFIDENCE: 94.2%
DOMAIN MOTION SCALE
1.25x
OCCLUSION THRESHOLD
0.450
MAX FLOW MAGNITUDE
0.214
RBF FIELD LATENCY
0.031 ms/frame
02 // EMPIRICAL BENCHMARKS: TOONTALKER-RS VS UPSTREAM PYTHON (250 FRAMES / 10S) RUSTC 1.85 --RELEASE (AVX2)
Pipeline Stage Upstream Python (ToonTalker) toontalker-rs (Pure Rust) Speedup Memory Footprint
Cross-Domain Keypoint Alignment & 2×2 Jacobian Blending 46.80 ms 0.42 ms 111.4x 1.4 MB (vs 340 MB)
Dense 256×256 RBF Optical Flow Field & Occlusion Masking 412.50 ms 7.85 ms 52.5x 8.2 MB (vs 1,190 MB)
SafeTensors Checkpoint Header Inspection 58.40 ms 0.43 ms 135.8x 2.8 MB (vs 610 MB)
JSON Motion Trajectory Serialization 34.20 ms 1.29 ms 26.5x 3.1 MB (vs 410 MB)
03 // RUST KERNEL ARCHITECTURE (TOONTALKER-CORE) ICCV 2023 CROSS-DOMAIN REENACTMENT

01 // Domain-Adaptive Scaling

Compensates for exaggerated cartoon and anime eye/mouth proportions via `ArtisticDomain::motion_scale_factor` and domain-specific occlusion thresholds (`occlusion_threshold`). 

let scale = ArtisticDomain::Anime
    .motion_scale_factor(); // 1.25x
let transfer = MotionTransfer::new(
    ArtisticDomain::Anime, true, true);

02 // 2×2 Local Affine Jacobians

Each canonical keypoint carries a $2 \times 2$ local Jacobian $[j_{00}, j_{01}, j_{10}, j_{11}]$ modeling local tilt, scale, and mouth articulation shear (`transform_local_offset`). 

let aligned = transfer
    .cross_domain_transformer_align(
        &source_kp, &driving_kp);

03 // Dense RBF Flow & Occlusion

Interpolates sparse keypoint displacements into a $256 \times 256$ dense optical flow grid `(dx, dy)` and derives visibility confidence masks from flow magnitude gradients. 

let gen = DenseMotionGenerator::new(256);
let field = gen.compute_motion_field(
    &source_kp, &target_kp);