Antirez/h3.c: MiniMax H3 inference engine for Mac computers

Native MiniMax-H3 conclusion for Apple Silicon. The task is being built arsenic a sequence of moving vertical slices: deterministic host/model metadata first, then portable Metal artifact parity, punctual encoding, prompt-to-video/audio, and first/last-frame conditioning and past ordered references.

Prompt-to-video/audio, first/last-frame conditioning, and ordered Ref2VA image/video/audio references activity extremity to end. The existent activity is incremental H3-specific Metal capacity and representation optimization connected M3 Max and M5 Max.

The examples presume that the Hugging Face snapshot is successful ./MiniMax-H3 and that FFmpeg and FFprobe are disposable connected PATH.

--info checks the exemplary layout and prints the selected Metal instrumentality without mapping each weights aliases generating media. Run ./h3 --help for the complete CLI reference.

Without -p, the aforesaid binary starts an Iris-style interactive session:

Type a punctual to make a numbered video. The convention keeps the nonstop BF16 prompt conditioning, prepared DiT, and video decoder successful memory, truthful repeating a prompt pinch different seed avoids loading and encoding them again. Useful commands are !status, !seed random, !seconds 2, !show, !save output.mp4, and !cache. Use !help for the full, short list.

First/last-frame conditioning is persistent successful the session:

Use !first clear aliases !last clear to region an anchor. Generated videos are written to the convention directory printed astatine startup.

For a wide Ref2VA conditioning image, usage !ref-image PATH instead. Images are appended successful bid and exposed to the exemplary arsenic <Picture 1>, <Picture 2>, and truthful on; filenames person nary meaning to the model.

!refs lists the existent order, !ref-remove N removes 1 entry, and !refs clear removes them all. Ref2VA references cannot beryllium mixed with !first/!last anchors.

Start pinch the validated balanced preset. It generates 22 frames astatine 24 fps (about 0.92 seconds), displays the evolving middle-video framework aft every denoising modulation successful a supported graphical terminal, and prints phase timings:

This is deliberately not the astir fierce configuration:

• --steps 20 performs the default 20 denoising passes.
• --reuse 2 computes 11 caller denoiser velocities alternatively of each 20 and extrapolates the skipped transitions.
• --layers 45 runs 45 of the 50 transformer blocks, reducing some clip and unified-memory use.
• --show is optional. It supports Kitty/Ghostty and iTerm2/WezTerm/Konsole graphical protocols. It loads a resident preview VAE, displays 1 typical middle-video framework aft each Euler transition, and past displays each last frames. Display dimensions default to 2x truthful the image has its intended logical size connected macOS Retina screens; usage --zoom 1 on a non-HiDPI display. This adds preview decode clip and astir 10 GiB of temporary exemplary residency; runs without --show are unchanged.
• --profile is optional and does not prime a different procreation path.

The first process invocation besides pays exemplary loading and filesystem-cache costs. Compare capacity utilizing repeated runs, and alternate variants when the machines are warming up because this workload is delicate to thermal throttling.

For a very short iteration, petition 4 denoising passes directly:

--steps N ever intends precisely N denoising passes. Four done 7 passes use the aforesaid schedule that won the low-budget comparison; expanding from 4 to 7 progressively improves item and motion. Keep --reuse 1 astatine specified small budgets truthful each requested walk runs the model. --show displays 1 preview after each pass.

Several tail-heavy schedules were evaluated because astir visible cleanup happens precocious successful a agelong run. They preserved excessively fewer early creation updates and produced woven texture, anemic motion, aliases clipped colors. The retained mode uses the released linear guidelines grid pinch 1 terminal point. On the 512-square, 22-frame fox test, the selected four-pass consequence had 0.556 full-video SSIM against a 29-pass reference; an independent surfer trial measured 0.547. The four-pass denoise took astir 3.5 seconds connected M5 Max, versus 26.4 seconds for the reference.

Change 1 power astatine a clip erstwhile evaluating quality. First reconstruct each layers, then each denoiser evaluations, and yet raise the default 20-pass schedule to the slower 50-pass reference:

The defaults are --steps 20 --layers 50 --reuse 1; support --steps 50 explicit for this adjacent path. It performs 50 complete 50-block denoiser forwards and is overmuch much costly than the default, but is the correct oracle when a accelerated mode changes the subject, anatomy, motion, aliases composition. Numerical pixel personality pinch MLX is not expected because the random-number and execution engines differ; the depicted contented and mobility should agree.

These controls are independent unless noted otherwise:

On M5, --use-int8-row-fc2 uses 1 activation standard per FC2 statement and a single full-width TensorOps product. It is optional because it is little numerically conservative than grouped int8. It reduced complete denoiser forwards by about 2.6% successful reciprocal tests. Matched four-step fox and surfer videos kept the same subjects, setting, and mobility (full-video SSIM 0.919 and 0.828). In the interactive session, usage !int8-row-fc2 on.

--reuse and --core-reuse are mutually exclusive. Layer thinning tin be combined pinch either one.

To make the first bid faster while keeping its output resolution, add token reduction:

At the validated 512 quadrate shape, token simplification trim the 45 layers + reuse 2 denoise floor plan from 16.69 to 12.60 seconds connected the IT M5 Max. Independent fox and surfer renders stayed coherent, but creation tin diverge much from the adjacent path.

For an fierce preview, render internally astatine 320 quadrate and upscale to the requested 512 quadrate output:

This operation produced a clean, recognizable 22-frame fox successful validation, but loses good item and tin alteration framing. Do not adhd --token-reduction to some --layers 40 and --reuse 3: that tested operation produced color ringing, outlines, and ghosted limbs.

As an replacement to whole-velocity reuse, this keeps the timestep-dependent patch and output heads caller astatine each transition:

Use --core-reuse 6 only arsenic an fierce preview. Values supra 6 are not exposed because validation mislaid taxable fidelity.

Width and tallness must each beryllium multiples of 32, astatine slightest 32, and their product must not transcend 768 * 1344 pixels. Those are mechanical limits, not a promise that each mini canvas has bully exemplary quality. H3-Base is simply a 768p model.

For a accelerated autochthonal 256-square preview:

At 256 square, H3 has only an 8x8 effective spatial-token grid, truthful it has less room for good item and analyzable composition. H3 automatically halves spatial RoPE coordinates astatine precisely 256 square. This removed repeating lattice artifacts successful agelong fox renders and stayed coherent connected an independent portrait, without adding tokens aliases runtime. Use --use-reference-rope to reconstruct the released/MLX coordinates for parity checks. Keep token simplification disconnected astatine this size. Native 128 quadrate remains unsupported: its 4x4 token grid did not recover a recognizable taxable moreover pinch adjusted RoPE.

--render-width and --render-height must beryllium group together, must person the same aspect ratio arsenic the output, and cannot transcend the output dimensions. The model and VAE usage the soul size; terminal frames and the encoded video clasp the requested output size.

H3 emits 24 fps and aligns framework requests upward to 5 + 17*n:

Use --seconds N for a duration-oriented request, aliases --frames N for direct frame control; the 2 options are mutually exclusive. Fractional seconds are accepted. Seconds are converted astatine 24 fps and past rounded upward to the next legal H3 temporal shape, truthful --seconds 10 produces 243 frames (10.125 seconds).

Short clips are useful for development. The released workflow is intended for roughly 4–15 2nd videos. A petition specified arsenic --frames 23 is rounded up to 39 frames alternatively than producing an arbitrary temporal shape.

A short punctual works, but the released strategy expects a Context-IR-like description. State the subject, action, setting, camera, lighting/style, and desired sound. For example:

Keep personality and entity counts definitive erstwhile they matter. --seed N controls the autochthonal random stream; the default is 42. Compare options pinch the same prompt, seed, resolution, framework count, and measurement count.

• --show displays a typical framework aft each denoising transition, followed by each frames from the completed video. Like Iris, it advertises 2x display dimensions by default for Retina terminals; --zoom N changes that factor without resizing the generated video aliases the encoded terminal image.
• --frames-dir DIR writes last callback frames arsenic PPM files. Intermediate --show previews are not written there.
• -o '' disables MP4 encoding; harvester it pinch --frames-dir erstwhile FFmpeg is unavailable.
• --profile reports shape wall time, Metal encoding/wait time, highest live tensor storage, cumulative allocation, and dispatch counts.

For example:

First/last-frame anchors prime the FL2VA path:

Ordered references prime the chopped Ref2VA checkpoint. Use the emblem matching the media semantics:

Reference flags whitethorn beryllium repeated and their command-line bid is preserved. Standalone audio must travel an image aliases video reference. Audio references must beryllium 2–15 seconds; astatine astir 3 audio inputs are accepted and their total decoded long is capped astatine 15 seconds.

make trial runs the deterministic big suite and, erstwhile the ignored MLX fixture is installed nether misc/fixtures/, compiles the Metal root astatine runtime and checks a complete artifact H3 artifact against named MLX outputs. Runtime compilation is intentional: it follows Iris and does not require Xcode's optional offline Metal toolchain. The trial covers some an F32 test way and the accumulation BF16 storage path; wide BF16 matrix products and SDPA usage cached MPSGraph graphs, with direct Metal correctness fallbacks. make parity runs only those Metal/MLX checks.

FFmpeg and FFprobe must beryllium disposable connected PATH for media inputs and MP4 output (H3_FFMPEG and H3_FFPROBE whitethorn prime definitive executables). Generated RGB24 and 32 kHz stereo F32 PCM are fed done concurrent pipes; nary intermediate uncompressed media record is created.

The remainder documents the implementation down the tutorial presets and the environment variables retained for nonstop A/B diagnosis.

The default sampler uses the released shifted video/audio schedule. --steps always names the number of denoising passes, pinch terminal zero added aft the last pass. Whole-denoiser reuse evaluates the first and past walk positive every requested interval, past extrapolates skipped video and audio velocities on their independent schedules. With very mini measurement counts, support --reuse 1.

For the low-budget path, the released linear guidelines grid won against actual-video-sigma linear spacing, quadratic and cubic warps, nonstop 30-point tail subsets, mild powerfulness warps, zero-order held full-grid velocities, linear velocity extrapolation, and RES. The much tail-heavy candidates often sharpened the taxable but damaged motion or near a repetitive woven background; sparse RES and agelong extrapolation intervals grounded overmuch much visibly.

Layer thinning ranks the checkpoint's existent AdaLN gates while protecting structurally important first and last blocks. Unused weights and schedule tensors are not retained, truthful --layers 45 and --layers 40 trim both transformer clip and unified-memory use. Core reuse holds the erstwhile full transformer residual while refreshing the spot projection and timestep-aware head; it remains mutually exclusive pinch whole-velocity reuse.

Every progressive DiT artifact fuses its attraction residual gross pinch the pursuing MLP AdaLN. The rounded BF16 residual is still written exactly, but the aforesaid statement is kept successful threadgroup representation for normalization, eliminating 1 dispatch and one global reread. Away from token-reduction boundaries, the MLP residual gross also produces the adjacent block's attraction AdaLN and carries that normalized state across the loop. H3_DISABLE_FUSED_GATE_ADALN=1 and H3_DISABLE_FUSED_CROSS_BLOCK_ADALN=1 reconstruct the two-kernel oracles. The last audio/video AdaLN kernels hindrance straight to offsets successful the residual stream, avoiding 2 portion blits and 18.8 MiB of scratch astatine 512x512 (29.4 MiB at the 864-class benchmark shape). H3_DISABLE_FUSED_FINAL_SLICE=1 restores the copy-plus-AdaLN oracle astatine load. The BF16 last heads past use AdaLN while loading their 16x16 projection tiles, preserving the standalone rounding and accumulation bid while removing different arsenic sized normalized activation. The 2 optimizations together prevention 37.5/58.9 MiB. H3_DISABLE_FUSED_FINAL_HEAD=1 restores the offset-AdaLN-plus-linear oracle astatine load.

--token-reduction is an independent fierce DiT mode. After artifact 3 it pairs adjacent horizontal target-video tokens while leaving text, audio, conditions, and reference tokens exact. The complete full-resolution authorities is kept arsenic a bypass. During the first 10 noisy evaluations it restores before block 40; consequent detail-forming evaluations reconstruct earlier artifact 30. Each token returns arsenic its original worth positive the update learned by its pair, so within-pair item is not discarded. The pooling kernel writes only true-pair baselines into a dense tail of the already allocated attraction scratch buffer; odd-width singleton tokens request no baseline. The afloat bypass uses the oversized QKV tail erstwhile it fits, pinch a guarded dedicated fallback only for reference-heavy layouts. Common text-only canvases truthful adhd nary activation arena astatine immoderate token-grid width. Pooling also snapshots some root tokens while their BF16 values are already in registers, avoiding a abstracted full-hidden blit and redundant root read. The same introduction kernel keeps each pooled statement successful threadgroup representation and emits the first reduced block's attraction AdaLN, eliminating different world residual read. At the reconstruct boundary, the first full-resolution attraction AdaLN is fused into description : a 10.5 KiB threadgroup statement avoids a world residual reread while still penning the nonstop bypass needed by the pursuing residual branch. On a thermal-balanced 512x512x22, 19-forward IT M5 Max A/B this reduced denoise time from 39.13 to 28.06 seconds (28.3%). Final video/audio latent comparative L2 was 5.56%/15.14%. First/middle/last fox frames retained 1 cleanable muzzle, coherent legs, and crisp fur; an independent surfer remained accordant with one rider and committee done the activity spray. It changes creation and is therefore opt-in alternatively than the close-reference default. H3_TOKEN_REDUCTION_BLOCKS tin override the later 4:30 interval; H3_TOKEN_REDUCTION_EARLY=STEPS:END overrides the early schedule and 0 disables it. H3_DISABLE_TOKEN_REDUCTION=1 provides an in-context nonstop oracle. H3_DISABLE_FUSED_TOKEN_POOL_ADALN=1 and H3_DISABLE_FUSED_TOKEN_ADALN=1 independently reconstruct the two-kernel introduction and exit boundaries for diagnosis. Token simplification composes cleanly pinch the validated --layers 45 --reuse 2 settings: connected the aforesaid 512 benchmark it reduced that floor plan from 16.69 to 12.60 seconds (24.5% marginal), and independent fox and surfer renders stayed coherent. Do not harvester it pinch some --layers 40 and --reuse 3; that 6.47-second research produced chromatic ringing and ghosted limbs despite acceptable latent norms.

--render-width and --render-height tally the exemplary and VAE connected a lower same-aspect soul canvas, past high-quality vImage-scale RGB frames to the requested output size earlier callbacks, terminal display, and encoding. This is an explicit quality/speed tradeoff: a measured 384-to-512 punctual render reduced M5 DiT clip by 33% and video-VAE clip by 18% while retaining a clean, recognizable photorealistic result. Both values must beryllium multiples of 32; the exact output canvas remains the default. For quadrate 512 output, 384 is the fast-quality constituent and 320 is the validated aggressive point. The second produced a coherent stepping fox and repeated at 8.02 seconds of DiT versus astir 15.82 seconds natively. Native 256 uses the same-cost spatial-RoPE adjustment described above; it remains a accelerated composition preview alternatively than a substitute for a 512- aliases 768-class last render. The video VAE automatically chooses a 256-320 pixel spatial tile from the requested canvas geometry, minimizing repeated overlap activity while keeping peak storage bounded. H3_VAE_TILE_PIXELS=256 restores the original conservative tile scheme for close-reference diagnosis.

On M5-class GPUs, persistent transformer weights are mapped straight from their safetensor shards alternatively of copied into anonymous shared buffers. This keeps the 37 GiB exemplary file-backed/reclaimable and somewhat improves full transformer time; M3 uses the faster copied-buffer path. H3_ZERO_COPY_WEIGHTS=0 disables the M5 action for diagnostics. The streamed Qwen matter encoder preallocates a mini ringing of early layer buffers and fills them connected 8 I/O workers while Metal executes the current layer. The default ringing extent is 2 layers connected M3/older hardware and 3 on M5, wherever the target instrumentality has 128 GiB. H3_QWEN_PREFETCH=0 restores the single-layer synchronous reference path; values 1-8 prime the worker count, and H3_QWEN_PREFETCH_DEPTH=1 done 6 overrides the ringing depth.

M5 GPUs automatically usage autochthonal BF16 Metal 4/TensorOps for the DiT QKV and attention-output projections astatine series lengths up to 2,048. The compact Morton schedule routes Q/K/V straight into head-major attraction inputs, avoids three MPSGraph input transposes, and is byte-identical to the portable path. It improves a complete 512x512 50-block guardant by astir 2% crossed repeated IT/US M5 Max runs. For 2,049-3,072 rows, including 864x480, 2 row-offset Morton dispatches sphere the businesslike tile geometry and amended the complete forward by astir 2% successful balanced runs. Still larger sequences enactment connected MPSGraph. H3_NAX=0 disables TensorOps for nonstop A/B diagnosis. The action is guarded at runtime and falls backmost to the unchanged portable room if compilation is unavailable.

H3_NAX=1 forces the broader autochthonal BF16 linear path. It passes the complete 50-block MLX fixture, but remains opt-in: exact-shape microbenchmarks favour its 128-row tile while afloat DiT runs presently favour MPSGraph scheduling. This keeps a moving NAX integration disposable for later quantized/fused kernels without making a benchmark regression the default. H3_NAX=mlp selects a much specialized Metal 4 path: paired FC1 gate/up TensorOps tiles use SwiGLU successful threadgroup representation and constitute only the 14,336-wide activated intermediate, past FC2 besides stays connected TensorOps. H3_DISABLE_NAX_MLP=1 keeps the MPSGraph MLP successful a discourse created this measurement for same-process A/B testing. The way is deliberately opt-in because scheduling depends connected the OS GPU stack: the superior macOS 26.5.2 M5 Max gained 1.3-2.0% in isolated real-weight MLP runs but mislaid astir 1-3% successful a complete 50-block forward, while an different identical macOS 26.5 M5 Max gained 1.4% successful a same-context forward A/B. The resulting 50-block velocities were adjacent (1.9% video and 2.4% audio comparative L2), but not byte-identical.

The constrictive DiT audio/video output heads person their mini released F32 weights to BF16 erstwhile and usage the Iris-derived 16x16 tiled linear straight on BF16 activations. At the accumulation 320-render geometry, isolated paired-head measurements are 2.30x faster connected M3 Max and 1.83x faster connected M5 Max, with relative L2 8.64e-4; the absolute M5 redeeming is astir 0.6 sclerosis per evaluated step. Full fox and surfer sequences remained cleanable and measured 29.9/38.4 dB against the F32-head renders. H3_DIT_F32_FINAL=1 restores the close-reference head and its other activation buffers. The F32 96->5376 video and 32->5376 audio spot projections usage a dedicated 16x16 cooperative tile, retaining F32 weights, inputs and accumulation while rounding the tile consequence straight to BF16. Paired production-shape measurements are 1.77x faster connected M3 and 1.62-1.78x on M5; the complete generated RGB watercourse is byte-identical to the scalar path. Fusing the last formed improves the 2835-row tile itself from 2.499 to 1.734 ms on M3 and 1.555 to 1.186 sclerosis connected M5, and removes 38.27/59.66 MiB of F32 scratch at 512/864-class geometry. H3_DISABLE_FUSED_PATCH_CAST=1 restores the tiled F32 output positive standalone cast; H3_SCALAR_PATCH=1 selects the scalar diagnostic path. The aforesaid tile binds its output straight into the packed hidden stream, removing the BF16 media staging buffers and their blits. This saves different 19.13/29.83 MiB and improves the 2835-row bound from 1.847 to 1.730 sclerosis connected M3 and 1.282 to 1.184 sclerosis connected M5. Contiguous T2VA uses byte offsets; FL2VA/Ref2VA usage compact destination-row maps truthful each modality remains 1 ample dispatch. A complete six-segment Ref2VA M5 ABBA remained byte-identical and improved 5.067 to 5.033 seconds per measured guardant pair. H3_DISABLE_FUSED_PATCH_PACK=1 restores the staging buffers and packing blits.

The DiT halfway is divided into 2 ordered Metal bid buffers truthful GPU execution of the first portion overlaps CPU encoding of the second. Thermal-balanced ABBA measurements prime a 60%-depth divided connected M5 (30/50, 27/45, and 24/40), with roughly 0.5-1.8% wins; M3 automatically splits only the validated 30/50 case, which measured 1.2% faster, because 24/40 regressed there. The cognition order and generated bytes are unchanged. H3_DIT_COMMAND_BLOCKS=0 restores one command buffer; values 1-50 override the divided for further tuning. DiT activation buffers besides travel their existent intra-block lifetimes: the QKV projection arena is reused first for attraction heads and past for the normalized MLP input, while the existent attention-output arena becomes the MLP output after its branch has been consumed. This removes 61.25 MiB astatine 512-class geometry and 99.63 MiB astatine 864-class geometry without changing dispatches aliases arithmetic. H3_DISABLE_DIT_ACTIVATION_ALIAS=1 restores abstracted diagnostic buffers. MPSGraph tensor-data wrappers for immutable DiT weights and biases are retained with their resident buffers. This avoids rebuilding the aforesaid binding metadata for each artifact and denoiser information without copying tensor storage; measured ABBA gains were 1.6% connected M3 Max and 0.4-1.1% connected M5 Max. Activation wrappers stay transient because retaining them regressed the M5. The outputs remain byte-identical, and H3_DISABLE_GRAPH_DATA_CACHE=1 restores transient wrappers for each tensors. On M3/older hardware, the 4 MPSGraph segments successful each DiT artifact besides reuse one MPSCommandBuffer wrapper for their shared underlying Metal bid buffer. Repeated thermal-balanced runs measured 1.0-1.6% faster connected M3 Max; M5 measured neutral, truthful it retains caller wrappers. H3_REUSE_MPS_COMMAND=0 aliases 1 overrides the automatic selection. Results are byte-identical. On M5, the serving Euler sampler keeps its patch-packed F32 latents and cached BF16 velocities successful Metal buffers. Each selected denoiser refresh is completed before the adjacent is encoded, avoiding MPSGraph back-pressure while removing all intermediate latent/velocity readbacks and repacking. Two lukewarm eight-run A/B sequences measured mini 0.1% and 0.3% gains pinch byte-identical last latents; the way besides saves astir 16 bytes of transient big authorities per video-latent element (about 136 MB astatine the 768p shape). M3 and older GPUs clasp the CPU sampler by default. H3_CPU_SAMPLER=1 restores it connected M5; H3_GPU_SAMPLER=1 selects the GPU-state way explicitly, and H3_GPU_SAMPLER_WINDOW=0 enables the slower unbounded encode-ahead diagnostic.

The released checkpoint stores DiT QKV rows interleaved per attraction head. Native Metal consumes that layout straight successful the fused QK-normalization/RoPE kernel, avoiding a checkpoint transpose and other RAM. The earlier identity interpretation was the origin of the noisy diagnostic outputs.

The nationalist procreation way decodes the associated audio latent pinch a streamed native BigVGAN/AudioVAE and writes synchronized H.264 positive 32 kHz stereo AAC. The native waveform agrees pinch the corrected MLX oracle to comparative L2 6.94e-5. --first-frame, --last-frame, and their operation usage the released visual VAE encoder, Qwen3-VL imagination building and three-deepstack multimodal presentation, 0.999 information augmentation, and fixed information rows successful the autochthonal DiT. The first image is stretched to the target canvas; the past image is aspect-cover scaled and halfway cropped, matching the reference implementation. --ref-image selects the chopped Ref2VA transformer, preserves ordered <Picture N> presentation, and uses the released down-only aspect-preserving reference canvas. --ref-silent-video additionally performs bounded 24 fps decoding, the visual VAE's causal ceil(T/4) compression, two-frame Qwen sampling, and timestamped <Video N> presentation. --ref-video preserves an embedded soundtrack, --ref-video-audio VIDEO AUDIO supplies an definitive replacement, and --ref-audio appends an ordered standalone clip. Reference audio is decoded as 32 kHz stereo F32, encoded by the autochthonal AudioVAE posterior-mean path, mixed as 0.999 cleanable latent positive 0.001 seeded noise, pinned to the audio condition timestep 1.0, and packed arsenic width-32 rows connected the aforesaid rotary timeline arsenic visual references. Audio inputs are 2-15 seconds, astatine astir 3 are accepted, their full decoded long is capped astatine 15 seconds, and a standalone audio reference must beryllium mixed pinch an image aliases video reference.

The autochthonal audio encoder matches the corrected MLX oracle astatine comparative L2 3.59e-6 connected a existent two-second stereo fixture. The correction is important: the original MLX reshape interleaved left/right samples, whereas the official PyTorch/SGLang way folds intact stereo channels into the batch dimension. On the 128 GB M5 Max, cleanable end-to-end image+audio and embedded-video+audio renders completed successful 74.58 and 76.99 seconds respectively, each pinch astir a 40.1 GB peak beingness footprint and zero swaps.

--profile reports each Metal-backed shape separately: wall time, CPU-side command encoding, complete commit-to-fence wait, root-command GPU timestamps, peak unrecorded tensor storage, cumulative allocation, and dispatch counts. The wait measurement is the complete bid turnaround; the guidelines GPU timestamp alone can omit kid buffers scheduled internally by MPSGraph and is labeled accordingly.

The DiT accelerated way evaluates each BF16 fc1 -> SwiGLU -> fc2 artifact arsenic 1 cached graph, avoiding abstracted chart boundaries and persistent intermediate tensors. Set H3_DISABLE_FUSED_MLP=1 to clasp the close-reference cognition boundaries for numerical diagnosis.

On supported M5 Metal 4 TensorOps hardware, the autochthonal int8 MLP motor is the default. It dynamically quantizes activations, uses per-output-channel weight scales, and gives the delicate FC2 input 1 standard per 1,024 channels. The selected FC2 kernel keeps scaled partial products successful backstage cooperative fragments alternatively of many times spilling a 32 KiB threadgroup tile. A fixed 50-layer, 19-transition 512x512 render measured 36.30 seconds pinch BF16 MPS and 25.80 seconds pinch int8 connected M5 Max. Beginning, middle, and last decoded frames retained the aforesaid subject, composition, and motion; mini separator and fur details can differ. The existent diagnostic implementation retains some BF16 and int8 MLP weights only erstwhile an A/B diagnostic requests them. Normal int8 loading releases each block's BF16 FC1/FC2 buffers aft their submitted quantization finishes, reducing measured highest tensor retention to 25.9 GiB from the BF16 path's 36.4 GiB. Runtime weight quantization still adds startup time.

The fastest M5 way besides quantizes each DiT QKV projection and writes its Q/K/V tiles straight successful head-major attraction layout earlier the existing Q/K normalization and RoPE kernel. In a fixed 50-layer, 19-transition 512x512 render this reduced denoising again, from 25.80 to 19.32 seconds. Sampled beginning, middle, and last frames remained a coherent elaborate fox walking through snow; quantized attraction tin alteration framing and good detail. Use --use-slower-bf16-qkv for the close-reference BF16 projection. Normal int8 loading releases the redundant BF16 QKV weights aft quantization.

The pursuing attention-output projection is int8 arsenic good connected the default M5 path. Crossed same-model tests amended a complete guardant by different 4.5-5.5% at 512 and 864. A decoded fox render remained cleanable and intimately matched the int8-QKV-only composition; its thermally basking denoise measured 19.18 seconds. Use --use-slower-bf16-attention-output to clasp that projection successful BF16.

On that int8 path, SDPA now leaves its consequence successful native [head,row,dimension] order. A specialized 256-thread kernel gathers and quantizes each H3 statement straight into the projection's row-major int8 buffer, eliminating the intervening full-width BF16 transpose without changing any output byte. Thermally controlled crossed runs amended complete 512 and 864 forwards by astir 0.2-1.2%. Use --use-slower-row-major-attention-output to reconstruct the definitive BF16 row-major SDPA output and mean quantizer.

The M5 way besides folds QKV and MLP activation quantization into the preceding gated AdaLN kernel. This removes 99 standalone quantizer dispatches per 50-layer guardant while preserving the erstwhile output bytes, improving crossed 512/864 measurements by astir 0.3-0.6%. Use --use-slower-unfused-int8-inputs to reconstruct the standalone quantizers.

The fused gated-AdaLN way loads its afloat 5,376-wide H3 rows arsenic BF16x4 vectors and writes int8x4. It stages the rounded values locally earlier computing the original per-thread RMS sequence, truthful the simplification character and each output byte remain unchanged. Crossed measurements prevention astir different 0.1-0.5%. The existing --use-slower-unfused-int8-inputs action retains the portable scalar and standalone-quantizer fallback.

Q/K RMS normalization and RoPE are performed wrong the int8 QKV projection tile arsenic well. The fused closing is byte-identical and improves complete forwards by 2.1-3.2% astatine 512 and 1.0-1.8% astatine 864 successful crossed M5 measurements. Use --use-slower-unfused-qkv-rope to reconstruct the abstracted Q/K kernel.

That closing processes 4 adjacent Q/K dimensions per activity point with BF16x4 loads and stores. The per-element arithmetic and BF16 rounding bid are unchanged, while crossed cool-state measurements amended complete forwards by about 0.4-1.0% astatine some 512 and 864. The same --use-slower-unfused-qkv-rope action restores the scalar standalone path.

At up to 2,048 rows, the nonstop RMS loop uses BF16x4 loads followed by four explicit ordered FMAs. This preserves each output spot and improves 512-class forwards by different 0.5-0.6%; larger shapes clasp scalar loads because the two forms necktie there. Use --use-slower-scalar-qkv-rms to unit scalar loads.

The int8 attention-output projection caches its 128 statement and file scales in 1 KiB of threadgroup representation alternatively of rereading them for each cooperative fragment element. Above 2,048 rows the fused QKV kernel uses the aforesaid thought and then recycles that retention for inverse RMS values; smaller QKV shapes retain direct loads because the 2 forms necktie there. Both are byte-identical and improve complete forwards by astir 0.2-0.7% wherever selected. Use --use-slower-uncached-int8-scales to reconstruct nonstop device-scale loads.

For sequences of astatine astir 2,048 rows, the H3 attention-output projection also compiles its 7,168-by-5,376 style into the TensorOps kernel. The consequence remains byte-identical while redeeming astir 0.2-0.8% successful crossed complete 512-forward measurements. Larger sequences clasp the dynamic-shape kernel because the specialization regresses there. --use-slower-uncached-int8-scales restores the wide dynamic, direct-scale-load implementation.

FC1 besides uses an H3-specialized, compile-time 5,376-wide TensorOps loop. It is byte-identical to the generic loop and saves astir 0.1-0.4% successful crossed complete forwards. Use --use-slower-dynamic-fc1-k to reconstruct the runtime-bound loop.

Use --use-slower-bf16-mlp to unit the portable close-reference MPS/BF16 MLP path for numerical comparison. Older Metal hardware selects that path automatically erstwhile the required autochthonal TensorOps kernels are unavailable. For FC2 activation quantization, sequences of astatine astir 2,048 rows usage an exact 128-thread reduction. Each thread retains its 8 BF16 input values while computing the group maximum, avoiding a 2nd device-memory publication erstwhile it emits the int8 values; crossed M5 measurements improved complete 512 forwards by about 0.2-0.8% without changing immoderate output byte. Larger sequences clasp the measured 256-thread kernel. --use-slower-grouped-quantizer forces the latter at each size for A/B comparison.

The autochthonal baseline targets the original FL2VA/ and Ref2VA/ checkpoint trees. Model phases are loaded and released separately truthful the 33B transformer, Qwen encoder, and decoders ne'er person to coexist successful unified memory.