Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

Gemma 4 26B-A4B conclusion successful astir 2 GB of RAM
A civilization Swift + Metal runtime for immoderate Apple Silicon Mac, moreover the 8 GB ones.

Quick start · Local server · Benchmarks · Contribute results · How it works · Experiments · References

Memory sewage expensive. So I gave a 26-billion-parameter exemplary a ~2 GB budget.

TurboFieldfare runs the instruction-tuned Gemma 4 26B-A4B without loading the full 14.3 GB exemplary into memory. It keeps the shared 1.35 GB halfway and FP16 KV cache successful memory, past streams only the experts needed for each token from SSD. This is what lets the exemplary tally connected Macs pinch 8 GB of RAM.

The runtime, streaming installer, CLI, and autochthonal Mac app are written successful Swift and Metal. TurboFieldfare is model-specific alternatively than a wrapper astir MLX or llama.cpp. The curated experiment record summarizes 103 measured results crossed kernels, caching, I/O, prefill, and decode.

On the first run, Swift Package Manager downloads and builds the Swift packages required by the tokenizer. The complete merchandise build includes the foreground Mac app and its related decode-service executable.

When the app opens, take Download and fto TurboFieldfare fetch and repack the pinned exemplary (about 15 GB). Once it is ready, take Load Model, type your prompt, and property Generate.

The measured consequence is simply a reference point, not a capacity ceiling. Prompt length, generated length, page-cache state, and hardware each impact throughput. To thief measurement different Apple Silicon Mac, travel the community benchmark guide.

TurboFieldfare provides a autochthonal Mac app, a command-line interface, and an experimental loopback OpenAI-compatible server. They usage the aforesaid .gturbo model directory, but only 1 model-owning merchandise should tally astatine a time.

The Swift package exposes six products:

• An Apple Silicon Mac; the validated target is an 8 GB M2 MacBook Air
• macOS 26 pinch Metal 4
• Xcode 26 and Swift 6.2 aliases newer
• Enough free retention for the ~14.3 GB exemplary installation
• An net relationship for the first exemplary install

The package is arm64-only. Older macOS and Metal versions are not supported.

The Mac app treats what you type arsenic an instruction and handles Gemma's chat formatting automatically. Just picture the task and see immoderate discourse the model needs.

Generation defaults to somesthesia 0.2, Top-K 64, and Top-P 0.95. Set temperature to 0 for deterministic greedy output. The exemplary tin still repeat itself aliases springiness incorrect answers, truthful cheque important results.

TurboFieldfare is text-only. The app and CLI support personification and exemplary messages plus optional strategy guidance; they do not expose aliases execute tools. The loopback server accepts function-tool declarations and returns model-produced instrumentality calls for the customer to authorize and execute. Images, audio, and video are not supported.

Clone the repository, past tally the app from its root:

Build the complete package truthful the app and its related decode work are both available. When launched from this checkout, the app stores the exemplary in scratch/gemma4.gturbo.

On first launch, the app checks the disposable retention and shows the download and installed sizes. Choose Download to begin.

The installer ne'er materializes the afloat root checkpoint. It streams the required byte ranges from the pinned Hugging Face revision and repacks them directly into the .gturbo layout arsenic they arrive. This avoids a 2nd full checkpoint connected disk and keeps scratch representation bounded.

The first installation transfers astir 15 GB done bounded Hugging Face range requests. Network velocity and Hugging Face consequence times vary, truthful it can take a while. The completed .gturbo installation occupies astir 14.3 GB and is accepted only aft its manifest and record hashes person been validated. Installation does not load the exemplary into memory.

After installation:

1. Choose Load Model.
2. Enter a punctual successful the composer.
3. Choose Generate, aliases property Command+Return.
4. Use the extremity fastener aliases Escape to extremity procreation early.

The position barroom shows procreation progress, decode speed, and representation use. Use the right pane to configure sampling, discourse length, expert-cache slots, and runtime options. See Runtime controls for details and defaults.

The CLI uses an existing .gturbo installation. If you installed the model through the Mac app, it is already disposable astatine scratch/gemma4.gturbo. Otherwise, instal it from the bid line:

Continue a cancelled aliases interrupted download:

Remove saved download state:

The runtime accepts only a completed .gturbo directory pinch a final manifest.json.

Verify an existing installation without loading the model:

Put chat messages successful a JSON array and walk it pinch --messages-file:

This formats messages successful the aforesaid measurement arsenic the Mac app. The CLI consequence limit is group pinch --max-new, which defaults to 1,024 tokens. The Mac app can generate until the selected discourse model is full.

--prompt is disposable for earthy completion and reproducible comparisons. It passes the matter straight to the exemplary without chat formatting. Use --messages-file for instruction-response conversations.

This illustration deliberately requests a short greedy completion.

Common procreation options see --max-context, --temperature, --top-k, --top-p, --repetition-penalty, --seed, and repeatable --stop strings. The nationalist CLI uses accumulation runtime defaults. Run the pursuing bid for the complete action list:

Generated matter goes to modular output. Timing statistic spell to modular error; add --quiet to suppress that footer successful scripts.

Build the server and constituent it astatine an installed model:

It listens connected http://127.0.0.1:8080/v1 and supports Chat Completions, streaming, usability tools, and single-prefix punctual reuse. The customer must authorize and tally each instrumentality call. Keep the server connected loopback; it has no remote authentication aliases TLS.

See Local server for a trial request, Python and OpenCode setup, punctual reuse, instrumentality handling, and the supported API subset.

Run the nationalist trial suite serially:

Before starting a exemplary run, adjacent memory-heavy apps and check memory_pressure -Q. If it reports small free memory, postpone the run. Run only 1 TurboFieldfare app, decode service, CLI, server, test, aliases other local-model process astatine a time.

To lend a comparable capacity result, travel the community benchmark guide.

At each transformer layer, Metal computes attraction and the router from resident weights. The CPU uses the router's top-8 master IDs to scheme against the layer's 16-slot LFU cache, past fills misses pinch bounded parallel pread calls into Metal-visible buffers. Metal computes the resident shared-expert branch while those sounds run, past combines the shared and routed outputs.

Prompt prefill uses chunks of up to 128 tokens truthful 1 fetched master tin serve multiple rows. Generation repeats the routed furniture loop 1 token astatine a time. The installer applies the aforesaid bounded-memory rule: it repacks distant ranges directly into .gturbo without staging a afloat shard aliases tensor.

For a ocular preamble to the exemplary architecture, spot Maarten Grootendorst's A Visual Guide to Gemma 4.

System design explains the .gturbo layout, memory ownership, prefill, router handoff, cb1/io/cb2 phases, Metal kernels, and correctness invariants.

TurboFieldfare presently includes:

• Remote streaming repack into the .gturbo exemplary format
• Instruction-tuned Gemma 4 26B-A4B pinch verified text-only chat formatting
• 4-bit MLX affine embedding, attention, shared-expert, and routed-expert weights, pinch an 8-bit router
• Custom Metal kernels for quantized GEMV, attention, MoE, normalization, RoPE, sampling, and accumulation fusions
• SSD-backed routed-expert streaming pinch a bounded master cache
• Chunked single-prompt prefill and token-by-token generation
• FP16 KV retention pinch bounded information retention for 25 sliding-window layers and linear retention for 5 full-attention layers
• Exact split-K/V decode attraction pinch chopped normalized K and V paths
• A Swift library, streaming installer, command-line interface, loopback OpenAI-compatible server, and autochthonal SwiftUI/AppKit Mac app pinch a one-shot local decode service

Current scope is text-only conclusion from the pinned Gemma 4 26B-A4B instruction checkpoint connected Apple Silicon Macs pinch astatine slightest 8 GB of RAM.

• Build iPhone and iPad apps, past measurement conclusion velocity and representation usage on mobile hardware.
• Benchmark much Apple Silicon Macs, particularly the guidelines 16 GB M4 Mac mini and other 8 GB models.

The experiments that shaped TurboFieldfare explain the largest wins, the plausible ideas that failed, and the early results that reversed nether stronger validation. The detailed experiment record keeps each 103 audited entries arsenic optional evidence.

Useful introduction points:

• Local OpenAI-compatible server
• System design
• Benchmarks
• The experiments that shaped TurboFieldfare
• Experiment inventory and summaries
• Implementation references

TurboFieldfare's root and archiving are licensed nether the Apache License 2.0.

Model weights are not included. The installer downloads them separately from the pinned Hugging Face checkpoint, and the weights stay governed by their source terms. See THIRD_PARTY_NOTICES.md for the model and Swift package licence review.

TurboFieldfare is an independent investigation project. It is not affiliated with, sponsored by, aliases endorsed by Google.

Thanks for checking retired this project!

My sanction is Andrey Mikhaylov. You tin find maine on LinkedIn. I americium the writer of TurboFieldfare and an iOS and Metal engineer. Most of my work is pinch images, video, and on-device AI.

I dedicate this task to my wife, Sasha, the astir supportive personification I know. She stands by maine moreover done the hardest times. She loves wildlife, goes birdwatching, and volunteers pinch our section birding community. Because of her, I person besides grown person to birds and nature.

TurboFieldfare is named aft the fieldfare, a personnel of the thrush family and my favourite bird. It is not the astir noticeable aliases brightly coloured bird, but it decidedly has a characteristic and unsocial features of its own. I deliberation the same is existent of this project: it whitethorn not beryllium the astir practical, but I built it with my favourite tools, particularly Metal, successful my favourite field, on-device ML inference. It decidedly has its ain characteristic and unsocial features.

Next clip you are outside, touch the writer and perceive to the birds. Sometimes it is the astir beautiful point you tin do. And if you can, support your local wildlife community. They do important work.

Thank you!