I tally a section LLM server connected my M4 Pro Mac mini pinch 48 GB of RAM. It handles everything from my Hermes supplier backend to speedy chat queries connected my phone. The full point takes astir 30 minutes to group up.
Here is the stack:
- Qwen3.6-35B-A3B-OptiQ-4bit: my main exemplary for thing that needs reasoning aliases depth
- Gemma-4-E4B-it-OptiQ-4bit: lightweight exemplary for elemental chats, formatting, and different regular tasks
- oMLX: the conclusion server
- Tailscale: tailnet connecting the Mac mini, my iPhone, and my MacBook
Hermes runs arsenic the supplier backend connected the Mac mini, pinch my MacBook moving the desktop customer and my telephone moving Telegram. For non-Hermes usage I usage Apollo connected iOS for speedy chats (reads for illustration Claude, bully for throwaway questions), Pi arsenic my coding supplier (I already wrote astir that setup), and Raycast AI connected my Mac for random things.
Why bother?
The main logic to tally local: unreality APIs are rented land. They tin alteration their pricing, deed your usage limits, aliases switch the exemplary being served down the scenes whenever they consciousness for illustration it. I was regularly maxing retired 2 $200/month subscriptions and it felt for illustration I was getting different things from them astatine different points. Sometimes a exemplary was fine, sometimes it degraded pinch nary notice.
Data privateness is different issue. You do not cognize what these companies do pinch your information erstwhile they person it. They mightiness limit really it gets used, they mightiness waste it, they mightiness expose it. Either way, it creates an operational information risk. If you activity pinch delicate code, customer data, aliases proprietary workflows, sending it to a third-party API is simply a determination you make erstwhile and cannot undo.
Then location is AI sovereignty. I person been watching really the US authorities has constricted the rollout of various models. That tin hap astatine immoderate constituent from immoderate government, for immoderate reason, and you person nary power complete it. If your workflow depends connected a unreality exemplary that gets restricted, you person to extremity aliases scramble. The only measurement to debar that is to ain your compute.
Other applicable advantages:
- Cost predictability. APIs are variable. Your usage spikes and your measure follows. With section hardware, the costs is the hardware acquisition positive electricity. Flat. After that, each conclusion is free.
- Latency. No web roundtrip intends faster responses for mundane tasks. The M4 Pro’s media motor handles conclusion astatine speeds that consciousness instant for astir prompts.
- Offline capability. No internet, still works. For supplier workflows that tally successful the background, this matters much than it sounds.
- No complaint limits. API providers throttle you erstwhile you deed usage thresholds. Your ain instrumentality does not attraction really overmuch you run.
How I really usage it
The Mac mini is ever on. It sits connected my table and I hardly announcement it isolated from erstwhile I request it.
Hermes runs connected the Mac mini arsenic well, utilizing a section exemplary connected the aforesaid machine. I entree my supplier done Telegram (on my phone) and the Hermes desktop app connected my MacBook. The Hermes desktop app acts arsenic a ‘shell’ and connects to a Hermes backend connected different instrumentality (in this lawsuit the Mac mini). This intends I stock a backend, speech history, and skillset crossed each my devices.
Then location is everything else:
- Apollo connected iOS for speedy throwaway chats. I want thing that sounds for illustration Claude but does not require an API cardinal aliases a subscription. Connect Apollo to http://[mac-mini-tailnet-url]/v1 and you are done. Good for “rewrite this paragraph” aliases “what does this correction mean” type questions.
- Raycast besides connected my Mac for random things I don’t want to instal thing for.
- Pi for coding. Already wrote astir that setup.
The constituent is not to switch API-based models. It is to grip the 80% of requests that do not request GPT-5 aliases Claude Opus. And erstwhile I do request those, they are already available. Local conscionable covers much of my day-to-day for free.
The exemplary breakdown
Running a ample exemplary locally comes down to 1 thing: really overmuch RAM it really needs successful memory. Most group look astatine the parameter count and get the incorrect idea, because the quality betwixt dense and mixture-of-experts (MoE) models matters a batch connected user hardware.
Here is really to publication the identifier:
Qwen3.6-35B-A3B-OptiQ-4bit
- Qwen3.6: exemplary family and version
- 35B: full parameters crossed each experts
- A3B: progressive parameters per token (3 billion, not 35)
- OptiQ-4bit: mixed-precision quantization (4-bit mostly, 8-bit connected delicate layers)
gemma-4-e4b-it-4bit
- gemma-4: Google’s Gemma 4 family
- e4b: encoding size, astir 4 cardinal parameters total
- it: instruction-tuned
- 4bit: azygous 4-bit quantization
The cardinal quality is the A3B part. A dense 27B exemplary has 27 cardinal parameters loaded successful RAM astatine each times, for each azygous token. An MoE exemplary for illustration the Qwen3.6-35B-A3B has 35 cardinal full parameters dispersed crossed 256 experts, but only astir 3 cardinal are really activated per token. The different 32 cardinal beryllium successful RAM doing nothing.
On my 48GB Mac mini, the Qwen3.6-35B-A3B successful 4-bit takes astir 20GB of RAM. That leaves 28GB for discourse windows, the operating system, and everything other moving connected the machine. The Gemma-4-E4B is astir 2.4GB. Small capable to support astir for elemental tasks wherever utilizing the afloat 20GB exemplary is overkill.
My friend’s MacBook Air had 16GB total. A dense 27B successful 4-bit needs astir 14GB. That is virtually everything the instrumentality has, minus room for the OS. So it useful for a moment, and past erstwhile it does not, it swaps to SSD and becomes painful.
MoE changes this. The 35B exemplary successful my identifier would fresh connected the aforesaid MacBook because only 3B of weights are really progressive per token, which intends the GPU/Media Memory footprint is much for illustration what a 6B dense exemplary would need. The 35 cardinal full parameter weights each beryllium successful unified memory.
How to cheque if a exemplary will activity connected your hardware:
- Look astatine the quantized record size first. A 4-bit exemplary is astir the number of parameters successful gigabytes (35B params ≈ 17-20GB depending connected the quantization method).
- Subtract your OS overhead. macOS takes astir 6-8GB connected Apple Silicon.
- Leave room for discourse windows. Every fewer 1000 tokens adds megabytes to the KV cache. Plan for 8-16GB overhead if you expect agelong conversations.
- For MoE models, the full parameter count is misleading. Look for the “active parameters” fig to understand existent conclusion memory.
- If your exemplary positive discourse still fits wrong your disposable unified representation pinch a 10-15% buffer, you are good. Anything person to afloat will switch to SSD.
Swapping models is easy
This is the portion cipher talks about. You tin switch retired your section models each fewer weeks arsenic caller ones drop. It is virtually a download and a restart.
The workflow:
- Download the caller exemplary into ~/models/
- oMLX auto-discovers it from the exemplary directory
- Pick it successful the oMLX app aliases restart the server
- Done
The oMLX admin dashboard has a built-in HuggingFace exemplary browser. Find a model, click download. Change the exemplary successful Hermes, Pi, Raycast, and Apollo, and I americium each done.
A batch of this tin beryllium done via CLI too, truthful I tin SSH into the Mac mini from immoderate of my devices.
Why this matters: the spread betwixt section models and API models is closing fast. What was “meh” value a twelvemonth agone is competitory for astir real-world tasks now. Coding, reasoning, instrumentality usage are wherever it matters. And the 4-bit quantization from OptiQ keeps value amazingly high. The 35B-A3B astatine 4-bit only loses astir 1-2 points connected astir benchmarks compared to BF16 (16-bit floating point, the uncompressed baseline). That is an acceptable tradeoff for 48GB of representation usage alternatively of 70.
The network
Tailscale creates a mesh betwixt each my devices. Mac mini, iPhone, MacBook, each connected the aforesaid backstage network. Nothing exposed to the nationalist internet.
The oMLX server listens connected larboard 8000. Any instrumentality connected the tailnet tin connect. Raycast, Apollo iOS, Hermes desktop connected my MacBook, they each deed the aforesaid endpoint. No configuration drift betwixt devices.
oMLX’s KV cache persistence besides matters connected the tailnet setup. Coding agents many times circle backmost done earlier discourse successful a session. oMLX caches each artifact to SSD, truthful erstwhile the supplier returns to a erstwhile prefix, it is restored from disk successful milliseconds alternatively of being recomputed. That makes the section setup really applicable for supplier work, which is wherever Hermes lives.
Closing out
Local models connected Apple Silicon are not a broadside research anymore. The M4 Pro Mac mini handles it without breaking a sweat, the models are bully capable for astir tasks, and you tin switch them retired whenever you want. You are not paying per token. You are not routing delicate information done third-party endpoints. And erstwhile a amended exemplary drops adjacent week, you tin effort it pinch hardly immoderate effort for the costs of immoderate difficult thrust space.
I person already ordered a 128GB M5 Max Mac Studio to beryllium delivered later this year, but I americium incredibly happy pinch the capacity of this M4 Pro Mac mini truthful far. If you person different Apple Silicon devices, effort it retired - you whitethorn request to alteration the exemplary based connected your specs, but the wide setup holds.
English (US) · 
Indonesian (ID) · 
