Learning to use local AI is exciting, overwhelming, and frustrating

I’m embarking on a journey I’ve so far been hesitant to go on: really using AI on the regular. Giving my most individual data to haze services has been a important clasp up for me, but it’s increasingly imaginable to run mighty models locally to see what they’re capable of. Is this tech so helpful that it’s value throwing gobs of prosperity at a device or a laptop alongside many of overpriced RAM? I’ll try to assistance you discover out. I’m not an AI expert, it’s going to be a journey for me, but perchance I can assistance you see if that journey is value taking — and I’ll be posting my way through it.

A big part of Apple’s throw for its new Mac desktops is how fine they are for local AI, and there’s an complete row of new RTX Spark Windows machines landing imminently alongside up to 128GB of RAM aimed at using agentic AI. Local AI and agents are just, like, so hot correct now. But, in all seriousness, the idea of having a small gofer living solely in the box on my array that answers lone to me and not overlords from OpenAI, Google, Microsoft, or Anthropic hits a bit distinct than trusting haze services or subscribing to a chatbot. Don’t get me wrong, there’s motionless no way I’m gonna have it compose for me, edit for me, or create ugly fliers. But now, alongside several privacy on my flank perchance I can eventually think of helpful occupied activity to “trust” it with.

So as part of my ongoing evaluation of the M5 Ultra Mac Studio, as fine as experimenting on another Mac and Windows systems, I’ve installed several local LLMs and I’m commencement to tinker. Here’s how things are going.

I started by installing the open-source Hermes Agent on the Mac Studio. Hermes is a self-hosted AI delegate desktop app that plant on macOS, Windows, and Linux. It’s entirely liberated to use if you rod to local LLMs. Since this M5 Ultra Mac Studio has 256GB of unified memory, I could run fair concerning any example I want. This is how things quickly get overwhelming. There are more models out there than I can count, and several have specialized uses. Generally, models alongside many billions of parameters are capable to do additional than smaller ones alongside lone a few billion. And since we don’t have to concern concerning paying for tokens whenever operating a example locally, I figured I’d go big to start.

Using Hermes’ straightforward onboarding interface and example picker, I chose the Qwen 3.8 Flash Next example — a 125-billion indicator example that’s about 105GB in size. I additionally scheme on spending several period alongside much smaller Qwen models on devices akin an M6 Mac Mini, M5 MacBook Air, an Asus TUF Gaming A14 alongside AMD Strix Halo processor, and, eventually, RTX Spark. (Care to portion what local models and hardware you use? You cognize anywhere to depart your comments. 🫡)

It wasn’t lengthy to get up and operating alongside Hermes and Qwen, and equal create them controllable via my phone by using a Telegram bot. But then, I was presented alongside my customary nemesis whenever it comes to using AI: the content box.

“What do I do alongside this thing?”

First up, to get things going, I followed several helpful YouTube advice and had Hermes create me a regular dawn briefing — to fair commencement small alongside a cron job and justify things are working. It scans my email and calendar, tells me item value my contiguous attention, and gives me a abbreviated climate report. It’s not all that useful, and you evidently don’t need a $12,000 device to build it, but it’s item I can create complete time. The concise kept failing to create at first, but afterward I eventually figured out that macOS couldn’t be idle whenever the job fired up at 7:30am. Duh! It plant akin clockwork now, but I have to think of additional info for it to drag for me to create the results additional helpful.

A additional helpful project was reorganizing my Steam library. I have complete 400 games on Steam, and I lately started categorizing them for whenever I desire to perform item new on a whim. Steam doesn’t categorize your games for you, so you alternatively have to do it manually, one-by-one. But here’s anywhere an agentic lackey could fit correct in. I asked, “Can you reorganize my Steam library?” Hermes could see most of my games by fair viewing the already installed Steam client. It quickly came up alongside several institution options for me. I decided to have it arrange my games by genre, but depart several of my own specialized categories(favorites, co-op titles, gathering games, games to perform alongside my wife, etc.).

You have to provision a local AI several permissions complete your device for it to obtain actions for you, and in the case of my equivalent archive I had to enroll a Steam web API key and provision it to Hermes. Once it had that access, it lone took a few minutes for it to churn through sorting all the games into categories. And the API key was uncomplicated to revoke correct after, since I don’t need it further. The idea is for it to fair do this dense lifting for me once, and afterward I’ll document new games into the categories as I eventually buy them. It did a fine job overall, and now my Steam archive is effortlessly browsable.

Another uncomplicated item I did was data analysis. I had Hermes obtain a appearance at financial records to crunch several numbers for me, and I additionally had it create me a spec difference spreadsheet for a new laptop. These are small things, but they’re the two tasks I cannot or volition not rely alongside any haze assistance — I don’t desire to upload that delicate data to the cloud, and in the case of a new laptop, that info was under embargo. Keeping them on my device was the difference in me using this tool or not.

The bigger bit of automation I’m currently operating on is a manuscript to run a sequence of laptop benchmark tests. I run all our benchmarks manually, and tending to a laptop to run a dozen or so tests — three times over, to obtain the average — is incredibly time-consuming. I’d affection to automate at smallest several of it. Even strolling Hermes through all of the required procedures and getting it to spit out vibe coded Python scripts has been a bit of an endeavor, and it remains a activity in progress. But I’m expecting to create several affirmative headway.

It’s motionless first days of me operating alongside Hermes, and I’m expecting this is fair the tip of the iceberg on the helpful things I get it to do for me. But it’s additionally apparent that equal these mighty local models don’t create for a magical aide that can do anything. Things motionless don’t continually activity as intended — my regular briefing already broke, multiple times.

I perspective Hermes and local AI as fair a tool, a device program to run on my computer. I don’t desire a digital friend. I’m not saying “thank you” or “please,” or typing “sorry” whenever I alter my intellect on something. I’m remaining as stone-cold and indifferent whenever ordering it about as I am clicking a mouse about an functioning system.

I anticipation to remain vigilant and careful during I continue evaluation and learning alongside local AI. Let’s see anywhere it takes me.