We benchmarked 34 CPU-only SLM configurations. The biggest performance win wasn’t model choice.

Running AI locally aliases connected CPUs usually intends accepting slower responses aliases utilizing smaller, little tin models than GPU-hosted ample connection models (LLMs). 

At the aforesaid time, mini connection models (SLMs) tin tally wholly connected CPUs and are progressively tin of handling specialized tasks specified arsenic customer support, translation, archive processing, and codification generation. 

While SLMs aren’t designed to switch LLMs, they connection a practical, lower-cost measurement to deploy AI for galore real-world applications without dedicated GPU infrastructure.

With truthful galore SLMs and deployment options available, the bigger situation is knowing which ones are really worthy deploying. 

To find out, we benchmarked 34 CPU-only SLM configurations crossed 4 open-weight exemplary families to place which ones delivered the biggest capacity gains.

The research we ran

We tested 34 CPU-only SLM configurations built from 4 open-weight exemplary families. Each family included 1 aliases much parameter sizes (where M = cardinal parameters and B = cardinal parameters):

• Qwen3.5 (0.8B and 2B)
• Gemma-4 E2B
• Gemma-3n E2B
• LFM2.5 (350M and 1.2B) 

Each exemplary family was tested astatine aggregate compression levels, from afloat precision down to aggressively compressed. Every configuration ran nether the aforesaid fixed settings each time, truthful the numbers are straight comparable and reproducible.

Then, we measured the SLM capacity crossed 5 axes: 

• Reasoning accuracy, tested pinch MMLU-Pro
• Cross-language comprehension, tested pinch Belebele
• Single-turn latency, meaning time-to-first-token and decode velocity connected standalone requests
• Multi-turn latency, measuring the aforesaid 2 metrics nether a back-and-forth chat workload
• Long-context handling, tested pinch a needle-in-a-haystack setup that measures the maximum discourse magnitude a exemplary tin grip earlier timing out

Testing ran connected 2 deliberately different CPU platforms – a Google Cloud Intel Xeon (8 cores/16 threads, nary AVX-512) and a Hostinger VPS connected AMD EPYC (8 cores/8 threads, pinch AVX-512). 

The afloat 34-config expanse ran connected the Intel Xeon machine, and an 18-config shared subset ran connected some Intel Xeon and AMD EPYC CPUs. Cross-machine comparisons are drawn from the shared subset.

Is compressing an SLM impactful?

One comparison shape kept appearing passim our benchmark. Fewer-parameter models matched compressed higher-parameter models successful conclusion velocity while delivering substantially weaker reasoning performance.

These 2 Qwen exemplary configurations showed a clear example:

And it wasn’t an isolated result. Similar trade-offs appeared passim the benchmark, pinch compressed models many times outperforming their full-precision counterparts.

This is because downsizing the AI exemplary configuration removes parameters, reducing some the model’s representation usage and its reasoning capacity. Meanwhile, compression keeps the aforesaid SLM architecture intact and stores its weights much efficiently, preserving overmuch much of the model’s capacity astatine a akin latency.

Our research besides revealed that not each compression levels are arsenic beneficial. Comparing the aforesaid Qwen3.5-2B exemplary crossed different compression levels makes that clear:

In this case, the Q8 version (the “Q” denotes a compressed exemplary and “8” refers to 8-bit precision) was almost a free optimization, delivering astir double the throughput without sacrificing reasoning accuracy.

But the gains didn’t proceed indefinitely. Compressing further to 4-bit precision (Q4 variant) traded a fewer points of reasoning accuracy from 36.6% to 31.3% for further speed.

SLM compression proved to beryllium a much effective optimization strategy than downsizing, but only up to a point. Our testing results propose that moderate compression hits the saccharine spot betwixt velocity and reasoning performance.

Which SLM performed the best?

Once we plotted each 34 exemplary configurations by reasoning accuracy and decoding speed, only 5 were ne'er beaten connected some fronts astatine once.

And among those five, only 3 SLM configurations showed salient capacity insights:

• Smartest – Qwen3.5-2B (Q8) delivers the highest reasoning accuracy while maintaining interactive consequence speeds. It’s the champion prime for assistants and workflows wherever reply value matters most.
• Most balanced – Qwen3.5-0.8B (Q4) offers 1 of the champion speed-to-quality trade-offs successful the benchmark, showing a ample jump successful throughput for a comparatively mini driblet successful reasoning performance. For galore accumulation workloads, this is apt the saccharine spot.
• Fastest – LFM2.5-350M (Q4) prioritizes latency supra everything else, making it well-suited for routing, classification, extraction, and different lightweight tasks.

Interestingly, nary of the surviving tiers usage a full-precision model. Each is simply a compressed variant.

Where these champion SLMs fell short

Speed and compression aside, these models are still small language models, and that shows up astir intelligibly successful the taxable and the connection it’s asked in.

On reasoning, the spread is subject-dependent. 

These SLMs grip knowledge-and-memory tasks, for illustration biology, history, and wide wellness questions, reasonably well. They struggle much pinch problems that require step-by-step thinking, for illustration mathematics aliases business logic. 

Language usage shows a akin pattern. 

All models performed champion successful English, pinch a humble driblet successful accuracy crossed German, Spanish, French, and Portuguese. Gemma held up the astir evenly crossed each five, making it the amended prime if multilingual output matters much than squeezing retired the past fewer points of reasoning accuracy.

Does newer hardware ever mean faster SLM results?

Running the aforesaid 18 SLM configurations connected Intel Xeon and AMD EPYC servers, we expected the newer AVX-512 EPYC processor to consistently outperform the older AVX2 Xeon. 

On average, it did, delivering 1.12x higher decode throughput crossed the shared configurations. But that mean hid 2 different stories.

Uncompressed SLMs nearly doubled throughput connected newer EPYC processors, thanks to AVX-512 and higher timepiece speeds that accelerated ample matrix multiplications.

On the different hand, compressed models ran faster connected the older Xeon server because it exposed 16 processing threads, compared to only 8 connected the EPYC virtual machine. For these workloads, thread count mattered much than newer vector instructions.

Reasoning accuracy remained effectively unchanged across some CPU types. This implies that your prime of hardware will beryllium chiefly driven by costs and latency, not by output quality.

Why do immoderate models still consciousness slow to respond, moreover erstwhile they make matter quickly?

In our experiment, we measured time-to-first-token for each SLM configuration. While tokens per 2nd measures really quickly a exemplary generates text, time-to-first-token captures really agelong users hold for the first word.

A astir 1,000-token prompt, specified arsenic a retrieval-augmented procreation (RAG), agelong strategy instruction, aliases archive to summarize, was added anyplace from 2 to 15 seconds earlier the first token appeared.

The punishment grew pinch exemplary size. Qwen3.5-2B Q8 waited astir 10.5 seconds, while Gemma 3n Q8 took astir 15 seconds.

For AI applications built astir RAG, agelong strategy prompts, aliases ample discourse windows, improving throughput unsocial won’t destruct the hold earlier the exemplary originates responding. Reducing hold besides intends keeping prompts concise, caching repeated strategy prompts, and choosing smaller exemplary tiers for retrieval-heavy workloads.

Choosing the champion SLM for your project

Our benchmark showed that optimizing SLM deployments is little astir choosing the “best” exemplary and much astir choosing the correct optimization strategy.

Compression consistently delivered much worth than simply choosing a smaller model, and each configuration that survived our mixed reasoning-and-speed trim was a compressed variant.

The aforesaid shape appeared elsewhere successful the benchmark. Newer hardware only improved throughput nether circumstantial workloads, while punctual magnitude often had a bigger effect connected perceived responsiveness than decoding speed. 

Together, these findings propose that getting the astir retired of an SLM isn’t astir applying arsenic galore optimizations arsenic possible. It’s astir knowing which 1 will person the biggest impact.