Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)

Hacker News by 1 min read 46x views
Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)

Share Post

Qwen 3.8- 27B on iPhone 18 Apple Neural Engine. Added outline example , dflash-2 at ~ 2.8 bpw ( blended 2-4 vector quantization alongside small autumn in acceptance rate) Base is @PrismML Bonsai-2, 1.585 bpw requantized to 1.5 bpw ( VQ 6 bits per 4 weights) Decoding speed is 15-60 tps.

Other Article Hacker News
↑
Close Right Ads
Close Left Ads