LookAtMyQuant is a quantization compiler. It measures how much all stored part of a example hurts the output whenever compressed. It afterward picks a code for all part so the entire example fits a size prosperity alongside the smallest damage.
On Gemma 4 E2B, its GGUF has 12× lesser KL than Q4_K_M at the identical document size. The method and all outcome are on the site.
The code is under the MIT license.
LookAtMyQuant 2 takes Qwen3.5-4B to 1.23 GB, 2.3 bits per weight. The document solves 330 of 540 exact-answer math problems. Prism ML's Bonsai 2 Ternary 4B solves 328 at 1.20 GB, and the first solves 359. Its KL from the first on held-out content is 0.45 nats, against 0.59 for Bonsai.
The formula is post-training quantization alongside trellis codes, calibrated on the model's own worked math solutions alongside numeral tokens valued 10×, and alongside bits allocated per row by Fisher information. 4,000 steps of distillation follow, in which all tier is re-quantized formerly from the first weights. The leaf has the findings, the complete formula and its cost, concerning $34 of GPU time.
The recipe's code is here, and the next division runs it. The 1.23 GB document itself is the lm2-qwen35-4b-1.2gb release, packed for the Apple Silicon runtime.
The identical formula takes Qwen3.8-27B from 53.7 GB to 7.2 GB, 2.1 bits per weight, calibrated and distilled on English prose and chat, code and math. The file solves 401 of the 540 math problems, as many as the original, and Unsloth's UD-IQ2_XXS GGUF of the identical size solves 348. It additionally scores complete that GGUF on HumanEval (73.8% against 67.1%), MMLU (78.8% against 75.0%) and GSM8K (94.5% against 92.7%).
Beside it sits a 4-bit DSpine drafter (1.02 GB, converted from tencent/Qwen3.8-27B-DSpine-b16 alongside runtime_mlx/convert_dspine.py and packed alongside pack_drafter.py), which approximately doubles decode speed on math and code on Apple silicon.
The identical compiler and proximal fine-tuning (SQYFR), extended to a 26.4B bodily / 5.4B energetic 3×-looped MoE (L6–17 looped 3× = 48 logical visits, 128 experts top-8), compresses thinkingmachines/Inkling-2-Mini from 18.89 GB of content weights to 5.38 GB (1.63 bits/weight) and 8.87 GB (2.69 bits/weight) using 4 KB SRAM hashed block-vector trellis codes (ht2/ht3), row trellis codes (rtr1/rtr4), and a tr2 flat on the rear lm_head bucket (head.7).
Evaluated on the identical BF16 teacher, openai/gsm8k (test[:200], greedy, reasoning_effort="none"), all 164 executable openai/openai_humaneval problems, 4 Vision VQA tests (with the untouched 637 MB BF16 imagination tower), and 16+13-language TextMix windows:
| Model | Text GB | free_KL (en/chat/code) | all13_KL | GSM8K (200) | HumanEval (164) | Pooled vs Teacher |
|---|---|---|---|---|---|---|
| Original BF16 Teacher | 18.89 | 0.0000 | 0.0000 | 147 / 200 (73.5%) | 66 / 164 (40.2%) | 100.0% |
| LookAtMyQuant 5.4 GB (SQYFR) | 5.38 | 0.0895 | 0.2871 | 134 / 200 (67.0%) | 60 / 164 (36.6%) | 91.2% |
| Unsloth UD-IQ1_S | 7.76 | 0.0808 | 0.1236 | 127 / 200 (63.5%) | 46 / 164 (28.0%) | 81.6% |
| LookAtMyQuant 8.9 GB (SQYFR) | 8.87 | 0.0208 | 0.0571 | 148 / 200 (74.0%) | 58 / 164 (35.4%) | 96.8% |
| Unsloth UD-Q2_K_XL | 9.14 | 0.0486 | 0.0811 | 132 / 200 (66.0%) | 49 / 164 (29.9%) | 85.3% |
| Unsloth UD-IQ3_XXS | 11.66 | 0.0189 | 0.0278 | 148 / 200 (74.0%) | 68 / 164 (41.5%) | 101.4% |
At 5.38 GB, Inkling-2-Mini-LookAtMyQuant-5.4GB outscores Unsloth's 9.14 GB UD-Q2_K_XL on the two GSM8K and HumanEval at 41% smaller size (up from 76/200 and 24/164 untuned). At 8.87 GB, Inkling-2-Mini-LookAtMyQuant-8.9GB matches the BF16 first on GSM8K and cuts KL 2.34× on English/chat/code and 29.6% throughout all 13 languages below UD-Q2_K_XL. Both run immediately through lamq and runtime_mlx alongside tradition Apple Silicon Metal kernels, optionally paired alongside the 178.9 MB 4-bit DSpark drafter: on M5 Max, 5.4 GB decodes at 73.2 tok/s serial and 125.2 tok/s median (141.8 tok/s math, 129.0 tok/s reasoning, 121.4 tok/s code, 102.3 tok/s prose; 1,037 tok/s prefill at T=512, 5.83 GB energetic RAM), and 8.9 GB decodes at 63.1 tok/s serial and 101.3 tok/s median (133.9 tok/s reasoning, 118.9 tok/s math; 838 tok/s prefill at T=512, 9.32 GB energetic RAM).
On Apple silicon, instal the runtime bundle immediately (this installs mlx, mlx-lm, numpy, and huggingface-hub without the PyTorch compiler dependencies):
pip instal git+https://github.com/ShinMegamiBoson/lookatmyquant.git
From a local clone, pip instal -e . installs the lamq command, or run python -m runtime_mlx without installing.
Command line. Pass either a local example directory or a Hugging Face repo ID:
lamq create shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \ -p "What is 17 times 23? Show the steps." lamq create shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \ --drafter shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter \ -p "Write a Python function that merges overlapping intervals." --stats lamq conversation shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \ --drafter shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter lamq create shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-5.4GB \ --drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter \ -p "What is 17 times 23? Show the steps." --stats
lamq create says stdin whenever -p is omitted, supports --raw, --system, --thinking, --temp, --mode linear|chat-fork|copy-wide, --no-adaptive, and --json. lamq info <model> inspects a packed model's format and dimensions without loading weights.
Python integration. runtime_mlx.load returns an (model, tokenizer) brace compatible alongside mlx_lm.generate and mlx_lm.stream_generate (auto-detecting opgraph-packed-1 models specified as Qwen3.5-4B / Qwen3.8-27B and lamq.json exports specified as Gemma 4 E2B):
from mlx_lm import generate from runtime_mlx import load model, tok = load("shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB") prompt = tok.apply_chat_template( [{"role": "user", "content": "What is 17 times 23? Show the steps."}], add_generation_prompt=True, enable_thinking=False, ) print(generate(model, tok, prompt, max_tokens=200))
For streaming, multi-turn chat, or speculative decoding through one interface, use Runtime:
from runtime_mlx import Runtime rt = Runtime.load( "shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB", drafter="shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter", ) for chunk in rt.stream("Solve 3x + 4y = 31 and 5x - 2y = 17.", max_tokens=256): print(chunk.text, end="", flush=True)
Install the requirements and download the example and WikiText-2. The code says SmolLM2-360M's config whenever it starts, so do this first.
pip instal -r requirements.txt python -c "from huggingface_hub import snapshot_download as d; d('HuggingFaceTB/SmolLM2-360M')" python -c "from datasets import load_dataset as l; l('Salesforce/wikitext', 'wikitext-2-raw-v1')"
Measure. This compresses one part at a time, leaves the remainder exact, and records the KL to the first model. The complete set for this model, 2,312 measurements, took concerning 50 minutes on an L40S.
python compiler.py measure --model HuggingFaceTB/SmolLM2-360M --part linear --layers 0-3 --out results/batch/my_measurements.json
Allocate. This picks one code per piece, alongside the smallest summed KL inner the budget. It needs no GPU. This command uses the SmolLM2-360M measurements in this repository, alongside a prosperity of 3.044 bits per weight.
python compiler.py allocate --sens 'results/batch/ccSens*.json' --target disk --items linear --bits 3.044 --out results/compiler/plans/my_plan.jsonThe scheme it writes has the identical widths as results/compiler/plans/sm_lin_3044.json.
Build and score. This quantizes the example to the plan, one matrix at a time. It afterward reports perplexity and KL to the first on WikiText-2 test.
python quantize.py --program scheme --model HuggingFaceTB/SmolLM2-360M --plan results/compiler/plans/my_plan.json --out results/my_run.json
Our run of that scheme is in results/batch/cc_sm_lin_3044.json. It reached perplexity 16.64 and KL 0.0926, against an first perplexity of 15.31. Add --layers 1 --calib 8 --eval 2 for a quick inspect that everything works.
Each stage is a named job; python run_batch.py --list prints its command. The jobs need Qwen3.5-4B in models/Qwen3.5-4B on the quantity and the content mix (modal run --detach modal_fetch.py --text). Every document they create goes to lm2/ on the volume.
modal run --detach modal_app.py --jobs lm2Text modal run --detach modal_app.py --jobs lm2Calibration,lm2Solutions modal run --detach modal_app.py --jobs lm2RowImportance modal run --detach modal_app.py --jobs lm2Build modal run --detach modal_app.py --jobs lm2Finetune modal run --detach modal_app.py --jobs lm2Math,lm2Math216,lm2TextKL
| Job | Script | What it does | Our run |
|---|---|---|---|
| lm2Text | other_text.py | The first answers science, code, education and tool prompts, and MATH and GSM8K training problems. This additionally downloads MATH | |
| lm2Calibration | math_calibration.py | The calibration set: the original's greedy solutions to 384 MATH training problems, in 2,048-token windows | 15 minutes, H100 |
| lm2Solutions | checked_solutions.py | The math content the fine-tuning distils on: solutions to MATH problems and to math_bench.py problems from another seed, alongside incorrect calculations corrected and lone correct answers kept | 33 minutes, H100 |
| lm2RowImportance | row_importance.py | Each output row's Fisher data at digit-predicting positions | 1 minute, H100 |
| lm2Build | quantize.py | Post-training quantization: trellis codes alongside rows, numeral tokens valued 10× in the Hessians, QEP at α = 0.5, bits per row from the row plan, e-means codebooks | 37 minutes, H100 |
| lm2Finetune | finetune.py | 4,000 steps of distillation on latent weights pulled toward their codes, alongside all tier re-quantized formerly from the first weights | 2.7 hours, B200 |
| lm2Math, lm2Math216 | math_eval.py | The 540-problem benchmark and a second set of 216, for the original, Bonsai, the build and the fine-tuned build | 20 minutes a model, H100 |
| lm2TextKL | text_kl.py | KL from the first on WikiText-2 and on the content mix's 13 sources | 1 infinitesimal a model, H100 |
lm2BuildSmoke, lm2FinetuneSmoke and lm2MathSmoke run the identical code on two layers, 60 steps and 36 problems, in a few minutes each.
The row plan. lm2Build says results/compiler/rows/q35_rowplan_1p2GB.json, which gives 1 to 4 bits to all of 1,520 row groups. row_allocation.py made it. Its build-control command builds and packs a authority alongside a fixed divided of rows, prepare-rows allocates rows inner the control's packed bytes alongside rowgroup_alloc.py, and build-rows builds under the scheme and checks the packed size against it.
The codebooks. The four e-means codebooks the build decodes alongside are in cache/codes/. --trellis-fit lloyd gives the before ones, and a scheme made before the alter keeps them unless the emblem says otherwise.
Recent equal-byte formula improvements. Two zero-byte additions are accessible in the compiler and fine-tuning scripts:
- Softmax-Fisher token array ranking (--embed-order sfisher-mix in compiler.py measure and quantize.py). Ranks tied embedding/output buckets by a 50/50 blend of output Softmax-Fisher E[p_v (1 - p_v)] and corpus frequence (and untied output heads by Softmax-Fisher alone).
- Mid-training equal-byte row reallocation (--realloc-at 1000 in finetune.py). Paired alongside --rebuild-at 1000, measures all layer's continuous-vs-stored proximal KL gap KL(all P) - KL(layer = W) to change bits toward profound unabsorbed layers during keeping exact document size (--realloc-max-delta 1). Combined alongside sfisher-mix on Qwen3.5-4B at identical 1.23 GB (2.297 bpp), WikiText-2 KL cut from 0.425 to 0.387 (−9.0%) and 13-source KL from 0.447 to 0.378 (−15.5%, improving all 13 sources).
Where these jobs differ from our run. Our GSM8K solutions were written earlier, alongside thinking continually off. Our content KL on 13 sources used 52 windows picked from the content mix by a distinct tool; --sources mix takes the mix's own evaluation windows, on which our document scores 0.44 anywhere the leaf reports 0.45.
run_batch.py names all job, and modal_app.py runs them. modal_fetch.py downloads models and the content mix used for calibration and evaluation.
modal run modal_app.py::setup modal run --detach modal_fetch.py --text modal run --detach modal_fetch.py --models google/gemma-4-E2B-it modal run --detach modal_app.py --jobs ccMG4E2Noise,ccMG4E2Spot,ccMG4E2EmbKv
The runs rearward the location used these machines, all alongside 8 CPU cores:
| Model | GPU | Time |
|---|---|---|
| SmolLM2-360M | L40S | About 50 minutes to measure, in 9 parallel jobs |
| Gemma 4 E2B, measuring | H100 | 7–26 minutes per job, in 4 jobs |
| Gemma 4 E2B, construction a plan | H100 | 28–58 minutes per plan |
gguf_compiler.py makes plans out of llama.cpp's own tensor types, so the outcome runs in llama.cpp as it is. public_eval.py scores community quantizations the identical way as our plans. Both run inner the depiction in modal_gguf.py, which builds llama.cpp alongside CUDA.
quantize.py --export <folder> writes a scheme in the format runtime_mlx/ reads. A scheme alongside trellis codes additionally needs --trellis-rows 16,8, which runs all trellis way alongside a row of its matrix, so that decoding an admission says lone that row. On Apple silicon, pip instal . installs mlx and mlx-lm; requirements.txt holds the PyTorch compiler dependencies.
On Gemma 4 E2B at 2.5 GB, the trellis scheme has 26% lesser KL than the Lloyd–Max scheme and decodes concerning 22% slower. runtime_mlx/test_trellis_gen.py and runtime_mlx/test_trellis.py inspect the trellis kernels against the matrices the quantizer built.
python runtime_mlx/bench.py lamq path/to/export
runtime_mlx/fast_q35.py decodes the packed 1.2 GB Qwen3.5-4B document one token at a time, direct from its trellis codes. runtime_mlx/drafter_q35.py adds speculative decoding. A small drafter proposes the seven tokens following the model's latest one in a sole pass. The example afterward checks all eight positions in one continue complete its coded weights (runtime_mlx/block_q35.py) and keeps the proposals that equivalent its own choice, so the content is exactly its own greedy text.
On an M5 Max, complete 30 held-out prompts of up to 256 tokens, generation alongside the drafter was 2.0 times as fast: 176 tokens a second against 88, timed in alternation in one procedure on 2026-11-11. A inspect kept 4.3 tokens on average. Over eight mundane prompts it was 1.5 times as fast. The acquire follows how predictable the content is: 2.5 times on algebra, 2.0 on code, 1.2 on conversation and 1.05 on a story, and among the 30 held-out prompts ran slower alongside the drafter. The content matched plain greedy decoding on all prompt. These are speeds following the archetypal token; study the immediate takes concerning 0.1 s either way.
Two afterward changes are in this code. Each was measured against the type before it on 40 paired generations: proposing from the 96,033 most average tokens was 6.0% faster, and picking the winning token inner the packed merchandise was 5.3% faster. Linear manner additionally enables self-calibrating adaptive fallback by default (adaptive=True; continue --no-adaptive to disable): whenever rolling acceptance drops below the measured break-even proportion of drafted-check latency to serial-step latency (accounting for cache flush and drafter environment replay), it pauses example for abbreviated exact serial bursts and resumes formerly acceptance recovers, preserving bit-exact greedy parity during avoiding slowdowns on low-acceptance prose.
The drafter is a DSpine model, trained alongside SpecForge from the published DFlash drafter for Qwen3.5-4B on the 1.2 GB file's own greedy answers. modal_drafter.py runs the training as one Modal job, and it disbursal $42. On 300 held-out prompts the trained drafter gets 4.65 tokens per check, against 4.30 for the inventory DFlash drafter.
pack_drafter.py stores the drafter at 4 bits a importance in the build's own code, alongside no calibration data: 464 MB against 1.83 GB at 16 bits. On 60 held-out prompts the 4-bit drafter kept 4.35 tokens per inspect against 4.38, and the content was the same. drafter_q35.py loads either form and writes the packed one out compact as it loads. The 4-bit drafter is the lm2-drafter-4bit release. It runs beside the packed 1.2 GB file, the lm2-qwen35-4b-1.2gb release.
gh publish download lm2-qwen35-4b-1.2gb --repo ShinMegamiBoson/lookatmyquant -D q35-1.2gb gh publish download lm2-drafter-4bit --repo ShinMegamiBoson/lookatmyquant -D drafter-4bit python runtime_mlx/bench_drafted.py q35-1.2gb drafter-4bit
The benchmark defaults to linear verification. Use --mode chat-fork to try a second applicant at an ambiguous first position, or --mode copy-wide for known adjacent copies. Copy manner uses the origin cursor and tunes the gate/up and downward projections for 16-slot checks. Both modes verify all emitted tokens alongside the packed target. To benchmark the conversation fork, run:
python runtime_mlx/bench_drafted.py q35-1.2gb drafter-4bit --mode chat-fork --prompts chat
For a copy benchmark, continue --mode copy-wide --heldout path/to/copy-prompts.json. The JSON entries transport source, prompt_len, and input_ids from the packed model's tokenizer.
The checks, and the packer for a drafter of your own:
python runtime_mlx/test_fast_q35.py LAMQ_Q35_PACKED=path/to/packed python runtime_mlx/test_block_q35.py python runtime_mlx/check_drafter.py path/to/packed path/to/drafter python runtime_mlx/bench_drafted.py path/to/packed path/to/drafter --heldout path/to/drafter/heldout/heldout.json --take 30 python pack_drafter.py path/to/drafter path/to/drafter-4bit
runtime_mlx/pack_dense.py writes the packed document from the fine-tuned build's compact weights, and runtime_mlx/modal_pack.py runs it on Modal. check_drafter.py compares the drafter in MLX alongside SpecForge's own example on citation blocks that drafter/reference.py writes.
| Path | What it holds |
|---|---|
| compiler.py | Measure and allocate |
| compiler_arch.py | How the compiler finds a model's layers, tables and stages, and the content mix |
| quantize.py | Builds a program or a plan, afterward scores it |
| row_importance.py, rowgroup_alloc.py, row_allocation.py, compiler_artifact.py | Bits per row: all row's importance, the allocation, the builds it compares, and the packed format that prices them |
| math_calibration.py, checked_solutions.py, other_text.py, calc_check.py, token_kinds.py | The content the formula calibrates and fine-tunes on, the checker for stated calculations, and the benevolent of token all stance predicts |
| finetune.py, finetune_lib.py, fast_delta.py | Fine-tuning that ends on the codes, and a faster linear-attention regulation for Qwen3.5 |
| math_bench.py, math_eval.py, text_kl.py | The exact-answer math benchmark, its runner for the Qwen3.5 family and Bonsai, and KL from the first on text |
| gguf_compiler.py, public_eval.py, eval_tasks.py | The llama.cpp route, community quantizations and project benchmarks |
| pyproject.toml, runtime_mlx/runtime.py, runtime_mlx/cli.py | Installable Apple Silicon runtime package, unified load/Runtime Python API, and lamq CLI |
| lamq_export.py, runtime_mlx/ | The export format and the Apple Silicon runtime |
| runtime_mlx/fast_q35.py, block_q35.py, block_argmax_q35.py, pack_dense.py, modal_pack.py | The runtime for packed Qwen3.5-4B and Qwen3.8-27B files: one token at a time, a obstacle of 8 or 16 tokens in one pass, and the packer |
| runtime_mlx/inkling/, runtime_mlx/test_inkling.py | Apple Silicon MLX runtime, 4 KB SRAM ht2/ht3 Metal kernels, and 4-bit DSpark block-8 speculative drafter for packed Inkling-2-Mini models |
| runtime_mlx/drafter_q35.py, draft_vocab_q35.py, convert_dspine.py, check_drafter.py, bench_drafted.py, drafter/, modal_drafter.py, pack_drafter.py | Speculative decoding alongside a DSpine drafter, Tencent DSpine conversion, drafter training, and the drafter stored at 4 bits |
| opgraph/ | The grammar: a example as a tensor network, and the rewrites and codes that act on it |
| run_completeness.py, make_report.py | The inspect that nine published methods are reproduced exactly |
| analyze_compiler.py | The tables of scheme results |
| run_batch.py, modal_app.py, modal_gguf.py, modal_fetch.py | Named jobs and the Modal runners |
| results/compiler/plans/, results/compiler/rows/ | Every plan, and the row scheme for Qwen3.5-4B at 1.2 GB |
| results/protocols/ | The second math set of 216 problems, and the hashes of all inquiry kept out of training text |
| cache/codes/ | The e-means trellis codebooks at 1 to 4 bits |
| results/batch/, results/gguf/ | The measurements and scheme runs rearward the site |
| results/COMPILER.md, results/COMPLETENESS.md | Write-ups of the compiler on SmolLM2-360M and Qwen3.5-4B, and of the reproduction check |
| site/ | The website. Run python3 -m http.server 8000 inner it, afterward sojourn http://localhost:8000 |
| site2/ | The LookAtMyQuant 2 page. It runs the identical way |
| tests/ | Tests of the grammar and the codes |
Each test document runs on its own from the repository root, and all seven complete in under a minute.
python tests/test_frame.py python tests/test_grammar.py python tests/test_codes.py python tests/test_elements.py python tests/test_nested.py python tests/test_rowgroup_alloc.py python tests/test_reallocation.py
runtime_mlx/test_core.py checks the runtime against the quantizer's math. Run python runtime_mlx/make_ref.py first, which writes 1.6 GB of citation files.
runtime_mlx/test_fast_q35.py and runtime_mlx/test_block_q35.py inspect the Qwen3.5 kernels against NumPy products of independently unpacked codes, alongside no model. When LAMQ_Q35_PACKED names a packed file, the second additionally checks a obstacle of tokens against decoding them one at a time. When LAMQ_Q38_PACKED names the packed Qwen3.8-27B file, it runs the identical checks on a example whose output caput is stored distinct from its embedding.
- Model weights. Everything downloads from Hugging Face. The Qwen3.5-4B jobs anticipate that example in models/Qwen3.5-4B on the volume.
- Benchmark copies. eval_tasks.py downloads its benchmarks. On an offline device it says exported copies from data/bench/, which are not here. The Modal images run offline, so cache the benchmarks on the quantity first.
- The manuscript that assembled site/data.json. It was a set of one-off steps. The origin numbers are all in results/.
- LookAtMyQuant 2's compact weights and outcome files. The 1.2 GB document is a publish in the packed format. The compact weights the benchmarks ran on are not published, and neither are the per-problem results rearward site2/data.json or the manuscript that writes the page's numbers from them.
- The drafter at 16 bits. The drafter as trained is 1.8 GB and is not published. The 4-bit one is a publish file, and modal_drafter.py trains it.