Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes

Somewhere betwixt “nanoGPT toy” and “you request a investigation lab” there’s a large, under-described region wherever 1 personification pinch a fewer 1000 dollars tin train a meaningful model.

I wanted to spot connection and knowing look from random weights for myself, and to study the parts you tin only study by starting from scratch. This task was written successful the evenings, debugged connected a 5090 and vanished connected rented B200s. It was heavy inspired by Andrej Karpathy’s nanochat.

The consequence is simply a 3.8B-parameter exemplary scoring 0.384 connected CORE, trained connected 65B tokens successful 43 hours for $998.

What follows is what worked, what didn’t, and what I still don’t know.

My exemplary is larger than nanochat d32 and took akin wall-clock time. B200s were amended worth per portion of activity than H100s. But for astir the aforesaid money arsenic nanochat’s $1,000 configuration, this lands meaningfully up of it. An encouraging information constituent astir what’s reachable extracurricular a laboratory aliases a mega institution pinch millions successful compute budget. As the frontier moves, $1,000 takes you further and further.

Setup

I’ve built little-lm arsenic a config-driven model for training mini decoder-only LLMs. Every tally is afloat specified by a YAML file: model, dataset, optimizer, schedule, callbacks. Components self-register into a world registry and get resolved by name, truthful swapping an optimizer aliases a dataset is simply a one-line config change.

Good infrastructure pays for itself almost immediately. Ordinary package engineering subject (Things for illustration separation of concerns, cleanable interfaces, components you tin switch in) matters a batch successful AI work. It costs maine a small astatine the start, and a mates much times afterward to hole bad contracts aliases suboptimalities. But this clip finance pays for itself astatine the first convergence problem you encounter. I recovered that a awesome infra is the infra that almost ne'er requires you to edit codification manually. If you tin publication the config and understand precisely what happens, and location are nary hidden mechanics, it intends you person done a bully job. The pursuing study is the consequence of being capable to definitive experiments arsenic a three-line YAML diff alternatively than a branch.

The last exemplary is Llama-style: RMSNorm, RoPE, GQA (24 query heads, 8 KV heads), relu² MLPs, QK-norm, logit softcap, per-layer learnable residual scalars, and ResFormer-style worth embeddings.

Worth noting that the worth embeddings are 19% of the parameter count. 14 tables of vocab × kv_dim, 1 connected each different layer.

Results

Early experiments

Before bully runs location were galore bad ones.

I trained an 858M Llama connected FineWeb-Edu for 16.4B tokens, 5.8 days connected a azygous A100. AdamW astatine 2.5e-4, cosine decay to zero, 5% warmup, batch 256 via gradient accumulation, 2048 context.

The result: PIQA 60.45%. GPT-2 124M scores astir 63%. I had spent six days of compute to build thing worse than a exemplary 7 times smaller, from 2019. Generations were repetitive and borderline nonsensical.

The nonaccomplishment curve told the story.

1. Cosine decay to zero. The curve went wholly level aft astir 70% of the steps. The last 30% of the compute fund produced fundamentally thing arsenic the learning complaint mightiness beryllium excessively low. Linear cooldown holds a useful complaint overmuch later.
2. Peak LR excessively conservative. 2.5e-4 is debased for 858M parameters. You tin beryllium rather fierce for those mini models.
3. AdamW connected everything. Muon should beryllium meaningfully amended per-token for the matrix parameters astatine this scale. In truth this was demonstrated beautiful quickly successful ablation runs.
4. The data. FineWeb-Edu is decent. It is not the champion available.

Five changes came retired of that post-mortem. Together they are the quality betwixt the tally supra and a exemplary that thumps GPT-2 by a wide margin.

Trapezoidal LR schedule. Warmup for 5%. Hold level and decorativeness pinch linear cooldown complete the past 50% to 5% of peak. The constituent is that the exemplary keeps learning until the extremity alternatively of coasting done the tail. In the 3.8B tally the eval nonaccomplishment was still descending astatine the last step, which is precisely the behaviour the 858M tally grounded to produce.

Muon for matrix parameters, AdamW for everything else. Muon is slower per measurement (Newton-Schulz orthogonalization isn’t free, astir 25% successful a shallow-accumulation benchmark) but that costs is paid erstwhile per optimizer step: astatine 7 gradient-accumulation steps it dilutes to ~4%. Measured against full tally clip the convergence is overmuch faster overall.

ClimbMix alternatively of FineWeb-Edu. This was a tremendous jump successful convergence speed. Exactly arsenic Karpathy recovered arsenic well.

FP8 + vocab padding. FP8 training via torch._scaled_mm pinch move tensorwise scaling connected each 3 GEMMs, and padding the vocab from 50,257 to 50,304 (a aggregate of 64) truthful the tensor cores are happy. Together, +33% throughput mostly from fp8.

1024 discourse alternatively of 2048. Halving the discourse astir doubles the batch size astatine fixed memory. Throughput hardly changes per token. We are still dominated by the MLPs which is simply a bully motion we are utilizing the hardware effectively. Below we will talk the effect of the discourse magnitude connected the model.

Here is the full run:

~480,000 tokens/sec successful dependable state, which puts 57.3B tokens astatine 33 hours. The wall timepiece was 35.9h. The quality is the CORE evaluations, which took astir 15 minutes each (ten of them complete the run) and consumed 7% of the total.

Re-running this identical look astatine 2048-token discourse scored 0.3840. Almost each of that spread turned retired to beryllium immoderate tasks that were very discourse dependent.

On the GPUs themselves: 92% SM activity, 40% SM occupancy. High activity intends the SMs almost ne'er went idle. No dataloader starvation aliases web waits, which is the payoff for downloading the shards locally alternatively of streaming, which would time off america susceptible to a mini hugging look web hang. The debased occupancy is what back-to-back ample GEMMs look like: matmul kernels waste and acquisition occupancy for register-tile size connected purpose. Compute-bound and good fed, awesome awesome we are utilizing the hardware good and we tin widen each dollar we walk into a amended model.

That’s astir 1,047 TFLOP/s sustained per B200, aliases ~25% MFU against Blackwell’s dense FP8 peak. (Against the bf16 highest it sounds arsenic 50%, which is the number that matters a spot much because not moreover each the linear layers tally successful FP8.)

The distributed strategy is plain aged DistributedDataParallel. At 3.8B connected a azygous node, gradient connection was ne'er the constraint, and the sharded-optimizer machinery turned retired to beryllium unnecessary.

Increasing throughput

Renting GPUs isn’t cheap, astatine activity you often deliberation astir the value of the exemplary earlier its cost. When it’s your ain money burning, throughput matters a batch much each of a sudden.

This took existent activity connected a azygous RTX 5090, earlier I ever rented a node. Baseline 858M model, bf16, compiled: 26,144 tok/s. Final: 37,621 tok/s.

FP8 (+25%). All 3 GEMMs (1 guardant and 2 backwards) successful FP8 pinch move tensorwise scaling. Requires SM90+ but that is rather a bully throughput jump.

Vocab padding (+33% cumulative). Padding 50,257 → 50,304 costs 47 unused embedding rows and unlocks the accelerated tensor-core path. Nearly free.

Fused linear cross-entropy (+44% cumulative). Liger’s FusedLinearCrossEntropyLoss fuses the lm_head matmul into the nonaccomplishment and chunks internally, truthful the afloat (B*T, vocab) logits tensor is ne'er materialized. Measured head-to-head astatine the aforesaid batch size it is 6% slower:

Even though it’s slower per step, it buys backmost a bully magnitude of VRAM (8 GB connected my 5090) truthful the summation successful micro-batch size much than makes up for the mislaid 6%. Claude was speedy to cull it because it was 6% lower, but wide it was a awesome measurement to claw immoderate other throughput.

Non-gated MLPs. Dropping the gross projection (SwiGLU → relu², 2 matmuls alternatively of three) connected the mini model: 183,035 → 214,173 tok/s and 6 GB little VRAM. One caveat from the ablations: a SwiGLU intermediate ratio of 2.75 does not transportation to relu². The exemplary learns noticeably worse. Use 4× for non-gated.

bf16 maestro weights. Keeping the optimizer maestro weights successful bf16 alternatively than fp32 trim VRAM 27% and raised throughput from 640K to 1.4M tok/s connected the 1.5B config. That was a immense speed-up, 2.2×. The value costs is existent but small: CORE 0.22 vs 0.23 astatine 4,000 steps. When you’re optimizing for capacity per dollar, observant dtype handling is 1 of the highest-leverage and underdiscussed knobs available.

Hardware. Same code, 150M model, FP8: RTX 5090 astatine 184,662 tok/s, B200 astatine 477,440 tok/s. 2.59× from hardware alone, earlier accounting for the other VRAM letting you push batch size further.

What didn’t work

Document-boundary masking pinch flex attention. Packing documents into 1 series lets tokens be crossed boundaries, truthful I fixed it properly: per-token archive IDs and disguise retired attraction truthful each token tin only be to its existent document. It was elegant, but I deleted each of it. Andrej Karpathy besides recovered that cross-document leakage does not make things overmuch worse nether BOS-aligned packing. Best-fit packing replaced it successful ~10 lines, and attraction went backmost to an unconditional F.scaled_dot_product_attention(..., is_causal=True). I judge this is besides conditional connected the dataset and the training documents.

Liger RMSNorm and RoPE. RoPE was 2.2× faster successful a microbenchmark and produced no measurable change successful end-to-end throughput. RoPE is not portion of the captious compute bottleneck astatine this scale. RMSNorm was outright slower than PyTorch 2.9’s built-in F.rms_norm (0.41ms vs 0.25ms). Both reverted, not worthy the complexity.

Nanochat-style initialization. Embeddings astatine N(0, 0.8), linear weights uniform, output projections zero-initialized truthful the residual watercourse starts arsenic axenic identity, LM caput astatine N(0, 0.001). Theoretically overmuch nicer than GPT-2’s N(0, 0.02) everywhere. The nonaccomplishment curve starts marginally little and the 2 curves overlap by ~1,500 steps. No measurable value difference. I kept it, but for aesthetics, not evidence.

Streaming datasets. Great for getting started, incorrect for a existent run. Even erstwhile the web looks healthy, section shards gave 2-3% much throughput, and occasional web dips costs acold much than that. For runs longer than a fewer hours, it’s worthy it to salary the download erstwhile astatine the commencement of training.

Ablation connected value-embedding

Value embeddings were 721M parameters for a 3.8B model. I trained the aforesaid exemplary pinch the aforesaid config pinch value_embeddings: mendacious and compared it against the original run, which I’d already paid for, retired to 12,500 steps and 29B tokens.

0.46% amended nonaccomplishment and 3.2% amended CORE, for 19% much parameters. The throughput is identical, because worth embeddings are lookups. They costs representation and optimizer authorities but fundamentally nary FLOPs.

Two absorbing findings:

• Value embeddings bought the balanced of astir 1,200 training steps. Here is really to value that: betwixt steps 10,000 and 12,500 my baseline nonaccomplishment fell 0.0194, truthful 2,500 steps buys astir that much. The value-embedding advantage is 0.0096, astir half of it — telephone it 1,200 steps retired of 25,000. So 19% much parameters is worthy ~5% much training.
• CORE moved astir 7 times much than nonaccomplishment did (3.2% vs 0.46%), and the spread shrank steadily during training. That’s worthy knowing if you’re utilizing CORE to make decisions: it’s an accuracy metric, truthful items adjacent the determination bound flip connected mini logit changes, and it’s centered against a random baseline, which amplifies comparative differences while scores are still low.

Value embeddings are useful for a mini exemplary and travel astatine almost nary throughput cost. Spending a small spot of VRAM connected this gives the exemplary a shape of bias toward definite concepts that mightiness beryllium useful for CORE.

Discussion

Misleading micro-benchmarks

We could beryllium tempted to judge that 1024 tokens discourse is plentifulness for a precocious CORE score. Going backmost done the per-task logs, that conclusion is incorrect connected immoderate tasks that are very discourse sensitive.

3 of the 22 CORE tasks person prompts that fundamentally ne'er fresh successful 1024 tokens:

SQuAD is the striking one. It doesn’t stagnate, it decays monotonically to precisely zero: 0.1478 → 0.0617 → 0.0099 → 0.0007 → 0.0000. The exemplary gets steadily worse astatine this task the longer it trains, which is not a point models usually do.

Two specifications explicate it. SQuAD is simply a 10-shot task successful the DCLM bundle, truthful each punctual is 10 worked examples followed by the existent one. Median of 1,998 tokens connected my eval data. Not 1 fits successful 1024. And erstwhile a punctual is excessively agelong my harness keeps the last max_seq_len tokens.

The trial transition sits astatine the end, truthful it ever survived; a trial illustration is only ~169 tokens. What sewage truncated was the 10 demonstrations. The exemplary was reference the transition and the question, and almost ne'er seeing the examples that thatch it the expected output format. Since SQuAD is scored connected exact-token lucifer against the golden answer, fluent prose scores zero each time.

That besides explains the decline. An early, high-entropy exemplary occasionally emits thing short and generic that happens to match. As it sharpens it commits to well-formed continuations, and the accidental hits disappear. Funnily enough, getting amended astatine connection made it worse astatine guessing correct by accident.

boolq shows a gentler type of the aforesaid shape. It peaks astatine measurement 10,000 (0.6294) and declines to 0.5131. Language recognition ne'er moves disconnected chance astatine all.

In short, 0.338 was measured pinch 3 of 22 tasks scoring near-zero for reasons that person thing to do pinch exemplary quality, conscionable the size of the discourse magnitude being fed to it.

The effect of larger context

As we person seen, if we want the highest CORE people imaginable we request larger context. But this has consequences connected the training throughput.

Double the discourse length, halve micro-batch to clasp VRAM constant, truthful tokens per optimizer measurement stayed identical. I stopped it astatine ~28,000 steps to prevention the past fewer hours of rental, truthful the learning-rate warmdown ne'er afloat completed and the number beneath is simply a little bound.

CORE went from 0.3384 to 0.3840.

At measurement 20,000 the 2 runs person the aforesaid eval nonaccomplishment to 4 decimal places (2.0160 vs 2.0164) and disagree by 0.034 connected CORE. It was astonishing to spot that debased level of relationship betwixt CORE and eval nonaccomplishment connected the ClimbMix dataset.

squad and boolq unsocial are 83% of the gain. boolq contributes the most, because its random baseline is 0.5 and CORE centers against that: a earthy +0.196 becomes a centered +0.517. Strip those 2 and the remaining 20 move +0.008 successful total, astir what 14% much tokens buys connected its own.

Language recognition went from 99.7% cropped to 14% cropped and moved +0.005. This is by acold the hardest task successful the CORE information benchmark for our existent model.

A mates of tasks sewage worse: commonsense_qa dropped 0.072, cs_algorithms 0.031. Across 22 tasks immoderate activity successful some directions is expected.

2048 was worthy paying for arsenic a measurement decision, not a value one. It costs 9% throughput (480K → 437K tok/s), and extracurricular the tasks that couldn’t beryllium scored astatine 1024 it bought almost nothing. 1024 is good for training and a “cheap” measurement of getting your exemplary to a bully CORE score. 2048 unlocks immoderate tasks that are very discourse bound.

Future work

Limitations

Four things I ne'er ablated. Peak LR, from nanochat’s sqrt(768/d_model). I didn’t really want to walk money to expanse learning rates. I moved from cosine to trapezoidal because of the 858M post-mortem, location could beryllium schedules retired location that are much efficient. QK-norm, connected by default and ne'er toggled off. And the GQA ratio, since it’s a bully lever to prevention connected memory.

Most of those are inherited from nanochat alternatively than tested here. That is simply a defensible measurement to walk a mini fund — personification other already paid for the research — but it intends I americium trusting that Karpathy’s results transportation to my model, information and scale.

Open questions

There is simply a batch of absorbing activity I’d want to prosecute if I had much clip and resources:

• Value embeddings versus reallocation. The comparison supra was VE against nothing. The 1 that matters is VE against spending those 721M connected thing else.
• 1024 versus 2048 astatine matched wall-clock. The rerun changed discourse and ran longer, truthful it settles the measurement mobility but not the value one.
• Why commonsense_qa regressed by 0.072 astatine the longer context, erstwhile thing astir that task involves agelong prompts.
• Sharding the optimizer, the measurement nanochat does. I utilized plain DDP pinch a single-GPU Muon, which intends each rank holds a afloat transcript of the optimizer authorities and redundantly recomputes the aforesaid Newton-Schulz update. nanochat drops the DDP wrapper wholly and does ZeRO-2 sharding wrong the optimizer, overlapping reduce-scatter, compute and all-gather. The representation triumph is the definite one, and freed representation turns into batch size, which is tokens for the aforesaid dollars. Whether the redundant orthogonalization besides goes distant depends connected really the sharding is done: Muon needs the afloat gradient matrix, truthful splitting a matrix crossed ranks doesn’t help, while giving each rank full matrices of its ain would. I haven’t explored that astatine each but I deliberation it would beryllium a awesome measurement to further summation the full training throughput astatine the costs of immoderate other machinery.
• Additional information exploration. I haven’t had a batch of clip for information study connected either the CORE benchmark aliases the ClimbMix dataset. I’m judge this would thief america claw moreover higher capacity pinch the aforesaid compute budget.

Closing thought

GPT-2 was a frontier consequence successful 2019, produced by a well-funded laboratory pinch a ample team, and its 1.5B exemplary scores 0.2565 connected CORE. 7 years later I hit that by a wide separator successful my evenings, for $998, connected hardware I rented by the hour.

The frontier moved, and everything came pinch it. Work that needed a laboratory tin now beryllium done by a azygous technologist successful the evenings. I wonderment what benignant of insane instrumentality we will beryllium capable to build successful 7 years from now!

Appendix: the config

The full run, flattened from the YAML includes into 1 block.

The AdamW learning rates travel nanochat’s sqrt(768/d_model) scaling rule; the Muon LR of 0.02 is inherited from location too.

Appendix: illustration matter generation