44% on ARC-AGI-1 in 67 cents

I trained a mini transformer from scratch successful 1.5hrs connected a 5090
Beats galore LLMs, and scores the aforesaid arsenic TRM/HRM

This is an upgrade to my previous model
Faster, better, cheaper and still unfastened source.

Also gets 7% connected ARC-2

Discussion connected Twitter, Code connected github

Performance connected ARC-1 nationalist eval. I only comparison against models that do akin trial clip training

This is the 3rd blog successful a bid of useful connected ARC-AGI. Prev: Blog 2, Blog 1.

Many ppl thought the prev consequence was impossible. It sewage attraction from apical researchers and went viral connected X. Eg: Discussions by Lucas Beyer, Jeremy Howard, Rohan Anil, and comments by galore others.

Why activity connected this?

I deliberation sample ratio is the astir important problem successful AI coming and I want to lick it.

The volition down this activity is to (1) find the limits of sample ratio erstwhile restricted to transformers / today’s heavy learning methods and (2) trim costs truthful loop is overmuch faster and cheaper.

ARC is simply a awesome benchmark to trial this:

• Very fewer samples (only a 1000 puzzles) successful a precocious dimensional space
• Its a metalearning benchmark, truthful each puzzle uses a different rule, pinch immoderate communal concepts
• Very fewer priors needed: each conception needed successful the eval group is coming successful the train set
• It is incredibly easy for humans to solve, and accessible to moreover mediocre AI researchers
• Benchmark is still unsaturated (for information efficiency, disregard LLMs and approaches that usage tons of synthetic information aliases quality inductive biases)

Next, I’ll activity connected caller investigation ideas to break these limits. I’ll effort to support costs debased truthful that anyone successful the world tin activity connected this.

How does it work?

The wide attack is akin to past clip (full method specifications here), but I added a bunch of upgrades. Here’s a speedy summary of the approach:

• Each input-output brace is converted to a series of tokens. These sequences are autoregressively trained connected by a mini transformer. This is done from scratch astatine trial clip connected some the train group and eval group puzzles (test labels hidden).
• To alteration cross-task learning, each puzzle is fixed a abstracted additive embedding (learnt). Since each series has 2 2D grids, positional are learnt utilizing 3D RoPE embeddings.
• The sequences are augmented pinch colour and dihedral permutations. During inference, the trial inputs are augmented, and the inverse aug is applied connected the outputs produced. The 2 astir communal outputs are submitted (AAIVR).

Changes since past time

The main extremity was to find improvements to the architecture / algorithm that amended the sample ratio of the model.

The biggest increases successful scores were owed to

• Modern architecture (SwiGlu alternatively of GELU, RMSnorm not layernorm, etc.)
• More information diversity, amended shuffling of data
• scaling up: 8 layers alternatively of 4,

Biggest decreases successful costs were owed to:

• Way less augmentations (more sample efficient!)
• AdamW -> Normuon
• flash attraction pinch varlen training + flex attraction kernels for inference

A awesome alteration is that I don’t train connected input tokens anymore. This intends the nonaccomplishment usability only includes output tokens (which makes the attack supervised). This. performs somewhat amended 40% $\to$ 44% but I don’t understand why. Perhaps finite exemplary capacity

I besides accrued the training information by adding the non-overlapping tasks from ARC-2. I did this very cautiously to guarantee nary leakage. You tin region the other information if you don’t for illustration it and it will still people ~40%, but it will request ~double the compute.

Context: ARC-2 contains 773 ARC-1 puzzles and 347 caller puzzles. Most eval puzzles of ARC-1 are repeated, truthful if you naively train connected ARC-2, past its a dataleak and you will people 100%. I debar this by cautiously filtering retired the 773 repeated puzzles (so nary leak!)

There are galore different changes that gave incremental improvements successful capacity aliases speed. Find the full database of changes here.

Interesting behaviour

Since I americium nary longer training connected inputs, this attack is now supervised. What’s weird is that the trial nonaccomplishment is now worse, yet it scores better! Also it is much unchangeable and there’s little variance successful scores.

Many ppl coming are moving connected sample ratio by aiming for the lowest val nonaccomplishment connected a mini dataset. I deliberation that’s great, but this points retired a nonaccomplishment mode successful specified an approach

I do deliberation the unsupervised style training will beryllium amended successful immoderate scenarios, and I americium evaluating this.

Before NorMuon, I tried vanilla Muon. Obviously it trained overmuch faster than AdamW, but the nonaccomplishment (and scores) would loiter astatine the extremity alternatively of converging. I recovered that cranking down the momentum and/or LR drastically astatine this constituent helped, but I didn’t want to make manually changes for illustration this. When I switched to NorMuon, the problem disappeared

Ablations

The biggest publication to capacity seems to beryllium bully representations (3D RoPE + per-task embedding).

Removing 3D RoPE aliases the per-task embedding gives a steep drop. Both ablations saturate astatine 25%

• Training connected inputs performs somewhat worse -> ~39%
• Restricting training group to ARC-1+ConceptARC only performs astir the same: ~40%
• Switching from 3D RoPE to 1D drops people to ~24%
• Removing the per-task embeddings drops people to ~24%
• Running the exemplary CompressARC style (training from scratch connected each task separately, and unsupervised), gives a drops capacity down to ~18%
• CompressARC but supervised gets ~15%

Finding the champion scores connected different ablations. Comparing costs makes small consciousness present arsenic each but the first ablation requires a batch much compute

How tin others contribute?

The codification is unfastened source. Feel free to modify it and amended people aliases trim cost. (Pls don’t summation training data)

Try reaching 65% – you won’t request galore modifications. Evidence: I took the national of each solved tasks from aggregate runs, and sewage 55%. Also a bunch of different tasks are “almost” solved. Some ideas:

• RoPE mixes positional and contented information, which astir apt worsens performance. PoPE should execute connected par aliases better. Or possibly invent a caller pos embedding
• The architecture tin decidedly beryllium modernised further

Costs tin astir apt beryllium reduced 10x pinch handmade GPU code. There are architectural changes that tin besides do this.

Lastly, fig retired really to region information augmentations. (I dislike that I utilized it, disregard everyone who thinks its okay). There are a fewer evident ways to do so, but the situation is keeping training costs low.

Misc

TBH, I didn’t expect to scope 45% pinch conscionable the transformer, I thought this would request caller ideas. I surely didn’t expect to scope it astatine specified debased costs/flops. The ablations show that a astonishing magnitude of perfomance is retained moreover without augmentations aliases synthetic data. Now I’m beautiful judge 65% tin beryllium reached wrong the transformer framework

I don’t understand why others didn’t fig this out. Its conscionable a transformer pinch the astir evident representation. This benchmark has been unfastened for 6 years, was precocious profile, and had a cardinal dollar prize! Maybe researchers underestimate heavy learning? Maybe the costs of experimentation was precocious capable that they couldn’t tally ablations properly? Blindsided by LLMs aliases utilizing harnesses?

Prev criticism/validation connected my attack from celebrated researchers

My aged consequence went viral connected X and galore knowledgeable researchers debated astir it, some for and against. Threads by Jeremy, Lucas, Susan, Andreas, Yoav, and galore more. I’m listing each the criticisms present pinch my answers.

Training connected the eval puzzles is cheating / “training connected test”

• No this is false. “Training connected test” specifically intends training connected the labels of trial data. The labels were not trained on.
• Also, ARC is simply a metalearning benchmark, truthful you’re supposed to study from the eval puzzles.
  • Jargon: ARC has a group of train puzzles and a group of eval puzzles. Each puzzle has illustration pairs and trial pairs. A brace consists of an input grid + output grid.
  • The ARC, the explanation is only the test pair’s output grid successful an eval puzzle.
  • These labels were not trained on. They are hidden. You tin delete it beforehand if you wish

Training connected the inputs of eval puzzles leaks information

• No, this is false. Such an attack is called transductive reasoning and has been studied since the clip of Vapnik.
• Also, this dogma of ignoring eval inputs doesn’t make consciousness successful a world trying to lick continual learning
• Other approaches train a metalearning algorithm and past deploy it to study by moving a CoT aliases by modifying latents done a recurrent loo. My attack aliases what I did present is straight metalearn by modifying the weights of a azygous guardant usability is nary different than learning by
• Note: successful the caller 44% result, training connected inputs has been removed arsenic it scores somewhat worse

Even if training connected eval puzzle inputs is allowed, the trial input specifically should beryllium forbidden

• No, this is false. The aforesaid “transduction” statement applies here
• A metalearning benchmark tin beryllium transductive successful 2 ways:
  • train puzzle $\to$ eval puzzles
  • within the eval puzzle, illustration brace $\to$ trial pair
• This disapproval is specifically answered by the latter

This is against testing policy

• No this is false.
• The argumentation says “test taker must not cognize what the trial will be”. People interpreted this arsenic saying TTT is banned. But it really refers to the quality designing the AI system, not the AI strategy itself.
  • Eg: to discourage designing inductive biases based connected the eval set.
• To anyone progressive successful the ARC community, this has ever been clear since trial clip training has been allowed and encouraged. Steven and Chew’s comments explain this and different concerns.
• TTT besides follows the tone of a metalearning benchmark, truthful its fine!

You are not including training costs

• No, this is false. I show the full life compute. This is the costs of training the exemplary from init + the full costs of moving conclusion connected all tasks. Yes it wholly amounts to 67 cents. Check the prices of a 5090 for 2hrs connected vast.ai

Test clip training is traditionally done 1 task astatine a time. Training connected each trial tasks astatine erstwhile is unrealistic

• Yes, this disapproval makes sense. But it’s nuanced
• I work together that its uncommon to spot to look problem sets successful existent life wherever each problem is fixed astatine once. Even if it is (like an exam), humans tin usually only effort 1 astatine a time
• But conscionable because humans don’t person a capacity shouldn’t mean it invalidates building an AI exemplary pinch that capability. Otherwise we could opportunity LLMs are unrealistic since humans can’t train connected the full net / can’t publication tokens arsenic fast
• Also, it is unclear if humans are constricted to 1 mightiness beryllium capable to train connected different information from aggregate sensory astatine a time, precisely like
  

Providing costs per task amortises costs of training since each trial tasks are trained connected astatine once. So comparing different models is unfair

• Yeah this is fair. In my defense:
  • That’s really the organisers comparison each model, including TRM which besides trains connected each trial tasks astatine once
  • I was besides much generous by including training and conclusion costs while LLMs and different models exclude pre-training/offline training costs.
• I person now switched to (a) showing life compute costs alternatively of per-task, (b) comparing only pinch TRM, HRM and CompressARC and not pinch LLMs / different methods and (c) I added ablations pinch comparable training styles

Answering disapproval astir ARC-AGI itself

When I posted past time, location was a batch of statement astir ARC-AGI itself. Some were valid, but a batch of them were questions Chollet has answered galore times before:

• What does ARC moreover trial for? (fluid intelligence)
• Why should we attraction astir ARC? (fluid intelligence isn’t afloat solved)
• Solving ARC-AGI will not lead to AGI (no 1 claimed that)
• ARC keeps shifting goalposts / its adversarially constructed for LLMs (Both are false)

Chollet’s insubstantial and these tweets1 are bully sources. Summing up his stance: The benchmark intended to trial fluid intelligence, which he considers basal but not capable for AGI. Solving ARC-1 / 2 implies non-zero fluid intelligence, but it isn’t an precocious bound. The benchmarks don’t awesome AGI is reached, they intend to constituent retired the correct investigation questions to ask. There were nary goalposts moved: ARC-1 precedes LLMs, ARC-2 was announced pre-chatGPT and ARC-3 was announced earlier ARC-2 was saturated. He’s besides happy astir advancement connected ARC since it documents advancement successful AI.

I chiefly attraction astir ARC since it tin beryllium utilized to trial for sample ratio which is an important unsolved problem today! It’s besides a good constructed meta-learning benchmark, and is accessible to GPU mediocre peeps. Historically, its been awesome astatine pointing retired the strengths and flaws of LLMs. I besides deliberation its cool that the benchmark stood unsaturated for 6 years, contempt being precocious floor plan / having a ample rate prize since we now cognize DL tin execute extraordinarily good connected ARC-1/2.

There are immoderate valid criticisms IMO:

• They should disallow synthetic information for ARC-1/2
  • Its against the tone of the benchmark and yet astir apical scores coming trust connected ample amounts of it
    • synthetic information lowers the barroom of fluid intelligence needed to lick puzzles
    • It only made consciousness till 2024 erstwhile DL scores sucked.
    • We now cognize LLMs/DL tin study thing fixed capable training data
  • This would besides make the benchmark a awesome trial for sample efficiency. It would complement ARC-3 very well
  • Question is really to forestall synthetic data? Simple:
• Ban offline training/pretraining. Models must train from scratch aft submission
  • Previously this was considered intolerable truthful rule. My exemplary shows this is possible
  • Guarantees nary synthetic information tin beryllium used
  • It makes the comparison adjacent crossed differet models. Otherwise immoderate models for illustration LLMs tin benchmaxx ARC by utilizing ungodly amounts of offline training. (Since the benchmark has been astir a agelong time, galore ARC-like datasets person been created)

• A azygous leaderboard chart comparing aggregate types of models doesn’t make sense. It brings the pursuing 3 problems (solution: abstracted charts)
  • The x-axis is cost/task. But it only counts online compute cost. Some of these models (like LLMs) person monolithic offline pretraining phases whose costs arent counted. You tin usage infinite training compute to efficaciously bring the trial group into distribution, truthful these models should beryllium evaluated separately.
  • Dividing costs by number of tasks makes nary consciousness for the models that train connected each trial tasks astatine erstwhile (like mine, TRM & HRM)
  • Comparing LLMs connected the nationalist eval group makes nary consciousness since the answers to the nationalist puzzles are disposable connected the internet
• The organisers drew premature conclusions from TRM and HRM and attributed occurrence to recursive loops+deep supervision. I deliberation this bias is because they presume axenic heavy learning can’t lick ARC (eg: guidelines LLMs still suck astatine ARC-2). I disagree
• The wording of the testing argumentation tin beryllium improved to region confusion. (Explained here)

Mistakes that I deliberation different approaches are making

Assuming recursion is the adjacent large thing (Eg: HRM, TRM, Arcprize blog)
I do spot the appeal, but location aren’t capable ablations to beryllium this. And my exemplary shows you tin scope the aforesaid capacity without recursion. The only confirmed use of recursion is allowing you to summation compute without expanding representation movement.

Misleading advertizing by HRM/TRM: I besides don’t for illustration that TRM advertised itself arsenic a 7M exemplary erstwhile location are O(100M+) embedding weights being trained. It is misleading, makes it much for illustration a lookup table, and calls into mobility what causes the performance. Worst lawsuit it should person been called 7M “active” weights. Same for HRM. Both didn’t mention this anywhere!

LLM based approaches connected ARC aren’t showing caller capabilities anymore:
Watching LLMs climb the ARC leaderboard has been highly useful arsenic explained below, but I don’t deliberation there’s overmuch to study from their ARC-1/ARC-2 scores anymore:

• Increases successful LLM scores are now chiefly driven by station training (evidence successful adjacent section) and are astir apt a usability of magnitude of synthetic data. They are learning to lick ARC tasks, not study wide absurd reasoning
• There’s besides excessively galore confounding factors to glean thing from caller scores. Comparing LLMs based connected benchmarks is bad subject successful wide (eg: differing amounts of training information aimed astatine a benchmark)
• For LLMs, only backstage scores should count. Their scores connected the nationalist leaderboard are useless arsenic the answers are disposable connected the internet, and are trained on.
• Using harnesses connected apical of LLMs to amended capacity makes small consciousness to me. All the post-training magic is happening wrong the frontier labs, and they tin build harnesses themselves. I deliberation its improbable continual learning will beryllium solved by a harness.

Anti-bitter instruction cheats
I person already argued before that synthetic information and augmentations are bad. Designing inductive biases into the exemplary is besides bad. The truth that we can’t standard this benchmark without cheating for illustration this shows that location are still breakthroughs waiting. I dream much group effort to trim specified tricks that are anti-bitter lesson.

Learnings from LLMs connected ARC-AGI

LLMs person now saturated v1 and v2 of this benchmark. Here’s what I infer from their progress:

ARC-AGI predates LLMs. They performed terribly connected the benchmarks initially, showing that pretraining doesn’t confer wide reasoning capabilities and that LLMs tin suck astatine tasks that are incredibly easy for humans

OpenAI’s O1 getting 75% was a large triumph for LLMs. It suggested that fixed capable data, LLMs tin study immoderate task during post-training. I presume this is what Sholto Douglas often argues about.

When ARC-2 came out, it reset advancement of each LLMs, including the reasoning ones. This suggests moreover post-training doesn’t confer wide reasoning capabilities, different a exemplary that performs good connected ARC-1 would automatically execute good connected ARC-2.

(Basically, the models are learning really to lick ARC puzzles, not wide absurd reasoning and its scores connected a task are limited connected really good it is represented successful its training data. Also, I’m not judge whether “general reasoning” moreover exists successful the first place? Maybe humans are specialised too)

Since then, reasoning LLMs person made dependable advancement connected ARC-2. People often deliberation this intends models are amended astatine wide reasoning BUT what they don’t announcement is that the guidelines models are stuck astatine azygous digits. Taken pinch other evidence, this suggests:

• Scores connected ARC-2 are driven by post-training. (Probably mostly dangle connected magnitude of synthetic ARC data?)
• Labs are benchmaxxing (probably coz customers attraction astir benchmark performance?)
• LLMs are not sample businesslike successful immoderate way.

Don’t get maine wrong, I americium very bullish connected LLMs. The trends connected ARC-2 show that capacity will support improving pinch summation successful compute and data. Its besides unthinkable to spot the simplification successful conclusion costs.

Full database of changes

Changes that modify training dynamics

1. Optimizer changed from AdamW-only to NorMuon + auxiliary AdamW
2. LR schedule changed from warmup+cosine to WSD schedule (warmup %, hold, past linear decay to floor).
3. LayerNorm was replaced by RMSNorm
4. FFN changed from Linear -> GELU -> Linear to SwiGLU-style gated FFN (chunk + SiLU gate).
5. Weight decay changed from “non-attention linear only” to definitive group-wise decay: attraction weights, token embeddings, and task/dihedral embeddings each person their ain WD knobs.
6. Training nonsubjective changed from outputs["loss"] (unsupervised style input+output LM loss) to outputs["output_loss"] (supervised style) only.
7. Training batching changed from smart bucketing based connected magnitude to existent random batching (bucketing retained for conclusion paths).
8. Straggler/incomplete batches are now dropped successful training (drop_last=True enforced).
9. Dataset building now supports/uses broader sources (ARC-1, ARC-2, ConceptARC, optional filtered cross-dataset tasks, submission/private modes), changing train information composition.
10. Color augmentation changed from 1 world epoch-level permutation to per-example augmentation tuples (color + dihedral).
11. Color permutation domain changed to excludes output-only colors, alternatively of unsighted 1..9 permutations.
12. Augmentation procreation now deduplicates transformed inputs via hashing crossed a task (higher unique-sample diversity).
13. Augmentation action is now epoch-cycled pinch shuffled campaigner bid (without-replacement per rhythm behavior)
14. Changes successful hyperparams: optimizer/hparams, epochs, augment cap/type, extent (n_layers), and dataset path.
15. A caller dihedral_embedding was added and is now summed into token conditioning. (Only a very mild capacity increase)

Speed increases without changing training dynamics:

1. Training batches changed from padded [B,S] to packed token watercourse pinch cu_seqlens (no pad tokens successful train path).
2. Attention way changed from padded SDPA masking to packed varlen flash-attention support (cu_seqlens), positive flex-attention decode kernels.
3. Dihedral augmentation moved from offline dataset description to online augmentation action astatine collate time.
4. Build-time training divided changed from ("train","test") to ("train",)

Misc.

1. Resume behaviour changed: optimizer-switch/hparam-change discovery now tin reset/rewarm schedule, altering resumed-run dynamics.
2. Scheduler stepping changed to fractional epoch advancement erstwhile training, alternatively of axenic per-step cosine progression.

TODO: ADD CITATIONS