An SLM trained on $8 ESP32-S3

This matter was written pinch the assistance of an AI. Careful: assistance doesn't mean the AI wrote it. It intends it corrected, reviewed and filled successful immoderate parts, but the writer is quality (or truthful I believe).

Léeme en español

The (Klingon) GPT cipher asked for but everybody needed.

And why did we request it? Because we return for granted that training a exemplary requires a GPU aliases a datacenter. Not always: sometimes thing arsenic mini and arsenic inexpensive arsenic an eight-buck micro is capable to train 1 from scratch.

Read that again: train. From scratch. Not tally a pre-cooked model. Train. Forward pass, backprop and weight updates, wrong the chip.

Edge AI is thing new. TinyML has been doing conclusion connected microcontrollers for years, organization ports of llama2.c — Andrej Karpathy's minimalist task — put ~260K-parameter transformers connected an ESP32, and conscionable precocious a superb task sewage a exemplary of almost 29 cardinal parameters moving connected an $8 ESP32-S3. They're fantabulous pieces of work, and though they hap to person overlapped successful clip pinch this experiment, they weren't the exemplary we were following.

The esp32-ai task by Slava S. (slvDev), and others for illustration it, each stock 1 thing: they are inference. The exemplary is calved location other — a GPU, a datacenter — trained connected its data, quantized, and only past loaded onto the spot truthful it tin run it. The encephalon is cooked extracurricular and served on-chip.

We asked ourselves a different question: what happens erstwhile the exemplary can't beryllium calved outside? What happens erstwhile it can't travel pre-trained — because the information it needs to study doesn't beryllium until the instrumentality is in place — can't transportation that information pinch it, and has nary net to download it?

We cognize what you're thinking: "Fine, but… why would anyone request to train a transformer connected an ESP32?" And it's a adjacent question. Maybe the reply is "no logic astatine all". But… if we're going to get creative, image this: a sensor bolted to a portion of workplace machinery successful the mediate of a field, which has to study the normal vibration of that circumstantial machine (different from immoderate different instrumentality successful the world) to spot erstwhile something's going incorrect and schedule attraction earlier it breaks (and no, you won't drawback maine out: I said there's nary internet, but… LoRa? ;). Or image a sensor successful a crippled of onshore that learns really that particular ungraded dries retired — its earth, its sun, its drainage — and predicts erstwhile it'll request watering, earlier the plants suffer. In those cases (and others we tin deliberation of) the information didn't beryllium until the instrumentality was installed: cipher could person pre-trained it. The spot has to study connected the fly, alone, correct wherever it stands.

We don't person a portion of workplace machinery lying astir to research with. So, to put the existent learning expertise of an ESP32 to the test, we designed the research astir the only point we did have: language. How acold tin an eight-buck spot get, learning a connection from scratch, pinch nobody's help?

Training wrong an eight-buck micro imposes rules of the crippled a datacenter doesn't have. And those rules dictate everything else:

Once again, representation rules. An ESP32-S3 has a fewer MB of RAM/PSRAM, not gigabytes. That caps the size of the model: we're talking hundreds of thousands of parameters here, not millions. A exemplary that "fits" and trains connected the spot is, by necessity, small.

The mini exemplary rules the task. A exemplary this size tin study the structure of a connection (how words are built, its phonotactics, immoderate grammar), but not the heavy semantics of an full tongue. The task has to beryllium trim to the model's size.

And the corpus rules the result. With a mini exemplary you don't request gigabytes of text. You request a corpus that is compact, cleanable and structured. And, if you're going to people it, you request it free of copyright headaches.

Put the 3 constraints together — mini model, constrictive task, cleanable and compact corpus — and the mobility becomes concrete: which connection ticks each of those astatine once?

I could show you we ran a brainstorming convention to determine which connection to research pinch successful this first phase, but sometimes it's amended to instrumentality to the aged "the first thought is usually the bully one"… and successful this peculiar case, the first connection that came to mind… was Klingon.

Yes, Klingon. The connection of the Star Trek warriors. And if you support reference you'll find the prime has little of the nerd astir it and much of the technologist than it seems.

Klingon isn't gibberish that spits retired alien-sounding noises and calls it a day. It was created by the linguist Marc Okrand successful 1984, and it is simply a constructed connection pinch systematic phonology, grammar and morphology: it has an different connection bid (object-verb-subject) and a morphology pinch strict rules. In different words: it has existent building to learn, which is precisely what a mini exemplary needs successful bid to beryllium it's really learning and not conscionable memorising noise.

And it fits our constraints:

• Rich structure, bounded vocabulary. Enough grammar for the exemplary to person thing to capture, but a corpus mini capable to fresh the micro's rules.
• Reduced alphabet. Working characteristic by character, a vocabulary of ~30 symbols keeps the exemplary mini — precisely what the ESP32's representation demands.
• A clean, free corpus. There's a organization dictionary nether an unfastened licence (boQwI'), truthful we could train and stock without stepping connected anyone else's ineligible turf.

Put different way: Klingon is an perfect testbed. A existent language, pinch existent grammar, but of a size that fits successful the thenar of your hand. The cleanable laboratory successful which to compression the motor earlier taking it — possibly successful different shape of the task — to existent number languages, pinch existent speakers. Sami? Quechua? Nahuatl? That… is different story.

Oh, and yes: it's undeniably cool. A low-cost spot whispering Klingon connected a 1.3" OLED has thing of the nerd astir it, but thing of magic too. But that… is conscionable the bonus.

qapla'. (Klingon for "success". It seemed for illustration the correct name.)

So cipher walks distant pinch the incorrect idea, and because rigour matters:

It is not a pouch ChatGPT. It doesn't chat, doesn't reply questions, doesn't reason. It's a mini generative exemplary that learns the shape of a language, not its meaning.

It does not nutrient semantically cleanable Klingon. It learns building — prefixes, suffixes, connection order, really phrases sound — and produces matter that looks for illustration Klingon and respects galore of its rules. But don't expect afloat meaningful sentences that a Klingon clever clever would motion disconnected connected without title (if personification aboard the Enterprise received a connection from our ESP32, it would astir apt extremity up causing a negotiated incident). And successful immoderate case, location isn't capable Klingon successful the world to propulsion that off, nor would the exemplary it would return fresh connected the chip.

It is not conclusion successful disguise. There is nary pre-trained exemplary hidden anywhere. The spot starts pinch random weights and nary anterior knowledge of Klingon, and learns from scratch. What you're seeing is existent learning, not a exemplary cooked location other and served here.

It is not fast. Let's beryllium serious: we've said it to death, but… it's an eight-buck micro, not a GPU! Training takes hours. That's precisely the point: that it's slow and still works.

What it is: impervious that a humble microcontroller pinch a maker's tone tin train a connection exemplary from scratch, wholly connected board. No more, nary less. Which is nary mini thing.

Here's what really matters, and what you tin verify for yourself successful the code. The full learning rhythm happens connected board:

Inside the ESP32-S3:

• ✓ Random initialisation of the weights (mind the seed, ahem)
• ✓ Reading and tokenising the corpus
• ✓ Forward pass
• ✓ Loss computation (cross-entropy)
• ✓ Backpropagation (gradients derived by hand)
• ✓ Weight updates (SGD pinch momentum + cosine LR)
• ✓ Saving a checkpoint to flash (LittleFS)
• ✓ Text procreation pinch the learned weights

Outside the ESP32-S3:

• ✗ Nothing.

There's thing caller here. It's a mini but complete transformer: 1 block, single-head causal attention, tied weights, a ReLU FFN and LayerNorm. It useful characteristic by character, pinch a vocabulary of ~31 symbols. In total, ~319,000 parameters.

There's nary PyTorch and nary autograd here. Every derivative of the guardant walk is written retired explicitly successful C. To make judge location were nary mistakes, each gradient is checked against a centred finite quality of the guardant walk — worst comparative correction 1.07e-08, against a period of 1e-4. The kernel was besides built pinch ESP-IDF and tally nether QEMU's xtensa emulation earlier rubbing existent silicon.

Both checks are successful tests/, and they tally against the header this repo publishes, truthful you don't person to return our connection for immoderate of it:

It's the astir delicate portion and the astir nosy to watch working.

A 319K-parameter exemplary "only" takes ~1.3 MB. But training isn't conscionable holding the weights. You need, each astatine erstwhile and successful memory: the weights, an arsenic ample transcript for the gradients, different for the optimiser momentum, different to support the champion model, positive each the intermediate activations the backward walk needs, positive the corpus. Several MB successful the chip's PSRAM, each told.

That's the existent quality betwixt inferring (the weights are enough) and training (weights + gradients + momentum + activations). And it's precisely what makes this task different: it doesn't shop a exemplary to tally it, it trains it, pinch everything that drags along.

The corpus we trained connected is successful the container (corpus/klingon.txt, Apache 2.0, regenerable pinch corpus/build_corpus.py), truthful you tin clone and tally without hunting for data. But the absorbing research isn't repeating ours — it's pointing this astatine your text.

What you'll need: an ESP32-S3 pinch PSRAM (the N16R8), an SH1106 OLED complete I2C (optional, but it's half the fun), and PlatformIO.

And that's it. The spot starts pinch random weights, trains, and periodically saves the checkpoint pinch the lowest training-loss moving mean it has seen. (There's nary validation split, truthful that's a training-loss checkpoint, not a "best model" successful the strict sense.) On the adjacent footwear it loads it and generates. Unplug it halfway done and you suffer immoderate it learned since the past checkpoint; unplug it aft it's done and the encephalon is still there.

One warning: this takes hours. Many. Even days. It's not a bug: it's training a transformer from scratch connected an eight-buck ESP32-S3.

All of this happened wrong the chip, powered by a telephone charger.

And the nonaccomplishment dropping, precisely arsenic it looked connected the screen:

(We person nary photograph of the very first boot: by the clip it occurred to america to archive it, the spot had already been astatine it for a bully while.)

The moving mean is the 1 that matters: each batch's nonaccomplishment jumps astir a batch (it depends which sentences it got), but the mean comes down steadily. The spot vanished astatine ~1.87.

(Yes, the surface says KlinGPT. The task went done a mates of worse names earlier settling connected Qapla', and KlinGPT stayed connected the OLED because it's the first nexus successful the chain.)

It's not afloat meaningful Klingon — we did pass you — but look astatine the morphology:

• SuvwI' is warrior: the guidelines Suv (to fight) positive the suffix -wI' ("one who does"). A existent word, correctly built.
• DujDaq is on the ship: Duj (vessel) positive the locative suffix -Daq.
• qarghtaHvIS carries -taHvIS (while, continuous + adverbial), a compound suffix stuck precisely wherever it belongs.
• And hIngan lands 2 letters short of tlhIngan, which is virtually the connection for "Klingon".

Other words are good formed but don't exist. Nobody gave the exemplary rules, grammatical labels aliases morphological segmentation — conscionable earthy text, characteristic by character. From that it picked up capable regularity to recombine roots and suffixes into shapes that clasp together.

How overmuch of that is genuine generalisation and really overmuch is statistical echo of the corpus, we can't show you from 2 samples. It's a 319K-parameter exemplary trained connected a mini corpus: expect a mix. What's not successful mobility is wherever it happened — wrong an eight-buck micro, pinch nobody's help.

Anyone pinch time, curiosity and capable willpower tin constitute a backprop by manus and train a small exemplary connected a micro. That's not the difficult part.

The absorbing spot comes next: pushing the motor to its limit pinch a existent task — a existent language, pinch existent speakers and a corpus acold larger than Klingon tin offer. That's wherever the laboratory ends and the existent engineering begins: getting a much tin exemplary to fresh and study connected a spot that hasn't grown.

We person an thought aliases two. But that, if it happens, is different story.

Code nether Apache 2.0. Take it, hack it, provender it your ain language.

All the training corpus comes from freely licensed sources (boQwI' / klingon-assistant-data, Apache 2.0). The Klingon connection was created by Marc Okrand; Klingon, Star Trek and associated marks are the spot of their respective owners (CBS Studios / Paramount). This is an open-source, acquisition and investigation project, not affiliated pinch aliases endorsed by them. See CORPUS.md for the specifications of sources and licences.

Oh, and the random number generator's seed is 1701. If you don't cognize why… past maybe, conscionable maybe, this isn't your repo ;)