Show HN: I trained a 125M model to autocomplete piano on-device

TL;DR: I trained a 125M-parameter transformer to autocomplete soft performances successful existent clip (~108 notes/sec connected an iPhone 15). The biggest improvements came from uncovering the correct MIDI representation, cleaning the training information aggressively, and adding DPO post-training.

Almost a twelvemonth ago, I started tinkering pinch an idea: link my MIDI soft to my phone, play something, and person AI autocomplete the opus for me. Think GitHub Copilot, but for piano.

It turned retired to beryllium a deeper rabbit spread than I expected. Fourteen experiments later, it is yet astatine a constituent wherever I americium happy capable pinch it to constitute about.

The app, RollTab, is disposable for free here if you person a MIDI keyboard and an iPhone/iPad. 1

A fewer sound samples

Each audio starts pinch a short prompt, followed by the model's continuation.

Pokémon, Pallet Town (8-note prompt)

Your browser does not support the audio tag.

Final Fantasy VI, Terra's Theme (16-note prompt)

Your browser does not support the audio tag.

Für Elise (16-note prompt)

Your browser does not support the audio tag.

A MIDI record is rather different from an MP3 aliases different audio formats. Rather than storing recorded sound, it stores euphony arsenic a series of events: a cardinal is pressed astatine a definite transportation and velocity, a cardinal is released, the prolong pedal changes state, and truthful on. Other events see switching instruments aliases changing volume.

These events are often organised into aggregate tracks. A popular aliases crippled MIDI mightiness person melody, chords, bass, drums, strings, and respective synth parts. This task is focused connected soft continuation, truthful I mostly kept piano-like worldly and removed aliases reduced the rest.

To train a transformer connected these performances, I first needed to move the MIDI events into a discrete series the exemplary could publication and predict. The astir evident mapping is to make a token for each MIDI event:

If you see transportation and velocity straight successful a NOTE_ON token, the vocabulary tin turn quickly. There are 128 MIDI pitches and 128 velocity values, truthful the naive mixed note-on vocabulary has up to:

tokens conscionable for note-on and note-off. In believe you would astir apt bucket velocity, but the basal rumor remains: galore combinations are rare, and the model has to study a batch of building from sparse tokens.

A communal betterment is to facet the practice pinch a grammar:

Now the output spaces are smaller:

You tin enforce the grammar during procreation by masking invalid adjacent tokens. After NOTE_ON, only transportation tokens are valid. After pitch, only velocity tokens are valid. This guarantees syntactically valid output.

I tried note-on/note-off style representations, but my models tended to drift. They would hide to emit note-off, time off hanging notes, aliases suffer way of active state. That was particularly bad for my target: a mini exemplary moving adjacent to real time connected a laptop aliases phone.

Another practice I tried was person to:

This avoids note-off drift because statement long is explicit. The clip displacement token advances the playhead erstwhile nary statement is played.

This worked amended musically, but it was slow. One philharmonic statement took roughly four autoregressive transformer steps. It besides burns done the discourse model quickly.

The last representation

The practice I yet settled connected was:

There is nary abstracted TIME_SHIFT arena successful the last version. Silence is represented by delta_onset connected the adjacent note: the clip since the erstwhile statement onset.

For example:

means: play C4, hold 24 clip steps earlier the adjacent statement onset, past play D4.

Chords are represented arsenic aggregate notes pinch delta_onset = 0, sorted by pitch2:

It's besides not a level token watercourse like:

Instead of spending 4 transformer passes generating the attributes of a note, the transformer advances the euphony by 1 complete statement astatine a time. In practice, this gets the ample exemplary to astir 108 notes/second connected an iPhone, good supra thing a quality would request for unrecorded playing.

Internally each statement has 5 categorical fields, each pinch its ain vocabulary3, pinch timing quantized to fixed steps.4

Each section gets its ain embedding. The statement token is the sum of each the embeddings:

The exemplary past has abstracted output heads: pitch, delta, duration, and truthful on.

There is simply a mini nested decoder betwixt the fields, truthful later fields tin information connected earlier predicted fields. But the costly transformer backbone runs only erstwhile per note, not erstwhile per field.

Sustain Pedal

As you mightiness know, pressing down the prolong pedal connected a soft makes notes play moreover aft you merchandise them. I didn't want to muddy the implementation pinch adding prolong pedal events. Instead, prolong is baked into statement long during preprocessing.

If the cardinal is released while the prolong pedal is down, the statement is extended to the pedal-up time. If the aforesaid transportation is played again first, the earlier statement is trim disconnected astatine the retrigger. The consequence is simply a statement long that approximates the existent sounding duration.

This loses the definitive pedal gesture, but it makes the modeling problem overmuch simpler: the exemplary only has to foretell pitch, onset, duration, and velocity.

I searched done a batch of publically disposable datasets and collections, focusing mostly connected older classical euphony successful the nationalist domain. The value varied wildly, truthful I ended up penning rather a fewer cleaning scripts.

The last dataset contained a fewer 100 1000 MIDI files, representing astir 300 cardinal statement events.

The last pipeline:

• selected piano-focused material
• removed aliases reduced pathological multi-track mixtures
• filtered by density and pitch/time coverage
• deduplicated by fingerprints that disregard world transposition and uniform tempo changes
• grouped alternate versions of the aforesaid creation into the aforesaid split

I tried scaling the dataset to astir 5x the size, hoping it would amended performance, but the resulting models were worse. Cleaning and selecting the information mattered much than simply adding much of it.

Initially, training is conscionable cross-entropy complete the 5 output heads, summed together:

This makes it easy to way pitch, duration, and velocity accuracy separately, alternatively than relying connected a azygous aggregate next-token loss.

Still, the training nonsubjective has an important limitation: euphony continuation does not person a azygous correct answer. A held-out opus only gives the exemplary 1 "correct" adjacent note, moreover though location are often galore continuations that would activity musically. Cross-entropy is useful for learning the mechanics of music, but not a awesome proxy for really bully a afloat continuation sounds.

Augmentation

Augmentation was important because the unrecorded input is not a pristine MIDI file. It is maine playing piano, severely capable that notes mightiness beryllium somewhat early, late, excessively hard, etc.

In the extremity I settled connected the pursuing augmentations:

• global transposition
• uniform tempo scaling
• duration/velocity jitter
• dropped punctual notes

Model

The architecture is fundamentally a reasonably modular decoder-only transformer: RMSNorm, rotary positional embeddings, causal self-attention, SwiGLU/MLP blocks, and autoregressive generation.

I chiefly trained 3 exemplary sizes:

The mini exemplary was awesome for speedy experiments, but the mean exemplary almost ever hit it. The ample exemplary performed better, though not by a immense margin.

I americium presently trying to get the mean exemplary adjacent to the ample model's quality, mostly to trim footprint and latency successful the iOS app.

Scheduled Sampling

My champion guidelines exemplary utilized scheduled sampling betwixt the fields of each note. Normally, during training, the long and velocity predictions get to spot the correct pitch. But astatine conclusion clip they person to activity pinch immoderate transportation the exemplary really predicted.

So during training I sometimes fed the exemplary its ain predicted transportation instead. I started astatine 0% for the first fewer epochs, past gradually accrued it during training, up to 50% successful the champion model.

Funnily enough, this accrued validation nonaccomplishment but improved the continuations.

At first, information was conscionable maine listening.

I generated continuations from held-out songs utilizing prompts of 4-32 notes, past compared exemplary outputs manually. This was slow and annoying and aft a while everything sounded for illustration noise.

Four-note prompts were the hardest: location simply was not overmuch philharmonic discourse to activity with. Eight notes worked better, while 16–32 statement prompts were substantially much reliable because the exemplary had capable building to infer what was happening.

Unprompted procreation is very overmuch deed aliases miss, but that isn't the use-case I'm gunning for.

I besides wrote a bunch of automatic metrics:

• repeated transportation n-grams
• pitch entropy
• pitch-class entropy
• pitch range
• note density
• long pauses
• chord density

These metrics were useful for catching evident failures, but they were not bully capable to prime the champion model.

Eventually I utilized Gemini 3.5 Flash for pairwise evaluation. Asking it to springiness a azygous absolute people was inconsistent. Asking instead, "given A and B, which continuation is better?" worked overmuch better, particularly erstwhile I mirrored each comparison to trim position bias. 5 This fto maine build a reasonably ample penchant dataset, which I past utilized for DPO.

Initially, Gemini overindexed connected really bully a continuation sounded successful isolation, alternatively than really good it followed from the prompt. The outputs often sounded amended connected their own, but felt disconnected from what I had conscionable played.

Better prompting helped, but I yet divided the information into 2 criteria: a continuation score, measuring really good the output follows from the prompt, and a sounds-good score, measuring its philharmonic value successful isolation. I utilized the continuation people arsenic the superior awesome for DPO.

DPO: Direct Preference Optimization

DPO made the biggest quality aft pretraining. It took the exemplary from occasionally producing a bully continuation to doing it overmuch much reliably.

For each prompt, I generated aggregate continuations and utilized pairwise information to take a amended and worse one:

DPO trains the exemplary to make the chosen continuation much apt than the rejected one, while keeping it reasonably adjacent to the original model.

After DPO, much than 69% of continuations were preferred complete the guidelines exemplary successful my pairwise evaluation.

The β worth controls really powerfully DPO penalizes moving distant from the guidelines model. In my sweep, β=0.01 and β=0.03 improved the model, while β=0.10 pushed excessively difficult and made it worse.

I besides tried a "consensus" dataset: alternatively of trusting each noisy penchant judgment, I only kept penchant pairs wherever the evaluator agreed consistently. That produced the champion consequence successful this sweep.

My gut emotion is that the guidelines exemplary had already learned a reasonable intelligence exemplary of music, conscionable not what makes a bully continuation.

What Did Not Work

A batch did not work:

• Note-on/note-off drifted excessively overmuch for mini real-time models.
• Grammar-masked token streams were valid but slow.
• Broader information made results worse erstwhile the information was noisy.
• Bigger models helped, but did not magically lick loops.
• Mirostat reduced repetition but often made outputs incoherent.
• Extra section auxiliary losses made training slower without clear listening wins.
• Absolute scalar Gemini ratings were worse than pairwise judging.
• Validation nonaccomplishment unsocial missed important differences successful rollout quality.
• Born-again networks (retraining a exemplary connected its ain soft predictions) didn't amended value here.

I exported the PyTorch exemplary to Core ML and quantized the weights to INT8. The first motorboat is still annoyingly slow while Apple's runtime optimizes the exemplary for the disposable hardware.

The exemplary was only trained pinch contexts of up to 512 notes, but I wanted to support longer sessions. Whenever the discourse gets adjacent to the limit, I support the astir caller 384 notes, rebuild the discourse from those, and proceed from there. This intends rebuilding the KV cache, but the exemplary is accelerated capable that it hasn't been a awesome problem.

I utilized RoPE for positional encoding, truthful successful mentation I could do thing neater pinch shifted positions and a ringing buffer. Unfortunately Core ML does not expose Q, K, and V directly.

At that point, though, I was mostly conscionable happy that it worked.

This has been a very nosy project. There are plentifulness of absorbing papers connected euphony generation, but I deliberately avoided reference excessively profoundly into them astatine first. I wanted the nosy of moving done the problem myself, alternatively than conscionable implementing personification else’s research. Only afterward did I spell backmost and comparison my attack pinch the existing literature. 6

It is still acold from perfect. It loops occasionally, short prompts are difficult, and location is plentifulness I want to improve. Think GPT-2, but for piano.

But I person yet reached the constituent wherever I really bask sitting down astatine the piano, playing a fewer notes, and seeing what we travel up pinch together.