On the grammar of demoscenes and the possibilities of unfinished code
October 31, 2026
Lately, I have been fascinated by the earth of demoscening and sizecoding and eventually I came throughout HellMood’s notable MS DOS demo “Memories”. Whilst study through his writeup, I became curious in how the author had arrived at several of its ocular effects. I was struggling to create any of my own, so I hoped to compression out a small bit of inspiration out of that.
I noticed that these effects could be traced rear to before experiments consisting of substitute implementations and multiple ideas borrowed from former demos. I began wondering how many another directions the identical underlying ideas power have taken. I decided to commencement alongside a few existing demos, the simplest ones of all, and modify them character by character to detect how those changes can power the ocular feel.
I came throughout Dwitter.net, which is akin Twitter but for 140 character JavaScript demos. They already supported “remix”-ing any dweet so that we can build on top of them. Co incidentally, the most liked Dweet on the location shared the identical consequence as the parallax checkerboard consequence from Memories!
Digging a small deeper, I noticed several remixes shared significant portions of their origin code, differing by lone a few characters, whilst others achieved akin ocular effects through entirely distinct implementations.
Here is an example, I traced rear d/36555 from d/367547 which in rotate is derived from d/36467. The gray underlined substrings display what is average throughout all of them. Blue shows what’s average in between the archetypal two, and green shows for the next couple.
In linguistics, tongue is productive ie a finite vocabulary and a finite set of grammatical rules authorize an enormous figure of distinct expressions. Programming languages portion this asset too. Now let’s regard the aforementioned Dweets. They merge overlapping substrings, but their rendered results differ. Suppose you obtain a 140 character program, and eliminate its final 30 characters. Even alongside a hypothetical alphabet of fair 64 character, there are nearly 1.5 x 10^54 imaginable completion strings that can be formed. Almost all volition be syntactically invalid or create uninteresting results. Nevertheless, the figure of possibilities is astronomical.
It’s not humanly imaginable to enumerate all of them exhaustively and it would be foolish to to create and test all imaginable strings. However, the continuations of a partial program are not likewise plausible.
Given a partial program, certain characters and expressions are considerably additional apt to prosecute than others. This is beautiful much how autoregressive tongue models work. I wondered whether I could use specified a example to example plausible continuations of unfinished Dweets and visualize the distinct directions all program could have taken.
And then, I could surgically delete substrings or mix-and-match them to my own imaginative taste.
I had additionally grown fairly disappointed alongside my own capability to “finish” a demo following starting one, so having an autocomplete that understood the particular tongue of Dweeting fairly looked akin an appealing potential to me.
Around the identical time, I came throughout Mithil Vakde’s tweet (you power have seen him on HackerNews recently).
That gave me another excuse to test - could I inject my own imaginative sensation as a previous into my autocomplete model? That’d be really amazing.
I comprehend that no one can compose a function that determines whether a demo is beautiful, but group can express their preferences by comparing examples, right?
With that in mind, I decided to put my GitHub Sponsors payout towards renting a GPU and article training a example of my own.
The archetypal hurdle was finding a suitable dataset. I initially experimented alongside generating synthetic Dweets using Claude Sonnet 5.5 but established them not at all aesthetically pleasant. Fortunately, Dwitter.net provides an API, which I used to build a humble dataset of 25k Dweets to be used for training alongside alongside ~1.3k additional as a validation set.
I used Qwen2.5-Code-7B as the basis model. It struggled to complete most of the Dweets I tested it on, which was understandable stated how particular their syntax is. Dweets create extended use of code golfing tricks and mathematical shorthand, which are fairly different compared alongside the code a broad intent coding example typically encounters.
I decided to try continued pretraining (CPT) on the corpus, expecting to change the model’s probability allocation towards the pecular tongue of dweeting. To see whether it had worked, I measured the surprisal of idiosyncratic tokens before and following training. Surprisal is merely the negative log probability a example assigns to a token stated everything preceding it. The small apt a token is, the additional amazing it is to the model.
In the fig below, darker red indicates greater surprisal. After CPT, much of the code becomes noticeably small amazing (particularly the unfamiliar programming idioms).
Across 30 held out dweets, perplexity cut from 28.63 to 8.57, which method that the example had rotate into substantially improved at predicting the construction of these small weird programs.
Although the generated programs were additional consistent, they motionless lacked a certain “charm”. Fortunately, Dwitter has an “awesomeness” mark for all Dweet. I decided to use it to build a curriculum.
I classified the top 5k dweets by their “awesomeness” scores and constructed a syllabus that gradually favoured the additional appreciated programs. Initially, all dweet had an equal probability of being sampled. Over 400 training steps, I increasingly risen the importance assigned to higher classified programs, until the top 1000 accounted for approximately 67% of the sampling probability.
| Rank bucket | Step 1 | Step 100 | Step 200 | Step 300 | Step 400 |
|---|---|---|---|---|---|
| 1-1000 | 20.00% | 35.89% | 48.74% | 59.01% | 67.23% |
| 1001-2000 | 20.00% | 27.97% | 29.60% | 28.01% | 24.99% |
| 2001-3000 | 20.00% | 20.03% | 15.23% | 10.41% | 6.75% |
| 3001-4000 | 20.00% | 12.06% | 5.63% | 2.41% | 0.99% |
| 4001-5000 | 20.00% | 4.05% | 0.81% | 0.16% | 0.03% |
This turned out to be a breakthrough! The generated continuations became much additional visually interesting.
Interestingly, the improvements in token likelihood extended equal to Dweets that had never been sampled during syllabus training. Perplexity cut from 7.42 to 5.65, although the gains were dispersed fairly evenly throughout awesomeness ranks.
Across 7000 Dweets, syllabus training improved token likelihood throughout all awesomeness ranks, including programs never sampled during training. The gains were particularly pronounced for programs the first example established difficult.
I now had a example that could create fairly beautiful continuations. I wondered whether I could obtain this additional by incorporating my own imaginative preferences.
In mid-September, I came throughout Surya’s part on Training AI to Paint alongside Code, anywhere they had used RL to create watercolor paintings using p5js. They experimented alongside distinct reward functions before settling on pairwise comparsions against a gathering of hand rated paintings. I remembered that idea and idea of trying item akin alongside Demoscener (it doesn’t awareness alright to call it “the model” repeatedly).
I opted to do RLHF using GRPO, alongside myself acting as the reward function.
For all unfinished Dweet, I sampled a collection of eight completions, rendered them flank by flank and selected the one I established most aesthetically pleasing. If none appealed to me, I skipped the group. I used these preferences to update Demoscener’s policy, afterward sampled caller continuations and repeated the procedure complete approximately 9–10 rounds of 500 groups each. Altogether, I manually inspected about 40,000 animations.
Somewhere about the 10,000th animation, I began to value the beingness of VLMs and HPSv3. Sadly, it was too late.
After multiple rounds of training, I wanted to see how much of my sensation had really rubbed off on Demoscener. I enlisted fairly a unwilling companion to fairness 400 sets of demos from a few of the checkpoints (I picked 4 randomly), without knowing which example had produced which.
The results showed that checkpoint 4 was the most favorable, so I used that to run conclusion on my Mac.
Now coming to the end of the flank quest, I built a small interface anywhere I could create multiple continuations from the identical prefix, render them flank by flank and branch off from whichever one caught my eye. I could afterward tweak the code, create additional continuations, and keep exploring.
Each branch represents a imaginable continuation, which I could difference visually, tweak by hand, and use as the starting item for additional exploration.
There is item fairly beautiful concerning being capable to roam through these possibilities. I think I'll be spending a small additional period doing fair that.
You can discover Demoscener-7B and the dataset on HuggingFace.
