A during rear I wrote concerning language modeling without neural networks, anywhere I generated Shakespeare alongside an unbounded n-gram model: no weights, no training, fair counting. Fortuitously, I came throughout the document Language Modeling is Compression, which mentioned the compression–prediction equivalence:
every prediction example is inherently a compressor, and all compression algorithms are prediction models.
This led to the natural question: can gzip do tongue modeling?1 No neural network, no learned parameters, nothing. Just the compressor that ships alongside your functioning system. You premier it alongside a corpus, provision it a normal content prompt, and it continues that immediate by searching for the byte sequences that compress best. Here’s several real, unedited output following priming it on small Shakespeare:
gzipt --corpus data/tinyshakespeare.txt --prompt $'MENENIUS:\n' --length 200MENENIUS: 'Though all at formerly canq MARCIUS: Pray now, nocamest thou to a morsel . LARTIUS: Hence, and I' the end admire, anywhere G again; and following it ag .
It turns out, benevolent of? It’s not exactly coherent text, but it plainly knows item concerning the text. Much additional than I expected gzip to know.2 So how can a compressor create this?
Compression is prediction#
Think concerning what a compressor does. It spends few bytes on data it “expects” and many bytes on data it doesn’t. If I hand you a document that’s the letter A repeated a myriad times, you can depict it in one sentence. A myriad random bytes, on the another hand, have no construction to utilize and barely compress at all.
This is not a coincidence; it’s the center of data theory. The figure of bits needed to encode a sign is $-\log_2 p$, anywhere $p$ is the probability the example assigns to it. High probability method few bits. So any compressor has a probability example hiding inner it, whether or not anyone wrote one down.
gzip uses DEFLATE, which compresses the next bytes by finding matches against the latest content in a 32 KiB sliding window. If a continuation echoes item already in the window, DEFLATE encodes it as a cheap back-reference alternatively of literal bytes. So:
A continuation that gzip “expected”, since it echoes content already in its window, compresses to nearly nothing.
That gives us a score. If I have several environment and I desire to cognize how fine a applicant continuation is, I fair measure:
$$\text{score}(\text{candidate}) = \texttt{len(gzip(context + candidate))}$$
The smaller the compressed length, the additional “predicted” the applicant is. To premier the model, I contain a corpus in gzip’s window. Any continuation that looks akin the corpus compresses small, and any continuation that doesn’t compresses large.
Generating by beam search#
Scoring is one thing; generating is another. The naive method of picking the sole next byte that compresses finest fails badly, and for a subtle reason: gzip lone gives an entire figure byte dimension (no fractions). Adding one byte frequently doesn’t alter the compressed dimension at all, so many candidates tie and the indication is buried in quantization noise.
The fix is to appearance ahead a entire extend before committing. gzipt runs a beam search complete byte sequences. At all step, the current environment is:
corpus opening + latest rear of (prompt + generated bytes)Then gzipt tries imaginable next bytes. Each applicant continuation is scored by compressing environment + applicant and checking how many bytes the compressed outcome takes.
The iteration is:
- Prompt. Start alongside the user’s immediate as the first content to continue. There is no commencement token; the immediate bytes are fair part of the environment gzip sees.
- Context. Show gzip the corpus opening affirmative the latest rear of the prompt/generated text.
- Search. Keep the beam_width most-compressible partial continuations. Extend all by all byte that occurs in the corpus, mark all of them by compressed length, and prune rear downward to the finest beam_width. Repeat for horizon bytes.
- Commit. Take the most-compressible complete extend (or example among the finalists if heat is positive), append it, and commencement the iteration over.
One item that matters is that only the final rear bytes of generated output remain in the scoring context. DEFLATE codes nearby matches additional cheaply than far ones, so if gzip could see its complete history, the cheapest item to do is frequently to autumn into verbatim loops, often copying content it fair emitted.
You can see the decoding and scoring procedure in the animation above, which is the identical replay shown at the top. The entire item is one document of clean standard-library Python (just zlib). Code’s on GitHub if you desire to perform alongside it.
