PSSA: A non-transformer tongue example written from scratch in Rust

PSSA is a small tongue example that is not a transformer. It says content one token at a period through a recurrent state-space layer, keeps a financial institution of episodic memories it can appearance things up in, and rewrites part of its own weights during it runs. It is written in Rust from scratch, alongside no PyTorch, no TensorFlow, and no ML example of any benevolent underneath it.

At matched parameters and on the identical corpus, it learns faster than a transformer and generates content concerning dozen times quicker on the identical CPU.

A transformer scores all brace of tokens in the context, so its disbursal per step grows alongside the quadrate of the sequence dimension and the entire environment is re-read at every step. PSSA carries one fixed-size province alongside the sequence in a single left-to-right pass, and looks things up in a recollection financial institution alternatively of re-reading the context, so disbursal grows linearly alongside length.

Two models, identical corpus, identical tokenizer, identical optimizer schedule, identical seed, same figure of parameters. One is PSSA, one is a norm transformer. Over 12.7M tokens of cleaned WikiText-103:

PSSA completed at 3.98 training cross-entropy, the transformer at 4.43. That is a gap of 0.45 nats, perplexity 53.7 against 83.7. The transformer spent its complete 12.7M-token prosperity to attain a defeat PSSA had already passed around 2M tokens in.

The two curves never cross, and they never touch:

Training defeat lone says a example fit the stream it was fed. So the two checkpoints were scored on a 198,939-token piece cut from a part of the corpus neither run ever touched:

Every checkpoint of the two runs, 64 PSSA links and 43 transformer links, scored on a bounded 9,934-token opening of that unseen slice. The curves never cross: PSSA is onward from the archetypal nexus and finishes 0.51 nats lower. The array below is the final checkpoint of all run on the complete slice.

The held-out gap, 0.43 nats, is basically the training gap. PSSA is not memorizing harder, it is generalizing better.

Generating 200 tokens on the identical CPU, identical prompt, identical sampler:

A recurrent example carries a fixed-size state, so the disbursal of all new token does not develop alongside the dimension of what came before. A transformer re-reads its whole context all step.

• A recurrent state-space core. Learned uninterrupted province matrices carry information onward in a fixed-size state, alternatively of notice complete the full context window.
• An episodic recollection bank. 512 slots alongside hyperbolic (Poincare-style) retrieval and bounded top-4 search, written to and peruse from during the run.
• Plastic weights. Fast updates fortify what works, novelty drives growth, and a refractory entrance rate-limits overwrites so repeated contradictory input does small damage.
• Closed-form consolidation. A ridge-regression stage folds the accelerated plastic updates rear into the basis passage matrix, the way sleep consolidates a day's learning.
• No framework. Hand-written linear algebra in Rust, alongside a CUDA way for training and a scalar CPU citation that all gradient is checked against (max gradient difference 2.98e-8).

Being direct concerning the scale, since the numbers complete are uncomplicated to over-read:

• These are 1.5M-parameter models on 12.7M tokens. That is a research prototype, not a competitor to item you have heard of.
• Text norm at this measure is mediocre for the two models. PSSA emits "a barget of the Prian Academy", the transformer "a matter circulation of the United States". The difference is concerning learning efficiency, not fluency.
• The speed difference is CPU-to-CPU, which is fair. The training throughput numbers additional downward are not hardware-matched and should not be peruse as an architecture result.
• Two experiments are motionless unmeasured: preservation of before skills following a corpus switch, and whether ablating the recollection financial institution changes the loss.

git copy https://github.com/Sparticle62ops/pssa.git cd pssa cargo build --release ./target/release/oxide_ai_pssa

Running it alongside no arguments gives you a residence display listing all command plus any checkpoint and corpus it finds in the operating directory.

The entire outcome complete was trained on a liberated hosted notebook alongside a single entry-level GPU, in 200,000-token links, since a meeting gets cut following a few hours. Every engaging inquiry left, whether the gap holds at 10x or 100x these parameters, whether the recollection financial institution matters at scale, how it does against a contemporary recurrent baseline, needs one thing: a GPU alongside genuine VRAM and allocations measured in days alternatively of hours. Anything meaningfully complete the entry-level cardstock this ran on changes what can be asked.

If you have compute to grant, or you activity location that does, that is the single highest-leverage item anyone can recommendation this project.

Sponsorship finances compute and nothing else. In come back you get named current and in the write-up of any outcome your hardware made possible. Get in contact before sending item so the particulars can be agreed.

Issues and drag requests are welcome. The parts most in need of hands: kernel performance, a contemporary recurrent baseline to difference against, and evaluation beyond next-token loss. Validate any branch alongside cargo test --release before opening a PR.

[email protected]

Solana: 4XPZ9uAa2BMoth6msoHRxTWL4mUrMfq3LGrxbAGja96h

Everything below is for running, training, and operating on the project.

• Rust toolchain alongside Edition 2024 support, including Cargo.
• Network admission lone whenever using an HTTP/HTTPS dataset or a Hugging Face dataset.
• Enough recollection and disk for larger corpora and serialized models.
• Optional: a CUDA equipment for the GPU training path. The CPU way is the reference and continually available.

Direct runtime requirements are ureq for dataset downloads and tokenizers for byte-level BPE.

Both chains ran 64 links of 200,000 encoded tokens, all nexus resuming from the previous checkpoint, so the learning-rate agenda and optimizer province continue across the entire run alternatively of restarting per link.

• Identical corpus: one clean-wikitext continue complete WikiText-103, reused byte for byte.
• Identical token IDs: the baseline pins --tokenizer-from to the PSSA chain's own checkpoint, so neither example sees a distinct vocabulary.
• Identical optimization: 30,000-update cosine horizon, no warm-up restart, 512 supervised mark tokens per update, kernel 42.
• PSSA: latent 256, recurrent province 16, 512 recollection slots, key width 32, vocab 2,048.
• Baseline: 1,541,120 parameters, 1 layer, width 256, 4 heads, FFN 448, vocab 2,048.

End-of-link training cross-entropy:

The baseline's archetypal meeting was cut at nexus 43 by the notebook meeting limit and its defeat CSV did not survive, so links 1 to 43 are peruse rear from that session's own run log instead. The sequence resumed from ck43 in a second session and completed all 64 links, and the two curves complete now shield the complete run.

PSSA trained on a Kaggle T4 at approximately 900 tokens/second. The baseline is CPU-only, since train-transformer has no GPU path, and held 212 tokens/second. Those two numbers say nothing concerning the architectures. On the same CPU-only Kaggle hardware the batched PSSA way measures 375 tokens/second against the baseline's 212, and the defeat difference complete is unaffected either way, since it is matched on tokens and updates fairly than on time.

The losses are end-of-link training cross-entropy on the stream being fit, not held-out evaluation. For a held-out difference on an unseen slice, use the compare command described in docs/COMPARISON.md. Generation norm at this measure is mediocre for the two models: PSSA emits "a barget of the Prian Academy", the baseline "a matter circulation of the United States".

Two experiments are not yet measured: preservation of before skills following a corpus switch, and whether ablating the 512 recollection slots changes loss.

bash kaggle/kaggle_continue.sh # the PSSA chain bash kaggle/kaggle_transformer_baseline.sh # the parameter-matched baseline

Both peruse TOTAL, WINDOW and FRESH from the surroundings and write --loss-csv, so the curve survives a cut session.

General form:

oxide_ai_pssa <COMMAND> [OPTIONS] 

Commands:

Options:

Positional arguments and long/short options can be mixed:

cargo run --release -- train data/downloaded.txt -e 2 -o data/experiment.pssa cargo run --release -- train --data data/downloaded.txt --epochs 2 --out data/experiment.pssa

Inside the REPL:

• /exit or discontinue exits the process.
• /info prints the loaded example path, recollection slot count, and adapter count.
• /temp <value> reports a heat value but does not use it to afterward turns. Pass --temp whenever launching conversation instead.

--skip-tokens, --max-tokens and --resume together let a lengthy corpus be trained as a sequence of abbreviated runs, so a sole run never has to last a meeting limit. If a opening crosses EOF, choice wraps to the commencement of the corpus. Each nexus trains its own opening and hands its optimizer province to the next:

cargo run --release -- train data/downloaded.txt -e 1 \ --skip-tokens 0 --max-tokens 200000 -o chain/ck01.pssa cargo run --release -- train data/downloaded.txt -e 1 \ --skip-tokens 200000 --max-tokens 200000 --resume chain/ck01.pssa -o chain/ck02.pssa

kaggle/kaggle_continue.sh drives this form end to end: it sets a opening size and a nexus count, walks the corpus offset by offset, and resumes all nexus from the former checkpoint. position afterward reports all checkpoint in the sequence alongside its form and optimizer stage count.

DatasetManager accepts one or additional comma-separated sources:

cargo run --release -- train discipline # built-in citation corpus cargo run --release -- train data/downloaded.txt # local content file cargo run --release -- train data/ # all readable document in a directory cargo run --release -- train https://example.org/corpus.txt cargo run --release -- train hf:owner/dataset # Hugging Face repository cargo run --release -- train science,data/downloaded.txt # multiple sources

Local records and directories are peruse directly; HTTP(S) URLs and definitive hf:owner/dataset sources are downloaded. Structured responses are reduced using average sectors specified as text, content, article, story, instruction, output, sentence, and summary; structured responses without a supported content site are rejected.

Byte-level BPE keeps exact UTF-8 case, whitespace, punctuation, and row endings, and has a complete 256-byte fallback alphabet, so valid UTF-8 never collapses to <unk>. The former lowercase term splitter, including its 10,000-word cap and <unk> behavior, is accessible lone alongside --tokenizer word.

Download a Hugging Face dataset into a local content file:

cargo run --release -- download wikimedia/wikipedia --out data/downloaded.txt

Network downloads are not validated or curated by Oxide AI. Review licensing, privacy, and satisfied before training on an external corpus.

Clean extracted wikitext-103-raw content before a caller training run:

./target/release/oxide_ai_pssa clean-wikitext wiki.train.raw --out data/wikitext-clean.txt ./target/release/oxide_ai_pssa train data/wikitext-clean.txt -o data/model.pssa # Also available: oxide_ai_pssa assistance clean-wikitext

The identical command can be used in Kaggle following extracting content from Parquet; it accepts a local UTF-8 content file, not Parquet itself. -o and --out are aliases. The output way is required and must not already be (including the input path or a nexus to it). This protects the first corpus; choose a new output name for another run. Read, UTF-8, and compose failures exit nonzero through the normal CLI error path, alongside partial output removed whenever possible.

The pass:

• Joins @-@, @.@, and @,@ to neighboring text: visitant @-@ leading → guest-starring, 52 @.@ 9 → 52.9, 500 @,@ 000 → 500,000.
• Drops stable heading lines specified as = Title = and = = Section = =.
• Removes <unk> and collapses remaining inline whitespace to sole spaces.
• Removes spaces before ., ,, ) and following (; trims all line.
• Retains at most one successive blank line, including at the start/end. Removing a heading does not current a blank line.
• Writes LF row endings, including a newline on the final retained line.

oxide_ai_pssa::dataset::clean_wikitext(reader, writer) is the reusable library API (BufRead / Write, returning std::io::Result<()>). The CLI uses buffered file I/O, and the cleaner retains lone its input/output row buffers: recollection is proportional to the longest line, not the corpus size. Library callers using a buffered author must flush it themselves; the CLI explicitly checks the flush. No new requirements are required.

Cleaning is opt-in: existing loaders, tokenizers, training commands, and kaggle/kaggle_continue.sh are unchanged. Do not toggle an in-flight resume chain to a cleaned corpus: cleaning changes token IDs/counts and the meaning of --skip-tokens offsets. Prepare and consistently reuse one cleaned corpus for a new sequence instead.

The train command performs two phases:

1. Continuous recurrent ingestion: token transitions are processed through the PSSA layer. The example updates state, memory, adapters, and routing behavior alongside a cosine learning-rate schedule.
2. Adapter consolidation: following all epoch, the synthetic adapter's accelerated coefficients are folded into its consolidated coefficients alongside the configured EMA rate.

Defaults are latent 256, recurrent province 16, memory-key 32, recollection capability 512, chunk dimension 64, learning charge 1e-3, 8 chunks per update, and kernel 42. The resulting binary holds weights, configuration, memory, adapters, and optimizer state. It is not an interchange format for another ML frameworks and have to be loaded through PSSALayer::import_from_pssa_bytes.

New saves use V7: the complete V6 training/resume payload affirmative a bounded, length-prefixed norm tokenizer JSON. A V7 BPE checkpoint is self-contained and restores its exact ordered vocabulary without admission to the training or evaluation corpus. create and conversation refuse --data for V7 BPE since retraining a tokenizer on external data would not validate provenance. V7 term checkpoints and V6 checkpoints keep the bequest optional --data exact-vocabulary comparison. Checked V5 artifacts remain inference-only and necessitate --data since they never contained tokenizer provenance.

Generation is autoregressive and uses heat 0.70, a top-24 applicant bounds followed by top-p 0.85 filtering, a 1.25 repeat penalty complete a latest 64-token window, contiguous self-transition suppression, <unk> suppression, and a default cap of 64 new tokens, ending first following two generated periods.

V7 BPE conclusion restores the exact embedded tokenizer and never rebuilds it from a selected dataset. Evaluation supplies its data lone as held-out content to the restored tokenizer.

cargo run --release -- benchmark

The suite exercises synthetic streams for contradictory facts, MQAR-style distractors, burst repetition, example serialization, and abbreviated generation prompts. It prints milestone results, is not wired into Cargo's test harness, and is not a norm evaluation on broad tongue tasks.

cargo fmt --all -- --check cargo clippy --release --all-targets cargo test --release

Integration tests live in tests/: allocations.rs, bpe_repair.rs, checkpoint_repair.rs, core_repair.rs, linalg.rs, and runtime_repair.rs, alongside shared artifacts under tests/fixtures/. They shield tokenizer circular trips, checkpoint import/export throughout versions, linear-algebra kernels, allocation behavior, and CLI runtime output. Clippy is spotless of errors; a figure of manner warnings in the numeric kernels are remaining in location deliberately, since rewriting indexed loops there would churn code the gradient tests pin down.

• CPU-oriented prototype alongside hand-written linear algebra. gpu-probe verifies a WebGPU equipment and a GEMM against the CPU reference, but training and conclusion motionless run the tier math on the CPU.
• The CLI parser is intentionally minimal: no shell-style quoting, and small validation beyond numeric parsing.
• A missing or unreadable dataset silently falls rear to the built-in discipline corpus in multiple loading paths.
• Model and tokenizer vocabularies must remain compatible; a size alert does not repair a mismatch.
• Model form cannot alter throughout a resume chain: latent, state, key, recollection and vocabulary must equivalent the checkpoint being resumed.
• Downloaded satisfied can be ample and may merge JSON, malformed text, or data unsuitable for training.
• The REPL heat command acknowledges a value without changing the energetic configuration.
• Benchmark output is milestone-oriented and does not measure perplexity, factuality, latency, or safety.
• Serialized .pssa records are project-specific binary artifacts without type immigration tooling.

See LICENSE for the project license.