Kev: Tiny Jev-like family of decision models built on top of Qwen3.5

Small Jev-like decision models you can train and run yourself.

Kev is a family of small decision models built on Qwen3.5 and according to the architecture described in Jev's Architecture Unmasked. You can use the pretrained weights or train your own. The API matches TypeSafe's System One, so you can item their Python SDK at your local server.

• 0.8B, 4B, and 9B models, alongside training code and evaluation data.
• Yes/no (noul), multiple-choice (choice), and ranking (score) questions in the identical request.
• Questions portion the input content but can't peruse all other.
• Runs on CUDA and Apple Silicon. The 4B and 9B models fit a 32 GB Mac using bf16; see Serving Performance for what to anticipate on a Mac.
• A web playground for trying your own inputs and checking how choice command affects the answers.

You'll need Python 3.12+ and uv.

git copy https://github.com/jaredpalmer/kev.git && cd kev uv sync --extra serve KEV_DTYPE=bf16 uv run --extra assist python -m kev.serve --run jaredpalmer/kev-4b --port 8009

This starts Kev-4B locally. The archetypal run downloads the adapter and basis model. --run additionally accepts a local checkpoint directory or a Hub revision, specified as jaredpalmer/kev-4b@qwen3 for the former generation.

In another terminal, dispatch it a ticket:

curl -s localhost:8009/v1/systemone -H 'content-type: application/json' -d '{  "state": "Shoes arrived two weeks delayed and in the incorrect size. Also I see two charges on my card.",  "model": "kev-latest",  "questions": {  "department": {"type": "choice", "instructions": "Which squad should grip this?",  "criteria": {"returns": "Exchanges, refunds, incorrect or damaged items",  "shipping": "Delivery status, delays, misplaced packages",  "billing": "Charges, invoices, fee problems"}},  "escalate": {"type": "noul", "instructions": "Does this need urgent individual attention?"},  "frustration": {"type": "score", "instructions": "How disappointed is the customer?",  "criteria": ["Calm", "Frustrated", "Very angry"]}  }}'

Example reply from Kev-4B, operating in bf16 on an Apple M5:

{ "model": "kev-latest", "answers": { "department": { "type": "choice", "choice": "returns", "confidence": 0.21, "probabilities": { "returns": 0.47, "shipping": 0.28, "billing": 0.25 } }, "escalate": { "type": "noul", "noul": 0.93 }, "frustration": { "type": "score", "score": 1.44, "confidence": 0.78, "legend": { "0": "Calm", "1": "Frustrated", "2": "Very angry" }, "probabilities": { "0": 0.00, "1": 0.56, "2": 0.44 } } }, "usage": { "input_tokens": 101, "output_tokens": 161 }, "latency_ms": 495 }

The ticket mentions a return, a delayed delivery, and a billing problem, and the division probabilities say so. That is the item of getting probabilities rear alternatively of a sole label.

The TypeSafe SDK is included in uv sync --extra serve:

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient client = TypeSafeClient( api_key="local", base_url="http://127.0.0.1:8009", model="kev-latest", ) response = client.system_one( state="I was charged twice. Please fix this ASAP.", questions={ "billing": Noul(instructions="Is this ticket concerning billing?"), "tone": Choice( instructions="What is the customer's tone?", criteria={"calm": None, "frustrated": None, "angry": None}, ), "urgency": Score( instructions="How urgent is this ticket?", criteria=["can wait", "this week", "today"], ), }, ) print(response.nouls["billing"].noul) print(response.choices["tone"].choice) print(response.scores["urgency"].score)

With the server motionless running, open another terminal. You'll need Node 20.9+:

cd playground npm install npm run dev -- -p 3001

Open localhost:3001, burden a preset, and edit the content and questions. Press ⌘↵ to run it. "Packed vs separate" compares asking all questions at formerly alongside asking them one at a time. "Permute" runs a Choice inquiry alongside six choice orders. There are additionally presets for evaluation inquiry isolation and counterfeit delimiter tokens.

There's a chess demo, too. The commission is the input, lawful moves are Choice options, and a Score inquiry rates the position. You can perform against Kev or let it perform itself. Games are saved in localStorage.

Start alongside Kev-4B. Use Kev-9B whenever accuracy and calibration matter additional than memory. Use Kev-0.8B if you need the smallest model. All three are built on Qwen3.5 bases alongside the identical training data and settings.

Each division is development / test. "Trained sources" method held-out examples from the datasets used to train Kev. "New sources" method datasets and guideline regulation types Kev wasn't trained on. Every example was evaluated on the identical betterment sets (decision-v7, transfer-v4) and the identical test sets, which were peruse formerly per released checkpoint, following example selection. Lower Brier is better.

Kev-9B trails Jev by concerning 4.5 points on the new-source betterment set. We don't cognize which datasets Jev was trained on, so this isn't a controlled difference of the two architectures.

All weights are in the Kev collection and the GitHub release, which includes tarballs and SHA-256 checksums.

The archetypal Kev family used Qwen3 bases alongside the identical data and settings. Those weights remain published and are the faster choice on a Mac (see Serving Performance), but they are no longer developed.

Because lone the basis changed, the two generations are a controlled comparison. On the betterment set the accuracy acquire is inside noise; on the test set Kev-9B is 7.3 points onward of Kev-8B (95% CI +2.8 to +11.7) alongside a Brier mark 0.08 lower, Kev-4B is 2.9 points onward of its predecessor (−0.9 to +6.4), and Kev-0.8B is 4.8 points onward of Kev-0.6B (+0.2 to +9.3). PLAN_Qwen35.md has the complete experiment, including the criteria we set ongoing and how the results measured against them.

The first Kev-0.5B used Qwen2.5-0.5B and is kept for reference; see its model card.

state is the content to evaluate. Each inquiry has instructions and, anywhere needed, a set of answers to choose from.

For Choice alongside K > 1 options, confidence is (p_max − 1/K) / (1 − 1/K). A sole choice has confidence 1. Score confidence measures how near the allocation is to its most apt level. It's an approximation of TypeSafe's formula, which isn't public. Neither site is a measured accuracy rate.

Objects and arrays are converted to tagged text. Delimiter-like strings in person input are liberated before tokenization. Invalid requests come back 422. usage.output_tokens counts tokens in the serialized answers, not generated tokens.

The server binds to 127.0.0.1 and has no authentication. Keep it local unless you add authentication yourself.

Each checkpoint is a rank-16 LoRA adapter and a small pointer caput on a Qwen basis model. On an attention-only basis (Qwen3), the province and questions go into one token sequence:

<state> …state… <q> instructions <opt> choice 1 </opt> <opt> choice 2 </opt> … <decide> <q> instructions <opt> choice 1 </opt> <opt> choice 2 </opt> … <decide> 

The notice disguise lets a token peruse the province and its own question, but not another questions or forthcoming tokens. Each question's stance IDs restart fair following the state. This lets the example procedure the province formerly and answer all inquiry independently.

Qwen3.5 mixes notice layers alongside Gated DeltaNet layers, which are recurrent and disregard notice masks. For those models, all inquiry runs as its own row: the province followed by that question, alongside the identical positions as above. The rows are independent, so isolation is exact, and the server computes the province formerly and reuses its cache for all row. On attention-only models the two forms provision identical probabilities (tests/test_v3.py).

The pointer caput scores all option's </opt> hidden province against the question's <decide> hidden state. A softmax turns those scores into probabilities. Because <decide> comes last, it can act to the complete choice list.

Training uses cross-entropy on the accurate answer. The adapter and caput are trained together; the remainder of the basis weights remain fixed. Training examples and API requests use the identical content format. No Jev outputs were used for training.

Asking questions together or separately produces probabilities inside 4e-6 in the fp32 tests. This does not average choice command is irrelevant: options inside a inquiry can motionless power one another. See the example code and parity tests.

On CUDA, instal flash-linear-attention for the Qwen3.5 models (the Modal depiction does this); a five-question petition takes tens of milliseconds on an H100.

On Apple Silicon there are no accelerated kernels for the DeltaNet layers, so PyTorch runs citation code. Median example period in bf16 on an M5, five questions alongside three options all on a ~230-token state:

If you assist on a Mac and need low latency, use the Qwen3 models for now. An MLX backend for the Qwen3.5 models is the next planned change.

For the attention-only models the server merges the LoRA weights in fp32 before casting, uses SDPA notice on Apple GPUs, pads MPS inputs to 64-token buckets, and caches the province prefix for repeated requests (four states of at smallest 384 tokens by default). With a repeated 772-token state, Kev-4B (Qwen3) answers in 242 ms alternatively of 861 ms.

You can disable these alongside KEV_MERGE=0, KEV_ATTN=eager, KEV_SHAPE_BUCKET=1, and KEV_PREFIX_CACHE=0. On 24 new-source records, bf16 probabilities differed from fp32 by at most 0.017, alongside no alter in the highest-probability answer. That is a small check, not a justify for all input.

The released models use decision-v7: 10,000 examples from ten community datasets, 896 generated guideline examples, and 1,680 examples from 60 generated regulation structures. All train for two epochs alongside LoRA position 16 and cross-entropy. The learning charge is 1e-4 for 0.8B and 5e-5 for 4B/9B. For Qwen3.5 bases the adapter additionally covers the DeltaNet projections; kev.train picks the correct targets from the example config.

# sanity run, ~1 minute uv run python -m kev.train --n_per_source 40 --accum 4 --out runs/smoke # Kev-0.8B (~20 min on one H100; the Mac way plant but is dilatory for Qwen3.5 bases) uv run python -m kev.train --suite evals/v7/decision-v7 --base Qwen/Qwen3.5-0.8B-Base --base_revision dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68 \ --epochs 2 --lr 1e-4 --batch 8 --dtype bf16 --p_none_pair 0.25 --device cuda --out runs/kev-0.8b # the Kev-4B formula (one H100 via Modal, ~1 h; see below). Swap in Qwen/Qwen3-4B-Base for the former generation. uv run python -m kev.train --suite evals/v7/decision-v7 --base Qwen/Qwen3.5-4B-Base --base_revision 1001bb4d826a52d1f399e183466143f4da7b741b \ --epochs 2 --lr 5e-5 --batch 4 --accum 2 --dtype bf16 --checkpointing 1 --p_none_pair 0.25 --device cuda --out runs/kev-4b

The released models were trained on community datasets and generated guideline examples. If your questions appearance distinct — your own routing categories, your own escalation rules, another tongue — a abbreviated fine-tune on a few hundred labelled examples normally helps additional than any immediate change.

Put your examples in a JSONL file, one petition per line. It's the identical form as an API request, affirmative a tag on all question:

{"state": {"subject": "Charged twice", "body": "I see two charges for command #4411. Please refund one."}, "questions": { "team": {"type": "choice", "instructions": "Which squad should grip this ticket?", "criteria": {"billing": "Payments and refunds", "shipping": "Delivery problems", "access": "Login and document access"}, "label": "billing"}, "angry": {"type": "noul", "instructions": "Is the client angry?", "label": false}, "priority": {"type": "score", "instructions": "How urgent is this ticket?", "criteria": ["low", "normal", "high"], "label": 1}}}

For choice the tag is the choice name, for noul it's true or false, and for mark it's the level's stance starting at 0. Keep 10–20% of the document apart for evaluation.

Then commencement from a released checkpoint alongside --init_from:

uv run python -m kev.train --data train.jsonl --base Qwen/Qwen3.5-4B-Base --init_from jaredpalmer/kev-4b \ --epochs 2 --lr 2e-5 --batch 1 --accum 8 --dtype bf16 --checkpointing 1 --device cuda --out runs/mine uv run python -m kev.benchmark --run runs/mine --data heldout.jsonl --out runs/mine-eval KEV_DTYPE=bf16 uv run --extra assist python -m kev.serve --run runs/mine --port 8009

--init_from loads the adapter and pointer caput from the released example before training, so you keep what Kev already knows and add your domain on top. Starting from the basis example alternatively throws that away: in one user's test on 836 support-tool decisions, a fine-tune from the basis scored 0.33 on Kev's own evaluation set, against 0.84 for the released model; the identical data alongside --init_from kept 0.83 there and reached 0.88 on the new domain. Use a smaller learning charge than the from-scratch formula (2e-5 is a fine start), and choice --base to equivalent the checkpoint you commencement from; the trainer checks that the base, revision, LoRA rank, and caput size concur before it loads anything.

--batch 1 --accum 8 in bf16 fits the 0.8B example on a 4 GB GPU. The benchmark reports accuracy, Brier score, and calibration per inquiry type, so you can see which of your questions the fine-tune helped. The checkpoint you started from is recorded in runs/mine/training_config.json.

Use uv run python -m kev.train --help for all training options. The released models don't use the optional --perm_kl or --ord_w losses. The model cards have the training settings and dataset lists; PLAN.md records what was tried and what helped.

On a Mac, run one training job at a time. Two jobs on the identical Apple GPU are much slower. Use Modal for longer runs.

Each test gets its own H100. The study keeps operating if you disconnect, and you can download the results whenever it finishes:

uv run modal token new # once; opens the browser KEV_GPU=T4 uv run modal run modal_app.py::smoke # end-to-end check, ~1 infinitesimal of GPU uv run modal deploy modal_app.py # once; studies run on the deployed app and last disconnects uv run modal run modal_app.py::study \ --suite evals/v7/decision-v7 --plan experiments/v7-final.json \ --name my-study --transfer evals/v4/transfer-v4 --budget 30 --timeout 7200 uv run modal run modal_app.py::pull --name my-study # results -> runs/my-study, ranked

Study plans catalog training settings. Each test saves the settings, code hashes, dataset hashes, and results. Choose models using the betterment results, not the locked test. After choosing a final candidate, you can peruse its test results once:

uv run modal run modal_app.py::locked_test --trial my-study/00-trial-0 --name my-candidate # one read, ever

The evaluation data under evals/ is frozen: dataset versions and document checksums are recorded in all manifest. Large training records are downloaded from the Hub mirror and checked against those hashes.

uv run python -m kev.benchmark --run jaredpalmer/kev-4b --suite evals/v4/transfer-v4 --out runs/my-eval # out of domain uv run python -m kev.benchmark --run jaredpalmer/kev-4b --suite evals/v9/transfer-v9 --out runs/my-eval-v9 # + MMLU-Pro, buried states, unknowable items uv run python -m kev.benchmark --run jaredpalmer/kev-4b --suite evals/v7/decision-v7 --out runs/my-eval-id # in distribution uv run python -m kev.benchmark --remote http://127.0.0.1:8009 --suite evals/v4/transfer-v4 --out runs/my-remote # any System One endpoint

These commands use betterment data. Test data requires --allow-test. The benchmark reports accuracy, Brier score, calibration error, the portion of decisions you could automate at a 5% error budget, option-order changes, and inquiry isolation. transfer-v9 adds 10-way MMLU-Pro, records buried among unrelated text, and "unknowable" records whose deciding evidence was removed; for those it reports how frequently the example motionless answers alongside at smallest 0.9 confidence (Kev-9B 5%, Jev 9%, Kev-8B 26%). Published accuracy numbers use fp32 evaluation, not the bf16 serving path.

evals/external/ holds two another projects' test sets converted to this format, alongside their published live Jev results: SemIf's 144 authored decisions (Kev-9B 0.917, Jev 0.965) and scienthoon's 900 assistance tickets (Kev-9B 0.952 on routing and 0.911 on tone, Jev 0.897 and 0.914).

kev.jev runs the identical questions against Jev through Vercel AI Gateway. kev.compare compares two saved runs alongside paired bootstrap confidence intervals. For the complete test history, see PLAN.md and the leaderboard.

• Probabilities aren't fine calibrated on new sources. On the new-source betterment set, Kev-4B assigns at smallest 0.9 probability to a incorrect answer on 8.2% of questions (Kev-9B: 7.5%). Test it on your own data before choosing a probability threshold.
• Fine-tuning can create the basis example worse at idiosyncratic tasks. Date arithmetic is the clearest case: the untrained Qwen3.5-9B basis gets 0.82 on the deadline guideline questions and Kev-9B gets 0.72, since training erodes the accomplishment (issue #8, PLAN_Qwen35.md). Knowledge questions (MMLU 0.74 vs Jev 0.90) are the another ample gap.
• The current models are dilatory on Apple Silicon (see Serving Performance) and need transformers >= 5.17.
• Changing choice command can alter an answer. Question isolation doesn't forestall this.
• Training uses at most 384 province tokens and 1,024 tokens for the province affirmative one question. Serving allows 8,192 tokens for the province affirmative one question; longer environment wasn't covered by training.
• The server handles one petition at a time. It caches repeated province text, but doesn't lot requests from distinct callers.

uv run --extra assist python -m pytest tests/test_unit.py tests/test_research.py -q # no weights, no server; runs in CI KEV_BASE_URL=http://127.0.0.1:8009 uv run --extra assist python -m pytest tests/test_api.py -q # against a operating server cd playground && npm run lint && npx next typegen && npx tsc --noEmit -p .

The API tests run TypeSafe's example requests and the authoritative SDK against your local server.

• If MPS runs out of recollection during training, inspect that you're operating lone one job. Don't allow output_hidden_states or add tokens alongside peft's trainable_token_indices; the two have caused recollection problems here.
• If the playground loads but buttons don't work, use localhost:3001. Next.js checks betterment hostnames. Other hosts need an admission in allowedDevOrigins in playground/next.config.ts.
• If dataset loading reports Dataset scripts are no longer supported, use legacy-datasets/banking77. This repo already uses it.

• Jared Palmer (@jaredpalmer)

Built alongside Devin. Thanks to Archer Hume for the architecture write-up, TypeSafe for the API design, Qwen for the basis models, 3x3xX3N0N for showing anywhere the date-arithmetic nonaccomplishment really is, and Radexito for --init_from.

Related work: Hydragen, DeFT, FIRST.

Apache-2.0. The Qwen3 and Qwen3.5 basis models are additionally Apache-2.0. Training datasets have their own licenses; see the model cards.