Last week TypeSafe AI released Jev, which it calls the archetypal “System One model”: a example that does not chat, does not write, and does not logic stage by step. It answers organized questions concerning an input, in a sole onward pass, alongside a probability attached to all answer. Most of the safety has focused on speed. I think the additional engaging assertion is the one concerning calibration, since calibration is the item that has quietly constricted all manufacturing classifier I have shipped, including the one in my COMPSAC paper.
This article is my attempt to activity out what Jev really changes, anywhere it fits in a genuine ML stack, and how I average to test the assertion fairly than obtain it on faith.
What Jev is, without the marketingLink to section: What Jev is, without the marketing
Jev is built about three ideas, per TypeSafe’s initiate post:
- Non-autoregressive output. A normal LLM produces its answer one token at a time, and all token depends on the last. Jev emits the complete organized answer at once. That is anywhere the speed comes from: TypeSafe quotes 70–500 ms end to end and “40x–200x faster” than frontier LLMs on equal tasks.1
- Typed questions, not prompts. You dispatch a state (text, organized data, or a communication history) and a set of questions. Each inquiry is one of three types: choice (pick from a set, get a probability per option), score (rate against ordered levels, get a uninterrupted mark and distribution), or noul (a yes/no, returned as the probability the declaration is true).2 Every inquiry in a petition is evaluated in parallel, so adding questions barely changes latency.
- Training for calibration. The example is trained alongside what TypeSafe calls reinforcement learning for calibrated decisions (RLCD). The stated goal is “epistemically honest probabilities” fairly than the human-preference or verifiable-reward objectives that conversation models are tuned on.1
The constraints are fair as crucial as the features. Jev cannot create liberated text. A choice inquiry supports at most 255 options. There is no depiction input yet. Pricing is $0.042 per myriad input tokens alongside output tokens free, and admission is currently by waitlist.1
So it is not a smaller GPT. It is nearer to a extremely fast, extremely broad tabular classifier that says unstructured input and returns a typed decision alongside a confidence you are meant to be capable to trust.
Why calibration, not accuracy, is the genuine bottleneckLink to section: Why calibration, not accuracy, is the genuine bottleneck
Here is the part of my own document I keep coming rear to. We predicted whether a drag petition would be merged, using lone signals accessible at submission time. Random Forest hit an F1 of 0.958. The majority-class baseline, which says “merged” to everything, hit 0.957. The figure that really divided a helpful example from a useless one was ROC-AUC: 0.676 for the timber versus 0.500 for the baseline. And equal at that, we wrote plainly that the models “should not be treated as absolutely calibrated probability models” and were fit for triage, not for automated accept/reject decisions.3
That is not a quirk of one dataset. It is the normal form of a manufacturing classifier:
- Accuracy saturates early. On imbalanced problems, most of the accessible accuracy is free. The difficult part is the ranking and the confidence.
- Downstream logic needs probabilities, not labels. “Route this command to manual assessment if the example is small than 80% sure” lone plant if 80% method 80%. If the example says 0.95 on things that are correct 70% of the time, all threshold you set is a lie.
- Miscalibration is invisible in the customary metrics. F1, accuracy, equal AUC are all threshold or position metrics. A example can have a fine AUC and awful calibration, and you volition not cognize until the endeavor regulation built on top of it starts misfiring.
The norm fixes are post-hoc: Platt scaling, isotonic regression, heat scaling. They work, but they are another fitted component that drifts whenever the data does. What Jev is claiming, if I peruse it correctly, is that the probabilities arrive out of the example already honest, since honesty was the training objective. If that holds on tasks exterior TypeSafe’s own benchmarks, it removes a entire tier of glue from manufacturing ML systems.
That “if” is the complete question, and it is testable.
Where a System One example fits in a genuine stackLink to section: Where a System One example fits in a genuine stack
I activity on ML inner a wholesale allocation business. Almost none of it is chat. Most of it is small, repeated decisions that sit between two systems:
| Decision | Today | Why it is annoying | Does Jev’s form fit? |
|---|---|---|---|
| Is this inbound command an elimination that needs a human? | Rules affirmative a small classifier | Rules rot; retraining the classifier is a project | Yes: a noul alongside a threshold |
| Which regulatory merchandise category does this new SKU pertain to? | Keyword rules, manual cleanup | Vendor descriptions are messy liberated text | Yes, if categories fit in 255 choices |
| How urgent is this client assistance message? | Nothing, or an LLM call that takes seconds | Latency and disbursal create it difficult to run on all message | Yes: a score complete ordered levels |
| Which shipment path should assimilate this delayed order? | Constraint solver | Not a classification issue at all | No |
| Write the customer-facing note explaining a substitution | LLM | Needs generated text | No |
The form is clear. Anywhere I have an LLM doing a job that is really classification wearing a conversation costume, a System One example is a plausible replacement alongside two orders of dimension small latency and cost. Anywhere I have hand-written rules that keep breaking since the input is liberated text, it is a plausible replacement for the rules. Anywhere the job is generation or optimization, it is the incorrect tool and TypeSafe says so themselves.
The ERP integration narrative is additionally attractive. A example that returns {"is_exception": 0.93} in 100 ms can sit inside a petition path. An LLM that returns a paragraph in four seconds has to sit beside it in a queue. That difference decides whether ML is a characteristic or a lot job.
The claims I am not prepared to obtain yetLink to section: The claims I am not prepared to obtain yet
A few things in the initiate matter deserve a skeptical reading.
“Zero hallucination.” What TypeSafe can justify is that the output type is continually valid: you asked for one of five categories, you get one of five categories, alongside probabilities that sum to one. That is genuine and useful, and LLM structured-output modes lone approximate it. But it says nothing concerning whether the chosen category is right. A confidently incorrect answer in a valid schema is motionless a incorrect answer. The honest framing is “zero schema errors,” and calibration is what has to shield the rest.
Calibration on whose distribution? A example can be fine calibrated on its training and benchmark allocation and drift seriously on yours. Calibration is a asset of a example and a dataset. The lone figure I volition rely is one measured on my data.
The difference baseline. “200x faster than an LLM on classification” is true and additionally a bit unfair, since the correct baseline for many of these tasks is not an LLM. It is a gradient-boosted tree on engineered features, which is additionally sub-millisecond and free. The engaging difference is three-way: traditional tabular model, LLM-as-classifier, and Jev, on the identical task, on accuracy, ranking, calibration, latency and cost.
The test I desire to runLink to section: The test I desire to run
I have exactly the correct testbed already built: the PR acceptance pipeline from my paper. It is leakage-aware, it has fixed 5-fold splits, and it has a published tree-model baseline alongside a known calibration weakness. Here is the design.
Task. Same as RQ1 in the paper: stated a PR at submission time, foretell merged vs. closed without merge. The Jev state volition be the PR title, body, and the identical submission-time metadata and diff data the trees see, serialized as text. Nothing that appears following submission (comments, CI, afterward commits) goes into the state. The leakage rules do not unwind since the example is new.
Questions. One noul: “This drag petition volition be merged.” Optionally one choice complete the task-intent tags (fix, feature, refactor, docs) to see whether Jev’s own study of intent agrees alongside our keyword rules.
Baselines. The paper’s Random Forest (400 trees), the identical timber alongside isotonic calibration fitted in-fold, and a frontier LLM asked the identical inquiry alongside organized output.
Metrics. Ranking and calibration, not fair F1:
- ROC-AUC, so the outcome is comparable to the paper.
- Brier score, the average squared error of the probability against the outcome:
Brier=1N∑i=1N(p^i−yi)2\text{Brier} = \frac{1}{N}\sum_{i=1}^{N}\left(\hat{p}_i - y_i\right)^2
- Expected calibration error, binning predictions by confidence and measuring how far all bin’s accuracy is from its stated confidence:
ECE=∑b=1B∣Sb∣N ∣ acc(Sb)−conf(Sb) ∣\text{ECE} = \sum_{b=1}^{B} \frac{|S_b|}{N}\,\Big|\,\text{acc}(S_b) - \text{conf}(S_b)\,\Big|
- A reliability diagram per model, since a sole ECE figure hides where a example is over- or under-confident.
- Median and p95 latency, and disbursal per 1,000 PRs.
What would alter my mind. If Jev matches the forest’s AUC and strikes the calibrated timber on Brier and ECE, without any post-hoc fitting, afterward the calibration assertion is genuine on a allocation TypeSafe never saw, and I would commencement moving classification-shaped LLM calls at activity onto it. If it strikes the uncalibrated timber but not the calibrated one, afterward it is a convenience, not a capability. If its AUC is materially lower, the speed does not matter.
I volition publish the numbers either way, and I volition nexus them from here.
What I would inform a squad todayLink to section: What I would inform a squad today
If you are deciding whether to attention concerning Jev correct now, my direction is:
- Inventory your LLM calls. Tag all one as generate or decide. The decide ones are candidates. In my cognition that is most of them.
- Measure calibration on what you already have. Compute Brier and ECE for your current classifiers. If they are bad, you have a issue Jev power solve. If they are fine, you mostly have a latency and disbursal question.
- Do not skip the traditional baseline. A gradient-boosted tree on decent features is the bar. Any new example has to attack it on your data, alongside your leakage rules, or it is not an upgrade.
- Treat “calibrated” as a hypothesis. Test it on your allocation before a endeavor regulation depends on it.
The idea rearward System One models is sound: most of the decisions application needs from ML are small, structured, and latency-sensitive, and a conversation example is a different tool for them. Whether Jev delivers on the calibration commitment is an empirical question. I have the dataset to answer it, and I average to.
Further readingLink to section: Further reading
- Introducing System One Models & Jev, TypeSafe AI
- Building a harness alongside Jev, LangChain, for the request/response shape
- Predicting Pull Request Acceptance the Moment It’s Opened, my write-up of the COMPSAC document this test builds on
- Guo et al., On Calibration of Modern Neural Networks, ICML 2017, the norm citation for ECE and heat scaling
If you have Jev admission and a tagged classification dataset alongside a known calibration problem, I would akin to difference notes. My contact details are on the homepage.
