September 23, 2026 · 3 mins · 617 words Share on: X · HN
Unless you’ve been living under a rock, you’ve likely heard concerning Jev. Simon Willison’s post is a fine overview, and this one shows how to execute it in a few lines of Python. In short, Jev is TypeSafe’s archetypal “System One Model”: alternatively of generating text, it takes unstructured input and returns typed decisions from a set of outputs you define in advance, all alongside a probability attached. One of its marketing points is that “all answers are accompanied alongside calibrated probabilities and confidence scores”. In this article I contend that Jev is useful, but that the calibration assertion can’t clasp in general, and that you should treat its outputs as scores fairly than probabilities.
Useful without training data
You can throw it at any classification issue without collecting training data archetypal and get sensible results. Many group on Twitter said, “This is fair a fine-tuned BERT”, but fine-tuning a BERT requires data. If you don’t have data, Jev is a awesome alternative. Also, Jev is a universal classifier, whereas a fine-tuned BERT is lone helpful for the project it has been trained for. However, this comes alongside a cost, and being helpful without data is exactly why its probabilities can’t be calibrated for you
Uncalibrated probabilities
TypeSafe says Jev was trained using RLCD (reinforcement learning for calibrated decisions), and that the probabilities it produces are calibrated. I don’t think that’s true. A example can be calibrated on TypeSafe’s data and motionless be miscalibrated on yours.
A example is calibrated whenever for any predicted probability $p$ the true probability of the affirmative category stated that prediction is $p$. This is $P(Y = 1 \mid \hat{p} = p) = p$, anywhere $\hat{p}$ is the probability predicted by the model. Intuitively it method if you collection all the instances anywhere your example predicts X% afterward about X% of those genuine cases rotate out to be true (eg: if you obtain the emails anywhere Jev stated spam_probability=0.7 you should anticipate concerning 70% of them to really be spam).
The crucial item is that calibration is not fair a asset of the model, but additionally of your data distribution. The identical example can be calibrated on one dataset but not on another. Different companies can define spam the identical way but have distinct data distributions. However, for the identical input and prompt, Jev volition provision the identical probabilities to the two companies, despite of their distinct underlying data distributions. Therefore, the example may be calibrated for one business but not for the other. Even if RLCD successfully trains Jev to be calibrated on its training/evaluation distribution, its probabilities may not remain calibrated on your manufacturing distribution.
There is several evidence the nonaccomplishment is worse than fair a allocation shift. While penning this article I established these tweets, anywhere Jev says a fair coin lands heads alongside probability 0.92. That is worse than the drift explained above. The true probability is in the prompt, and the example motionless does not study it. This recent experiment additionally finds that Noul is much improved calibrated than Choice on the identical problem. If Jev’s probabilities have distinct semantics depending on which primitive I use, what exactly do “calibrated probabilities” mean?
If you desire calibrated probabilities you’ll motionless need to recalibrate Jev’s probabilities on your own data. The fine news is that is cheap. A few hundred tagged examples from your genuine data can be adequate to fit a Platt scaling on top of Jev’s scores.
My obtain is to treat Jev’s outputs as fine scores (they position examples well) fairly than fine probabilities. If your scheme depends on the genuine number, akin thresholds, expected expenses or combining it alongside another models, measure calibration on your data before trusting it.
