I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%. I also tested with a fair coin 200 times and got 0.92 probability.

I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.

Write-up: "Jev Does Not Play Dice" https://kantahayashiai.github.io/posts/jev-does-not-play-dic...

I like this post. I haven't had time to dig into Jev (they aren't accepting new signups), but calibrated probabilities is one of their pitches that caught my attention. And I was wondering how does one offer them on user data. Standard calibration essentially ensures that if a score of 0.8 accompanies a positive prediction (assuming the simple case of binary classification), then if you gathered together all predictions with a score of 0.8, around 80% will be correct.

If you have just one example you're sending to a model, how would they guarantee 80% over your data?

FYI, for an overview, scikit's page on calibration is great [1], and my answer on Quora from a long time ago covers a specific type [2].

[1] https://scikit-learn.org/stable/modules/calibration.html

[2] https://www.quora.com/How-is-isotonic-regression-used-in-pra...

This is why I don't understand why everyone's freaking out about it. By far the biggest problem with LLM classifiers is that they treat every individual business as the blurry average of all businesses in their training data. Being lighter is fine if you control for everything else, but at least at my company we would actually have room for a significantly more expensive / slower classifier if it were demonstrably better at following instructions.

This hype is caused by the price and the speed since most people don't know about small fast models and use big models for everything.

Article says that it _can_ be calibrated, it just isn't out of the box. So the title seems contradicted by the body.

> If you want calibrated probabilities you’ll still need to recalibrate Jev’s probabilities on your own data. The good news is that is cheap. A few hundred labeled examples from your actual data can be enough to fit a Platt scaling on top of Jev’s scores.