The whole point is the quantified output. If you just ask an LLM to type out its confidence "manually", it'll make up some nonsense. The logprob numbers are more reliable.

I got this technique to work extremely reliably last year. However there were a bunch of caveats: 1) Firstly, you must institute a check that the multiple choice tokens dominate the output distribution. They should sum to 95% or more, ideally 99%, or the LLM is not following instructions properly. This is also the problem with constrained decoding - if the LLM really doesn't want to output a valid answer, the one you extract will not be high quality. 2) You need to ask it multiple times, permuting which option corresponds to which letter, and average the results. LLMs are surprisingly biased towards picking "A", especially if they're otherwise not sure. 3) For the same reason, performance improves if you frame the prompt as if it were the middle of a quiz. "Question 1" carries baggage that "Question 12" doesn't. 4) You must be exceedingly careful with tokenization.

But when all was said and done, I got a general purpose A/B classifier that gave high resolution quantitative output for the cost of a couple dozen tokens ingested and a couple inference passes.

In my experience as well using logprobs to try to quantify uncertainty, LLMs are a poor fit. Neural nets in general struggle with 'calibration' --- ie. if a prediction is truly 50/50, neural nets are often prone to predicting overconfidently [0].

I ran some tests using GPT-4 to do some basic classification a couple years ago. On ambiguous options which had to be escalated to a human, the LLM would regularly output something like a 99.8% probability, compared to 99.99% for a correct answer.

0: https://arxiv.org/pdf/1706.04599

+1, llama.cpp has a --grammar parameter which you can pass a BNF style grammar file to constrain generation. It can be used in Python llama.cpp wrapper

https://til.simonwillison.net/llms/llama-cpp-python-grammars

If we're talking about running it locally, what about passing a partial response as part of the input?

Prompt part: "What is better, toast or bread?"

Incomplete answer part: "The answer to this question is "

and then have the LLM finish the answer. I did this with subtitle translation using llama.cpp (with Python) and had great success. Just past 5 already translated subtitles as the incomplete answer, and the LLM infallibly just continues to translate. No markdown, and usually no talkback if the subtitles contain nasty subjects like bioweapons or nuclear stuff. It just works.

Agreed, it's a real issue, but it can probably be vastly reduced by having the schema in the system prompt and by giving the model an expectation of a fixed value: no decent modern would pick a prose ligament over a provided value.

To completely squash the issue, a few cheap LoRa iterations will do the trick just fine.

Wow! TIL! I've been running a for loop around the two ordering variations to catch the winner of each turn and the difference is quite noticeable. In the options-after-body case in 47 of 100 attempts it classifies as phishing, whereas in the options-before-body case it classifies clearly as rickroll (94 out of 100 attempts)

Payroll sends you an email with a link to a Youtube video that plays a song.

Options after body:

    Average probabilities:
    Rickroll   0.5158 ( 51 wins)
    Phishing   0.4561 ( 47 wins)
    Spam       0.0281 (  2 wins)
    Joke       0.0000 (  0 wins)
    Legitimate 0.0000 (  0 wins)

Options before body:

    Average probabilities:
    Rickroll   0.9293 ( 94 wins)
    Joke       0.0549 (  5 wins)
    Phishing   0.0140 (  1 wins)
    Spam       0.0018 (  0 wins)
    Legitimate 0.0000 (  0 wins)

This was Gemma4-26B-A4B-NVFP4 by the way.

EDIT

Gemma4-12B-it-NVFP4 seems way less sensitive to option/body ordering:

Options after body:

    Average probabilities:
    Rickroll   0.9867 ( 99 wins)
    Phishing   0.0133 (  1 wins)
    Joke       0.0000 (  0 wins)
    Spam       0.0000 (  0 wins)
    Legitimate 0.0000 (  0 wins)

Options before body:

    Average probabilities:
    Rickroll   0.9401 ( 93 wins)
    Phishing   0.0336 (  3 wins)
    Spam       0.0250 (  4 wins)
    Joke       0.0010 (  0 wins)
    Legitimate 0.0002 (  0 wins)

Anyway, this for-looping stuff doing 100 calls to even a local VLLM API takes around 5 seconds in total, so this isn't anywhere close to sub-second Jev territory.

What a time to be alive, repeating questions to a model twice to increase accuracy.

Prompt Repetition Improves Non-Reasoning LLMs: https://arxiv.org/abs/2512.14982

This works very very well :).

https://github.com/Mushroom-Systems/lichen

The parody note at the end is bizarre. Everything above it reads pretty seriously:

> Everyone on Twitter is all over Jev, how it's the next frontier of large language models and the AI paradigm. We don’t really think so.

> nothing about its...

Non deterministic systems have furthered the "brain rot" in our industry.

Lots of people were happy to ignore the code in their "supply chain" before LLM's - but suddenly not reading the LLM's output is a problem. I get they are different but we're in the same realm.

The lack of real data on performance of what ever application that one is trying to pitch is getting appalling. It's a lot of "trust me bro" this works better hand waving. And it's getting gross.

And how do we even measure nondeterministic systems? Because if I told you that Anthropic was spending millions of dollars having 1000's of agents "pre solve" benchmarks to build into their next version of the system you would scream they were cheating. Every one is focused on the "hacking" in the hugging face incident and no one is looking why they were even playing with those benchmarks in the first place.

"Trust me Bro"...

latency and compute comparisons highly depends on your local setup.

you can swith to a better model for lower error rate.

Yeah it says it's a parody, but then in the same sentence it refers to the other "OpenJev" implementations, which are basically the same thing with marginally more effort. And it doesn't imply that those things are parodies too (and I don't think they are parodies).

Somehow the HN crowd has a bunch of "professionals" who don't care about error rates and think that a Qwen model running on a potato is frontier intelligence.

>we didn't call an api

>calls an api

ok

The 25 lines is the only thing that makes Jev different compared to Solution apparently.

Off the top of my head it's 3 things it advertises:

- By not being a optimised for chat, it can deliver confidence for answer and not for how an answer should be phrased

- Speed. It can take seconds for OpenAI to compile schemas, jev can respond before openAI has even begun thinking

- Token efficiency and price. I think its the output token they don't even charge for because they are negligible, and the tokens they do charge for are at a fraction of a comparable model.

If you are using structured output, I think those 3 together is a really big deal.

>But their example is classification but that would also be possible and faster with a classic BERT model.

I believe the things you can classify with ChatGPT without any tuning or training is way beyond what BERT can do.

You need to train data for a BERT-based classifier, and then there's a risk that it will pick up specific biases from the data instead of what you want.

As far as I understand, the idea of Jev is zero-shot or few-shot classifier: it learns a lot of stuff at pre-training, but unlike a classic LLM it doesn't need to learn how to chat, so it can be much smarter at a particular size

I am in no way trying to sell Jev here as some panacea of the modern world; I'm only responding to your questions:

> But their example is classification but that would also be possible and faster with a classic BERT model.

With BERT, you need a large, labeled dataset, and you have to train/fine-tune the model. Jev is pitched as a zero- or 'few-shot' model. You define the schema in code, give it instructions, and it works without a traditional training pipeline.

> So their pitch is a task specific smaller model or am I completely misunderstanding the whole thing?

Yup; that about sums it up: it is more or less an optimized, task-specific small model with the flexible understanding of a traditional LLM.

I think this discourse is still in the "figuring it out" phase. But here's where my thoughts are currently:

If you accept the premise that there are use cases where you might ask a frontier model a classification-shaped question and expect an ok enough answer, rather than creating a purpose specific classifier on some dataset that you have, then it follows that this is quite an inefficient thing to do, because you're doing extra work to turn the output tokens into a structured output and mostly throwing them away. So then if you could instead train a frontier level model that skips the output tokens and directly returns the structured classification information, that would be more efficient, and that's what jev seems to be.

But a lot rides on that initial premise of whether this is a use case that makes sense. But if you find yourself asking a model like Opus arbitrary yes/no questions and then maybe you switch to a faster and cheaper model because it's too slow and expensive, it seems like jev might be a great replacement for that.

> is already fully solvable with structured output.

Not particularly. There is still the problem of hallucinations and varying results across runs.

That's more of what type-safety means for their team. Every run gives the same results. It's type-safe