Confidence and thresholds

Every Jev answer comes with a number that says how sure it is. How that number is calculated, and how to turn it into rules for when to act, when to ask, and when to hand off to a person.

Probability vs confidence

These two sound alike but answer different questions.

• Probability is "how likely is this option?" Each option gets one.
• Confidence is "how clear is the winner?" There is one per question.

If the probabilities are billing 0.95, technical 0.03, account 0.02, the winner is obvious and confidence is high. If they are billing 0.40, technical 0.35, account 0.25, Jev still picks billing, but confidence is low. The options looked alike to it.

For three options, TypeSafe gives this approximation: confidence = (3 × top probability − 1) / 2. All weight on one option gives 1.0. Spread evenly, it gives 0.

Turn it into a policy

team = result.choices["team"]

if team.confidence > 0.9:
  assign(team.choice)
elif team.confidence > 0.5:
  assign(team.choice, needs_review=True)
else:
  send_to_human()

This is the real payoff of Jev over a chat model. A text answer does not tell you when it is guessing. A calibrated number does, so you can automate the easy 80% and send the hard 20% to a person.

Scale thresholds by risk

One number does not fit every action.

For a Noul, pick the threshold based on which mistake hurts more. Set it higher when false alarms are expensive. Set it lower when missing a real case is expensive.

Check both numbers

For a Choice, you can also look at the chosen option's own probability. Vercel's guide uses both:

const selectedProbability = department.probabilities?.[department.choice] ?? 0;

if (departmentConfidence < 0.6 || selectedProbability < 0.7) {
return { action: "human-review", reason: "ambiguous department" };
}