Community · E2 · artifact verified
Fit confidence thresholds on your own data
jevcal measures a typed decision model on private labeled data, fits per-question thresholds to a target accuracy, and fails CI when a model update drifts.
01 · Role in the system
What Jev does here
Jev supplies the typed answers while jevcal owns measurement, and the tool deliberately publishes no Jev benchmark numbers of its own. It fits a threshold per question to a target accuracy on one half of the data, verifies it on the held-out half, estimates how much traffic still escalates to an LLM and at what cascade cost, surfaces confident-but-wrong rows for label review, and re-checks locked thresholds in CI against aliases like jev-latest that move between releases.
02 · Control boundary
Where Jev sits
Labeled JSONL and typed question definitions in, threshold lockfile and reliability report out; the model keeps answering while code owns calibration and gates.
Code owns the loop, permissions, thresholds, validation, and side effects. Jev owns only the bounded judgments described above.
03 · Known limits
What this evidence does not prove
- Thresholds fitted on fewer than roughly one hundred labeled rows should not be trusted, per the author.
- The bundled demo output comes from an overconfident simulator, not from Jev.
04 · Attribution
Public sources
This is a Community record: the project was published by a third-party community author.
- abhixhek ↗Community · github · public · checked 2026-09-18