Measure confidence-gated routing on labelled data

01 · Role in the system

What Jev does here

Jev answers one classification question per row, and its confidence drives a cascade that escalates only uncertain rows to a larger fallback model. On 500 Banking77 examples, routing at threshold 0.67 reached 80.2 percent accuracy for about 0.103 dollars per 500 decisions, beating either model alone; on 500 Web of Science abstracts the same routing tied Jev alone for 46 percent more money. The protocol was frozen before results were read, and the raw JSONL, figures, and analysis code are committed.

A frozen protocol and a negative result, both kept

This is the methodologically strongest record in the index. The protocol was frozen before results were read: Jev classifies each row, confidence drives a cascade that escalates uncertain rows to a larger fallback model, and coverage, accuracy, latency, and cost are measured against the pre-registered rule. Raw JSONL, figures, and analysis code are committed.

The result is two honest findings for the price of one. On 500 Banking77 examples, routing at threshold 0.67 reached 80.2 percent accuracy for about $0.103 per 500 decisions — beating either model alone. On 500 Web of Science abstracts, the same routing tied Jev alone while costing 46 percent more. The winning setting did not transfer, and the record leads with that rather than the win.

What the negative transfer teaches

The caveat is structural: Web of Science labels come from publication metadata rather than per-document annotation, so part of the measured error is label ambiguity — and the record says the two datasets produced opposite outcomes with nothing transferring between them. That is precisely the finding anyone copying confidence-gated routing needs: the threshold is not portable, it is a per-dataset measurement you have to run yourself.

As the research cluster's only labelled-data study of Jev's own confidence, it sets the evidentiary bar the reproduction program is asking the rest of the index to rise to — frozen protocol, committed data, and a published result the author did not like.

02 · Control boundary

Where Jev sits

Per-row typed answers with a confidence threshold to an escalate-or-accept rule; code measures coverage, accuracy, latency, and cost against the frozen protocol.

Code owns the loop, permissions, thresholds, validation, and side effects. Jev owns only the bounded judgments described above.

03 · Known limits

What this evidence does not prove

• The two datasets produced opposite outcomes; no threshold, accuracy-gap sign, or cost result transferred between them.
• Web of Science labels come from publication metadata rather than per-document annotation, so part of the measured error is label ambiguity.

04 · Attribution

Public sources

This is a Community record: the project was published by a third-party community author.

• FirasSX914 ↗Community · github · public · checked 2026-09-18