Community · E2 · artifact verified
Measure a memory loop around Jev decisions
Four business-decision benchmarks, three arms: Jev on policy alone, Jev plus retrieval, and Jev with SenseLab's memory loop - the memory arm learns an unwritten rule to 98% and cuts wrong decisions from 59 to 6. Rebuildable from committed per-episode rows.
01 · Role in the system
What Jev does here
Every arm runs the same Jev model (jev-1.13.0), the same 120 leads and the same written policy; only what Jev is told about the situation changes. On lead routing, policy-only Jev got 51% and lost $142,900; the best retrieval lookup the authors could build got 94%; Jev with SenseLab's memory loop got 95% and lost $10,400. A rule nobody wrote down (EMEA mid-market deals close only through partners) hits 42 of 120 leads: the memory got it wrong once, then never again (98%). When the business changes at episode 60, memory switches action after three deliberate failures; a wording change at episode 90 leaves both memory arms unharmed, because lessons are keyed to the situation. The memory slot is capped at 3,200 characters and stays flat however long the run. About six misses per run are the honest price of learning from outcomes.
02 · Control boundary
Where Jev sits
Jev as the decision core of an episode loop: each decision receives policy, the current case and a bounded memory digest maintained by SenseLab's improvement loop (traces, incidents, fixes, canaries); outcomes are scored against hidden truth and per-episode rows make every number recomputable.
Code owns the loop, permissions, thresholds, validation, and side effects. Jev owns only the bounded judgments described above.
03 · Known limits
What this evidence does not prove
- SenseLab is a third-party (AMFS) product; the full improvement loop needs a SenseLab account and API key.
- Benchmarks are author-run against simulated hidden truth with injected review noise; business scenarios are synthetic.
- About six misses per run are inherent to learning from outcomes, per the authors' own analysis.
- Sample results ship in the repo and rebuild with --replay without any keys; five scenario packs, only lead_routing committed in full.
04 · Attribution
Public sources
This is a Community record: the project was published by a third-party community author.
- raia-live ↗Community · github · public · checked 2026-10-16