Drive Android from semantic UI state

01 · Role in the system

What Jev does here

Jev receives a compact semantic description of the current screen together with a small set of pre-validated actions such as tap, go back, wait, or escalate, and answers with one action plus confidence and a probability margin. Coordinates, MCP calls, and invented steps are impossible by construction because the controller maps each answer to a prepared call. Confidence and top-two margin gates decide execution, repeated states trigger escalation, and ranked action pages keep crowded screens from overwhelming the model.

Only pre-validated actions can be chosen

The Android loop refuses to let the model invent steps. Each cycle stabilizes the screen, normalizes it into a compact semantic description, and attaches a short list of pre-validated actions — tap here, go back, wait, escalate. Jev picks one and returns confidence with a probability margin; the controller maps each answer to a prepared call, so coordinates, invented steps, and tool calls are impossible by construction rather than by instruction.

The escalation ladder is what makes this safe enough to publish: confidence and top-two margin gates decide execution, repeated identical states trigger escalation instead of loops, and payments, account changes, and destructive actions are out of scope in plain text. Every decision lands in a JSONL trace.

One device, one app, all boundaries stated

The evidence is honestly narrow: read-only navigation of Android Settings on one real device, with no benchmark published and other apps unsolved. That is the correct posture for a proof of concept — the mechanism is demonstrated, the generalization is not claimed.

What transfers is the action-gating pattern: pre-validated action lists with margin-based execution gates convert mobile UI automation from a generation problem back into a choice problem, which is the same pattern the desktop and browser records in this index use at larger action-space sizes.

02 · Control boundary

Where Jev sits

Observe-stabilize-normalize loop to one bounded action Choice; code owns UI stabilization, execution, safety gates, escalation, and JSONL traces.

Code owns the loop, permissions, thresholds, validation, and side effects. Jev owns only the bounded judgments described above.

03 · Known limits

What this evidence does not prove

• Exercised only against read-only Android Settings navigation on one real device; other apps are unsolved.
• Payments, account changes, and destructive actions are explicitly out of scope, and no benchmark is published.

04 · Attribution

Public sources

This is a Community record: the project was published by a third-party community author.

• Friedjof ↗Community · github · public · checked 2026-09-18