Drive macOS from structured screen state

01 · Role in the system

What Jev does here

Jev classifies the next action and, when relevant, the visible or off-screen target from a numbered screen inventory. Capture, OCR, accessibility traversal, date arithmetic, action execution, and stop conditions remain deterministic. A writing model is isolated to free-text fields and proposed URLs, and password fields are refused.

Screen reading as inventory, not pixels

The loop fuses OCR with the macOS accessibility tree into a numbered inventory of the screen, then asks Jev two bounded questions: which action class comes next and, where relevant, which on- or off-screen target it applies to. Everything else — capture, date arithmetic, execution, stop conditions — is deterministic code, and password fields are refused outright rather than handled carefully.

Isolating the writing model is the quiet design decision. Free-text fields and proposed URLs are the only places a generative model touches, so the blast radius of a hallucinated phrase is confined to text content, while every decision that moves the machine stays a typed choice.

What the cost number does and does not say

The 0.0002-dollar-per-step decision cost is an author-reported comparison on the same screenshot and goal, excluding the optional writing model. It demonstrates the economic argument for small bounded judgments — classification costs a fraction of open-ended generation — but it is a controlled pair, not a billing history.

Coverage limits are stated rather than discovered: canvas content, icon-only controls, and apps with weak accessibility trees fall outside the fusion's sight, and the tool targets macOS with Screen Recording and Accessibility permissions. As E2 evidence the repository matches the description; the edges are where the next revision of this record will be tested.

02 · Control boundary

Where Jev sits

OCR and accessibility observations become parallel Choices; code owns permissions, safety stops, and input execution.

Code owns the loop, permissions, thresholds, validation, and side effects. Jev owns only the bounded judgments described above.

03 · Known limits

What this evidence does not prove

• It targets macOS and requires Screen Recording and Accessibility permissions.
• OCR and accessibility coverage miss canvas content, icon-only controls, and applications with weak accessibility trees.

04 · Attribution

Public sources

This is a Community record: the project was published by a third-party community author.

• awlevin ↗Community · github · public · checked 2026-09-18