Original community demo · Video hosted on X
Desktop & device control
An open-source Android experiment chooses actions from the current accessibility tree, with a small language model handling text entry.
Opens the author's original post in a new tab.
The implementation ideas below are our interpretation of the community demo. Check the original post for the author’s actual setup.
A device assistant can use labeled controls to navigate an app when the next action fits a known set of taps, scrolls, and text-entry steps.
Build an action list from the current accessibility tree, ask for a choice, execute the selected action, and refresh the list after the screen changes.
These examples explain a possible workflow; they are not recorded demo results or API request and response formats.
Apps with missing accessibility labels or custom controls may need another observation method. Text entry uses a separate model in this demo, so the full workflow is not a single-model system.
Define the context and a small set of choices in the playground, then use the API guide to connect your application.
View demoJev Ultrafast turns indexed page elements into candidate actions, demonstrated through a flight-search workflow.
View demoA Rust desktop application classifies live network traffic to highlight potential anomalies as they arrive.
View demoScore earlier tool calls for relevance, then remove less useful outputs from an agent’s working context.
