Does it need an internet connection or a server?

No. The on-device agent loop runs entirely on the phone with no server. Cloud agents (Claude, ChatGPT, Codex) are optional and used when connected, including for apps the small model can't handle.

Which phones get NPU acceleration?

Currently only Snapdragon phones — the encoder was compiled via Qualcomm AI Hub and tested on a Galaxy S25 (Snapdragon 8 Elite). Pixel, MediaTek and Exynos phones fall back to the GPU or CPU until separate compiles exist.

How does the app pick which encoder to use?

At startup it tries the NPU, then the GPU via LiteRT, then the CPU. Each candidate must pass a self-test on a bundled screenshot (output within cosine 0.97 of the CPU encoder) and be faster than the CPU; the CPU path always works. Settings shows which encoder won and how long a screen takes to encode.

How much does it cost?

The Android app is free at getopposable.com. The screen model is a 1.5 GB download from the app's settings.

Opposable — An on-device 2B model that operates your Android apps

Opposable is an Android app that operates other apps on the same phone. It reads the screen through the Android accessibility service and taps and types in any app. When Claude, ChatGPT or Codex is connected, those agents can drive the phone from a computer; when none is connected, a 2-billion-parameter screen model (GUI-Owl 1.5 2B, quantised to 4 bits and run through llama.cpp) runs the same operate-the-screen loop entirely on the phone with no server involved. The post documents the engineering: on a Mac M2 CPU with 4 threads one agent step takes 7.3 s, down from 10.6 s, at equal accuracy on a 36-screen test set, and the vision encoder runs in 236 ms on a Galaxy S25's NPU (Snapdragon 8 Elite) instead of 3.9 s on the M2.

Key Features

On-device screen agent

A 2-billion-parameter model (GUI-Owl 1.5 2B, a Qwen3-VL fine-tune from Alibaba's Mobile-Agent team) looks at a screenshot and outputs one action — a sentence of intent followed by a JSON tool call with tap coordinates. It runs in the app via llama.cpp at 4-bit quantisation: 1.05 GB for the language

Works without any server

The full perceive-decide-act loop runs on the phone itself. There is no API call, no cloud inference, and no server in the on-device path.

Falls back to cloud agents when connected

Claude, ChatGPT or Codex can drive the app from a computer. The on-device model is deliberately used only for tasks it can't handle otherwise — apps it hasn't seen are handed to the cloud agent when one is connected.

Fixed code for ~40 common tasks

About forty common tasks — setting an alarm, texting a contact, switching on Wi-Fi — run as fixed code with no model at all, and routines a large agent worked out earlier replay from the on-screen text.

Screenshot pruning pipeline

Status bar, navigation bar and keyboard are cropped before encoding; after the encoder, image tokens for flat background patches that border nothing are dropped (25–40% fewer tokens) while keeping original grid positions. The 900-token system prompt is prefilled once and its KV cache kept. Result: 7

Grounded early exit

When the model's intent sentence matches exactly one element in the accessibility tree, the app taps that element and stops generating — coordinates are never decoded, cutting decode from 2–4 s to 0.8–1.3 s per step.

NPU vision encoder with automatic encoder selection

The vision tower is exported to ONNX (a 24-layer ViT, ~0.4B parameters) and compiled via Qualcomm AI Hub; on a Galaxy S25's NPU in fp16 it encodes a screen in 236 ms vs 3.9 s on the M2 CPU. At startup the app tries NPU, then GPU (via LiteRT), then CPU, each gated by a self-test on a bundled screensh

Subgoal decomposition with verification

Instead of giving the model a whole task at once, the app gives one subgoal at a time and checks each claimed success against the accessibility tree before proceeding. On eight held-out tasks this raised passes from 4/8 and 3/8 (whole task) to 6/8 in both rounds.

How It Works

Reads the screen and acts via accessibility service

The app reads the screen through Android's accessibility service and taps and types in any other app — no root or special permissions beyond accessibility are described.

One step = encode, prefill, decode, tap

Each step turns the screenshot into image tokens (vision encoder), reads them plus the task (prefill), writes ~45 tokens of intent plus a JSON tool call (decode), then taps. On the M2 CPU after optimisation: 3.6 s encode, 2.5 s prefill, 1.1 s decode.

Offline model is a 1.5 GB download

The screen model is downloaded from the app's settings (1.5 GB total: 1.05 GB language model + 445 MB vision encoder) and stays on the device.

Hardware fallback chain

NPU (Qualcomm, fp16) → GPU (LiteRT/OpenCL) → CPU (llama.cpp), each candidate behind a self-test at first load; the CPU path always works. First NPU load takes 72 s once per install, 0.39 s afterwards. The 640-token input size failed to compile for the NPU, so steps needing it still encode on CPU.

Pros & Cons

Pros

The full agent loop runs on-device — no server, no cloud inference needed for the on-device mode
Quantised 4-bit model keeps the footprint to about 1.5 GB total
Real optimisation effort with published numbers: 10.6 s → 7.3 s per step on a Mac M2 CPU at equal accuracy, and 236 ms vision encoding on a Galaxy S25 NPU vs 3.9 s on CPU
Automatic encoder selection (NPU → GPU → CPU) with self-tests, so the CPU path always works even on unsupported chips
Subgoal decomposition with accessibility-tree verification measurably improves held-out task success (4–3/8 → 6/6/8)

Who It's For

Best for

Android users who want app automation that works with no cloud connection and keeps screen contents on the device
Owners of recent Snapdragon phones (e.g. Galaxy S25) who can use the NPU-accelerated vision encoder
Developers and researchers interested in running small multimodal GUI agents on-device with llama.cpp and llama.cpp's multimodal library
Users who already use Claude, ChatGPT or Codex and want those agents to operate their phone from a computer

Not ideal for

iPhone users — the app is Android-only
Phones without a Snapdragon NPU build (Pixel/Tensor, MediaTek, Exynos), which lose NPU speed and fall back to GPU or CPU

Use Cases

Operating other Android apps by instruction — e.g. opening Clock and switching to the stopwatch — with no server involved

Handing your phone to a cloud agent (Claude, ChatGPT, Codex) driving it from a computer

Running ~40 common phone tasks (set an alarm, text a contact, toggle Wi-Fi) as fixed code without any model

On-device automation in environments where cloud access or sending screen contents off the phone is not acceptable

FAQ

What is Opposable?

An Android app that reads the screen through the accessibility service and taps and types in any other app. Claude, ChatGPT or Codex can drive it from a computer, and when none is connected, a 2-billion-parameter screen model runs the same loop on the phone with no server involved.

Which model does it use and where does it run?

GUI-Owl 1.5 2B — a Qwen3-VL fine-tune from Alibaba's Mobile-Agent team — quantised to 4 bits and run in the app through llama.cpp and its multimodal library. It's 1.05 GB for the language model plus 445 MB for the vision encoder, loaded once and kept in memory between steps.

How fast is it?

One agent step takes 7.3 s on a Mac M2 CPU with 4 threads (down from 10.6 s unoptimised) at the same accuracy on the 36-screen test set. The vision encoder alone runs in 236 ms on a Galaxy S25's NPU versus 3.9 s on the M2 CPU.

How accurate is the on-device model?

Opposable