Stop Guessing Pixels: AT-SPI-First Grounding for Desktop Agents

The last post was about giving an agent eyes: OmniParser looks at a screenshot, detects the clickable boxes, and captions the icons. It works, and there's an unavoidable catch baked into the approach — it's inferring. A vision model decides where the button probably is and what it probably does. Most of the time it's right. Some of the time it's four pixels off, or it reads a disabled button as clickable, and the agent clicks the wrong thing with total confidence.

There's a way to not guess. On Linux, the desktop will just tell you where everything is — if you ask it the way a screen reader does.

The desktop already publishes its UI

Every well-behaved GTK or Qt application exposes an accessibility tree: a structured, live description of its interface, published over D-Bus through AT-SPI (the Assistive Technology Service Provider Interface). This is not a hack or a side channel. It's a first-class API that apps are supposed to implement, and it's the exact interface Orca, the Linux screen reader, uses to read the screen aloud to blind users.

Walk that tree and every widget hands you, for free, the four things a vision model was trying to reconstruct:

role — what kind of thing it is: push button, menu item, text entry, check box, page tab.
name — its actual label: "Terminal Emulator", "File Manager", "Save". Not a caption a model guessed — the string the app itself set.
exact bounding box — screen coordinates via the Component interface's get_extents(), which returns (x, y, width, height) in real pixels. Not "roughly here." Here.
states — a live status set: focusable, enabled, showing, sensitive. You can tell a greyed-out button from a live one before you click it.

No detection model. No captioning model. No inference. The app labeled everything itself, for accessibility reasons, years ago.

AT-SPI-first grounding, the flow

So the loop that OmniParser does with two vision models becomes a tree walk:

Walk the tree for the focused application (or the whole desktop).
Filter to what's actionable and visible — keep widgets whose role is interactive and whose states include showing, enabled, and sensitive. This alone kills a whole class of mis-clicks: you never target something that isn't really there or isn't really clickable.
Build a labeled list — name, role, and exact bbox for each. It's already in words; no captioning step.
Let the model pick by meaning, not by coordinates: "click the 'File Manager' button." The LLM chooses a widget by its name and role — the thing it's actually good at — instead of estimating an (x, y).
Click the exact center of that widget's box. Pixel-perfect, every time, regardless of theme, font scaling, or DPI.
Verify by re-reading states. Did the button's state flip to pressed? Did the menu's expanded go true? Did focus move? The tree gives you a built-in success signal for the action you just took — something the screenshot approach has to infer all over again.

Why this is better where it works

Exact coordinates. The number one failure mode of vision grounding — a click that lands just outside the target — simply cannot happen. You have the real rectangle.
Real semantics. The name is the application's own label, not a model's best guess. "Save" is "Save" because the app said so.
Free preconditions and postconditions. States tell you whether a widget is clickable before and whether the click worked after. Grounding and verification from the same source.
Cheap and stable. No model inference to perceive the screen, and nothing breaks when the user switches themes, bumps the font size, or changes DPI — all of which quietly wreck pixel-matching.

Why you still keep the vision model

AT-SPI-first is not AT-SPI-only, and the reason is one word: coverage. The tree is perfect when the app fills it in. Plenty don't:

custom-drawn canvases and game UIs that paint pixels and expose nothing,
some Electron/Chromium apps and web content, which need the browser's own accessibility bridge turned on,
half-broken apps that expose a role but no name, or a stale box.

Where the tree is blank, you're back to pixels — and that's exactly the gap OmniParser fills. So the sturdy design is a hierarchy: try AT-SPI first; fall back to vision only for the regions the tree can't describe. You get exactness wherever the app cooperates, and coverage everywhere else. The two techniques from these two posts aren't rivals — they're the two layers of one grounding stack.

It's not just Linux

The idea generalizes. macOS has the Accessibility (AX) API; Windows has UI Automation. Same principle on all three: the operating system already maintains a semantic tree of the UI for assistive tech, and an agent can read it instead of guessing at pixels. AT-SPI is just the Linux member of that family, and the most open one.

The nice irony

The cleanest interface for a computer-use agent turns out to be the one built for screen-reader users. All that accessibility work — roles, names, exact geometry, live states — was done so a blind person could navigate a desktop by keyboard and speech. It happens to be precisely the structured, labeled, coordinate-exact view an agent wants too. Grounding that reads the labels instead of guessing the pixels. Ask the desktop where the button is; most of the time, it already knows.

Reference: AT-SPI2 (freedesktop) · Atspi.Component.get_extents · Orca.