Screen context

How Lace reads your screen by fusing on-device computer vision with the macOS accessibility tree to understand any app’s layout and content.

Lace reads your screen by combining on-device computer vision with the macOS accessibility tree. Pixels provide layout. Accessibility data provides text, labels, roles, and interactive state.

The floating pill

The pill is a small capsule that floats on your screen. It shows what Lace is doing: idle when no session is active, watching when capture starts, a breathing animation while parsing, and ready when screen context is available. Hover to reveal controls. Press ⌘L to show the main window if minimized.

The pill also houses the Review button. Click it to capture and annotate what’s on screen. See Decision Canvas for the full workflow.

How capture works

Capture is tied to the active chat thread. Draft chats can start capture automatically. Existing chats start capture when you use the recorder control.

1. Screenshot and accessibility scan

   Lace captures a screenshot and reads the accessibility tree: text labels, button names, input values, element roles.

2. Vision analysis

   A vision model detects UI elements and identifies spatial relationships between components.

3. Fusion

   Vision results and accessibility data are merged into elements with both visual position and semantic meaning.

4. Context injection

   When you send a message with screen context, the parsed screen is included as structured context for that turn and can update the thread’s latest screen-context cache.

What Lace can see

Data type	Source
Text content	Accessibility tree + vision
Element roles	Accessibility tree
Visual layout	Screenshot + vision
Page structure	Both
Interactive state	Accessibility tree

The overlay

Press ⌘O to toggle the screen context overlay. Detected elements are highlighted so you can see exactly what Lace has parsed.

Caching

Lace keeps the latest parsed screen for the live screen-context session. If the active window has not meaningfully changed, Lace can reuse the cached context. When content changes, Lace captures and parses a fresh frame automatically.