Protocol Overview

UIAP in One Sentence

UIAP is a standard contract between a running application and an AI agent, so the application can tell the agent in a structured way what’s currently visible, what’s possible, what’s risky, how things may be executed, and how success or failure can be recognized.

What UIAP Should Be

• A UI contract protocol
• A capability and state protocol
• An action and feedback protocol
• A policy and audit protocol

What UIAP Should Not Be

• Not an LLM prompt format
• Not a voice protocol
• Not a replacement for accessibility
• Not a replacement for browser automation
• Not just an SDK
• Not magic that makes bad apps suddenly controllable

The Specification Modules

#	Module	Purpose
1	Core	Message envelope, session lifecycle, errors, versioning, extensions
2	Capability Model	UI roles, states, affordances, actions, risk, success signals
3	Web Profile	DOM, ARIA, PageGraph, iframes, Shadow DOM, routes, forms, modals
4	Action Runtime	Action execution, verification, result reporting
5	Policy Extension	Permissions, confirmations, sensitivity, audit
6	SDK API	Client-side integration API for web applications
7	Workflow Extension	Skills, onboarding flows, step orchestration
8	Discovery Mapper	Automatic UI element discovery and classification
9	Authoring/Manifest	Manifest formats, authoring workflows, validation
10	Conformance Suite	Test modules, harness model, evaluation rules

End-to-End Flow

1. App integrates the UIAP SDK
2. Session starts (session.initialize)
3. App sends capabilities (capabilities.list)
4. App sends current live state (state.snapshot)
5. Agent plans next action (e.g., action.request(click video.new))
6. SDK checks policy and execution mode (prefers appAction, falls back to semanticDom)
7. SDK optionally animates ghost cursor (presentation only)
8. SDK executes the action and sends action.started
9. App observes result (route change, toast, field value, dialog close)
10. App sends feedback + delta (action.succeeded + state.delta)
11. Agent plans next step (until goal reached or user takes over)
12. Session ends and trace is finalized (session.end)