Using OpenClaw's Canvas Tool for GUI Automation

Most of OpenClaw's power lives in the terminal — shell commands, file edits, API calls. But sometimes your agent needs a visual surface. A dashboard. A form. A live preview of something it just built. That's where the Canvas tool comes in.

Canvas is a lightweight, agent-controlled HTML panel built into the OpenClaw macOS app. It lets your agent present web content, run JavaScript against it, capture screenshots, and even push structured UI components via the A2UI protocol. Think of it as a programmable browser window your agent owns entirely.

What Canvas Actually Is

Canvas is a WKWebView panel embedded in the macOS app. It's borderless, resizable, and anchored near the menu bar. Each session gets its own file root under Application Support:

~/Library/Application Support/OpenClaw/canvas/<session>/

Files are served via a custom URL scheme:

openclaw-canvas://main/           → index.html
openclaw-canvas://main/app.css    → app.css
openclaw-canvas://main/widgets/   → widgets/index.html

If no index.html exists, the app shows a built-in scaffold page. The panel auto-reloads when local canvas files change, so your agent can write HTML to disk and the Canvas updates live.

Core Actions

The Canvas tool exposes six actions through the Gateway:

Present and Hide

Show or dismiss the Canvas panel. Simple visibility control.

canvas action=present    → opens the panel
canvas action=hide       → closes it

You can target a specific node if you're running multiple devices, and optionally set width and height.

Navigate

Point the Canvas at a local path, an HTTP URL, or a file URL:

canvas action=navigate url="/"
canvas action=navigate url="https://example.com/dashboard"

Local paths resolve against the session's canvas root. This is how you switch between different views your agent has built.

Evaluate JavaScript

Run arbitrary JavaScript inside the Canvas and get the result back:

canvas action=eval javaScript="document.title"
canvas action=eval javaScript="document.querySelectorAll('.item').length"

This is powerful. Your agent can build an HTML interface, then interact with it programmatically — reading form values, triggering clicks, extracting data from rendered DOM elements.

Snapshot

Capture the current Canvas content as an image:

canvas action=snapshot
canvas action=snapshot outputFormat="png" width=1200 height=800

The snapshot returns an image the agent can analyze, send to a chat, or use for visual verification. Useful for confirming that a generated UI looks correct before showing it to a user.

A2UI: Structured UI Without HTML

A2UI (Agent-to-UI) is a protocol for pushing structured UI components into Canvas without writing raw HTML. Instead of building a webpage, your agent sends JSONL messages describing a component tree.

How It Works

A2UI messages use a declarative component model. You define surfaces with components like Text, Column, and others, then tell Canvas to render them:

{"surfaceUpdate":{"surfaceId":"main","components":[
  {"id":"root","component":{"Column":{"children":{"explicitList":["title","body"]}}}},
  {"id":"title","component":{"Text":{"text":{"literalString":"Status Dashboard"},"usageHint":"h1"}}},
  {"id":"body","component":{"Text":{"text":{"literalString":"All systems operational."},"usageHint":"body"}}}
]}}
{"beginRendering":{"surfaceId":"main","root":"root"}}

Quick Push

For simple text content, there's a shortcut:

canvas action=a2ui_push jsonl='{"text": "Hello from your agent"}'

Or load from a file:

canvas action=a2ui_push jsonlPath="/tmp/dashboard.jsonl"

Reset

Clear the A2UI state and start fresh:

canvas action=a2ui_reset

A2UI currently supports v0.8 commands: beginRendering, surfaceUpdate, dataModelUpdate, and deleteSurface.

Practical Use Cases

Live Dashboards

Your agent can build a monitoring dashboard as an HTML file, present it via Canvas, and update it periodically. Combine this with cron jobs that write fresh data to the canvas directory, and the auto-reload feature keeps the display current.

Visual Verification

Building a report or webpage? Have your agent render it in Canvas, take a snapshot, and analyze the screenshot before delivering the final product. This closes the loop between generation and visual QA.

Interactive Forms

Write an HTML form to Canvas, present it to the user, then use eval to read back the form values after the user fills it in. Canvas can trigger agent runs via deep links (openclaw://agent?message=...), so the form can kick off follow-up work.

Data Visualization

Generate charts or graphs as HTML (using libraries like Chart.js or D3), render them in Canvas, snapshot the result, and send the image to a chat channel. Your agent becomes a visual reporting tool.

Triggering Agent Runs from Canvas

Canvas supports deep links that trigger new agent runs:

window.location.href = "openclaw://agent?message=Review%20this%20design";

The app prompts for confirmation unless a valid key is provided. This creates a feedback loop where Canvas content can request agent action — making truly interactive agent-driven applications possible.

Security

Canvas is sandboxed by design:

• No directory traversal — files must live under the session root
• Custom URL scheme — no loopback server required for local content
• External URLs — only loaded when explicitly navigated by the agent
• User control — Canvas can be disabled entirely from Settings → Allow Canvas

Getting Started

1. Make sure you're running the OpenClaw macOS app (Canvas is a native macOS feature).
2. Have your agent write an index.html to the canvas directory, or use a2ui_push for quick content.
3. Call canvas action=present to open the panel.
4. Use eval and snapshot to interact with and capture the rendered content.

Canvas turns your agent from a text-only operator into something with a visual presence. It's not a full browser automation tool (that's what the browser tool is for), but for agent-owned UI surfaces — dashboards, previews, forms, visualizations — it's exactly the right tool for the job.