Designing the ref system for AI-first browser automation

The hardest problem in AI browser automation isn't navigation or data extraction — it's element targeting. How does an AI agent reliably refer to "the submit button on the checkout form" in a way that survives DOM changes, dynamic IDs, and cross-session inconsistency?

The selector problem

CSS selectors and XPath are deterministic and precise — but they assume the DOM is static and predictable. Real websites use React with randomly generated class names, IDs that change on every render, and deeply nested component trees where the "right" selector is 47 levels deep.

Telling an AI agent to use a CSS selector is telling it to navigate by street address in a city that renames its streets every day. It works until it doesn't, and when it fails, the failure is silent.

Snapshots as structured perception

Instead of exposing the DOM directly, Scout's browser-snapshot tool returns a filtered accessibility tree. The accessibility tree is the semantic graph that screen readers use — it contains only nodes that have meaning to a user: buttons, links, form inputs, headings, tables. Purely decorative elements are excluded by default.

Every interactive node in the snapshot gets a short @eN identifier. These refs are stable within a snapshot session and deterministic — the same node gets the same ref on every snapshot of the same page state.

# browser-snapshot output (summarized)
document
main
heading "Checkout" @e1
form "Order details" @e2
group "Shipping address" @e3
textbox "First name" @e4 required
textbox "Last name" @e5 required
textbox "Address" @e6 required
group "Payment" @e7
textbox "Card number" @e8 required
textbox "Expiry" @e9 required
textbox "CVV" @e10 required
button "Place order" @e11

Using refs in tool calls

The agent passes @e refs directly to browser-interact. No CSS, no XPath, no natural language selector translation at action time. The ref lookup is O(1) against the session's element registry.

// Agent reads snapshot, identifies @e11 as the submit button
await client.callTool({
  name: "browser-interact",
  arguments: {
    action: "click",
    elementRef: "@e11", // "Place order" button
    sessionRef,
  },
});

Ref lifetime and invalidation

Refs are valid for the lifetime of a single snapshot. After any navigation or DOM mutation, refs are invalidated. The agent should re-snapshot before acting whenever the DOM may have changed. This is a feature, not a limitation — it forces the agent to maintain an accurate model of the current page state rather than acting on stale perception.

Snapshot filtering for token efficiency

A full accessibility tree for a complex SPA can easily exceed 10,000 nodes. Sending that to an LLM on every tool call would be expensive and slow. Scout's snapshot tool supports aggressive filtering: exclude decorative nodes, filter by role, set a maximum depth, or scope to a CSS subtree. The agent can snapshot just main to get the primary content without nav, sidebar, and footer noise.