Find and control app elements

UIHelper locates controls by their properties instead of hard-coded screen positions. Start with the inspector or MCP observe_screen, inspect the screenshot, and test the suggested selector before saving it in a script.

Pin the app and check the target

The identifiers below are examples, not universal app identifiers.

const bundleId = "com.example.app";
if (!openApp(bundleId)) throw new Error("Could not open target app");
const ui = UIHelper.forApp(bundleId);
const matches = ui.findElements({id: "searchField", visible: true, enabled: true});
if (matches.length !== 1) throw new Error("Expected one search field");
matches[0].setText("Hello");

forApp() does not launch an app. Without it, UIHelper uses the foreground app. Most actions require the target app to be foreground. If the helper is blacklisted, use the appropriate screen-based APIs or change the blacklist deliberately.

Choose selectors

Other exact keys include name, text, value, className (alias class), selected, accessible, focused, index, x, y, width, height and rect.width-style geometry keys. Contains filters also exist for name, identifier, label and value. name uses the identifier, falling back to the label; a string selector is shorthand for {name: string}.

Types accept Button or XCUIElementTypeButton. className means the actual native class. Text filters are case-sensitive by default. Case-insensitive options do not change type, class or explicit predicates. index is a child position, not a durable identifier.

Predicates accept documented attributes and boolean comparisons, for example {predicate: "label CONTAINS[c] 'search' AND visible == true"}. Type comparisons inside predicates use full names. Arbitrary function expressions and ANY/ALL modifiers are unsupported. Prefer ordinary selectors when they express the same intent.

Groups accept 1..32 selectors, with at most 8 nesting levels and a combined 256-node/key budget. Keep them simple.

Wait for state, then act once

const ui = UIHelper.forApp("com.example.app");
ui.waitForElement({id: "continueButton", visible: true, enabled: true}, 5000).tap();
ui.waitForGone({id: "loading", visible: true}, 10000);
ui.waitForElement({id: "resultScreen", visible: true}, 5000);

Explicit waits default to 5000 ms and 100 ms between polls. Options accept {timeout, interval, throwOnTimeout}. throwOnTimeout:false returns null for an optional element wait, or false for waitForGone. Helper failures, stale roots, invalid selectors and cancellation still throw. A helper call can outlast the polling deadline.

findElement() returns the first match, not proof of uniqueness. waitForAny() selects by tree order, not the order of your selectors. waitForAll() returns matches in input order; the same element can satisfy several selectors.

Scope repeated controls

const panel = UIHelper.waitForElement({id: "profilePanel"}, 5000);
panel.waitForElement({id: "editButton", visible: true}, 3000).tap();

Element queries search descendants. Reacquire the parent after navigation or view recycling. snapshot is cached data from a query; getAttributes() fetches fresh fields together.

Scrolling and text

element.scroll("down") changes a native scroll view’s position without simulating a finger. It may look instantaneous. Use swipe() if the test requires a touch gesture. Verify progress and limit attempts; reaching the end can return successfully without moving.

setText() replaces a supported field’s value. typeText() inserts text, optionally with {interval:80} milliseconds per Unicode code point. Check the existing value after an interruption to avoid duplicate input. Do not log passwords or secure-field contents.

References and visibility

A JavaScript element reference belongs to the app process that created it. A navigation, restart or recycled view can invalidate it. MCP observation references such as e4 also belong to a particular observation, nominally expire after 60 seconds, and may be evicted earlier. Saved scripts should use selectors instead.

Visibility checks cover frames, clipping and hidden state, not every overlapping window. A successful tap is not proof the intended transition happened. Inspect a fresh screen when uncertain, and use known interruption handlers rather than tapping through overlays.