Screen coordinates, OCR and images

Use UIHelper when the app exposes a useful control. Use OCR or image matching for content drawn inside custom views, and physical gestures when the interaction itself matters.

Start with current geometry

Touch APIs use native image pixels with a top-left origin. A screenshot displayed smaller in a browser is still a larger source image. Convert from the displayed image to the original dimensions before using coordinates. Reinspect after rotation, scrolling or navigation.

The inspector can provide element bounds, a selected pixel and OCR rectangles. A rectangle is {x,y,width,height}, not a single point. Its center is x + width/2, y + height/2.

Read text and its rectangle

const [items, error] = ocr({level: 0, coordinates: "pixels"});
if (error) throw new Error(error);
const matches = items.filter(item => item.text === "Continue");
if (matches.length !== 1) throw new Error("Expected exactly one text match");
console.log(JSON.stringify(matches[0].rectangle));

The word is an example and may be translated. Inspect confidence and context before tapping. Use level:0 for accuracy or level:1 for speed. recognizeTextSupportedLanguages() reports supported languages on the current device.

OCR defaults to points. Request pixels when results will drive screen touches. Regions and results use the selected coordinate mode. Results already include the crop offset; do not add it again. If the input is a file, its geometry may differ from the current screen.

Find a visual target

const [hits, error] = findImage({
  targetImagePath: "assets/button.png",
  threshold: 0.95,
  count: 2
});
if (error) throw new Error(error);
if (hits.length !== 1) throw new Error("Image target is missing or ambiguous");
console.log(hits[0]);

Image matching uses the template’s original pixel size, without automatic scaling or rotation. Use an opaque template and a tight search region. A match is a candidate; verify the target’s context and the next state after acting. Theme, resolution and app updates can change a template.

searchColor finds individual RGB colors. findColors checks several relative color anchors. getColors samples many points in one capture. Limit regions and result counts instead of capturing the full screen in a tight loop.

No match is different from failure

Visual searches return [[], null] when nothing matches, and [null, errorString] on failure. OCR follows the same pair convention. Do not ignore the error and treat every failure as “not on this screen”. Use a bounded wait/recovery policy and preserve the original error when stopping.

# Screen coordinates, OCR and images

Use UIHelper when the app exposes a useful control. Use OCR or image matching for content drawn inside custom views, and physical gestures when the interaction itself matters.

## Start with current geometry

The inspector can provide element bounds, a selected pixel and OCR rectangles. A rectangle is `{x,y,width,height}`, not a single point. Its center is `x + width/2, y + height/2`.

## Read text and its rectangle

```js
const [items, error] = ocr({level: 0, coordinates: "pixels"});
if (error) throw new Error(error);
const matches = items.filter(item => item.text === "Continue");
if (matches.length !== 1) throw new Error("Expected exactly one text match");
console.log(JSON.stringify(matches[0].rectangle));
```

The word is an example and may be translated. Inspect confidence and context before tapping. Use `level:0` for accuracy or `level:1` for speed. [recognizeTextSupportedLanguages()](/doc/api/recognizetextsupportedlanguages) reports supported languages on the current device.

## Find a visual target

```js
const [hits, error] = findImage({
  targetImagePath: "assets/button.png",
  threshold: 0.95,
  count: 2
});
if (error) throw new Error(error);
if (hits.length !== 1) throw new Error("Image target is missing or ambiguous");
console.log(hits[0]);
```

[searchColor](/doc/api/searchcolor) finds individual RGB colors. [findColors](/doc/api/findcolors) checks several relative color anchors. [getColors](/doc/api/getcolors) samples many points in one capture. Limit regions and result counts instead of capturing the full screen in a tight loop.

## No match is different from failure

Visual searches return `[[], null]` when nothing matches, and `[null, errorString]` on failure. OCR follows the same pair convention. Do not ignore the error and treat every failure as “not on this screen”. Use a bounded wait/recovery policy and preserve the original error when stopping.