Macro Handler

Macro Handler

Android Automation Platform

Object MethodScanning and Detection

Capture.isolateObject(image, backgroundTolerance?, timeoutMs?)

Separates the object of a Base64 reference crop from its border background and returns it as a PNG with a transparent surround. Searching that image with Region:find excludes its transparent background pixels. Matching still depends on visible object detail and the chosen threshold. The background is estimated from the crop border, which must be mostly background.

Documented for app version 1.0.59Updated:

Detailed Explanation

This section explains when to use the API, how to call it, and which structures it works best with in production flow.

How To Call It

Capture is a global table: call its functions with a dot (Capture.<method>(...)), not a colon.

When To Use It

In detection flow, refresh the screen with Snap.screenRefresh() first, then narrow the search area and confidence using Region and FindParam.

Parameters and Return

image is a Base64 PNG/JPEG reference. Optional backgroundTolerance 0–64 (24) is the per-channel distance from the border colour; timeoutMs 1–60000 (2000) is the time budget. Returns ok, status, reason, image (PNG Base64 with a transparent surround), width, height, foreground (object pixel share), background (dominant border colour), eroded, frameRemoved (true when detached peripheral frame fragments were removed) and left/top/objectWidth/objectHeight (object bounds inside the crop). Status: ok, ambiguous_background, insufficient_foreground, insufficient_background, insufficient_detail, crop_too_small, crop_too_large, timeout. Invalid options or images raise errors; stop requests propagate. Each image is limited to 950000 Base64 characters and 4194304 pixels. Enclosed areas in the border colour count as background; flat single-colour objects are rejected. Pass the returned image to Asset.image with the data:image/png;mh-coordinate-space=screen;base64, prefix; scaleMin/scaleMax/scaleStep in the find option table set the template scale window. Use strict_image with rotationFallback=false to search in the same orientation. Frame cleanup removes only small, detached, background-like fragments surrounding at least three sides of a dominant interior subject; it is not a guarantee for every frame or complex background. Images are not sent to a server.

Best Combined With

It is typically combined with Snap, Region, Asset, and FindParam; you can feed the result into quickTap(), click(), or a follow-up OCR step.

Example Usage

The snippet below is a starter pattern that can be applied directly in runtime flow.

lua
Try in Editor →
local reference = Capture.regionToBase64(Region(20, 100, 160, 160), "png", 100)
local isolated = Capture.isolateObject(reference, 24, 2000)
if isolated.ok then
  local template = Asset.image("data:image/png;mh-coordinate-space=screen;base64," .. isolated.image):mScore(0.8)
  local hit = Region(0, 300, 720, 600):find(template, {engine = "strict_image", rotationFallback = false, scaleMin = 0.5, scaleMax = 1.5, scaleStep = 0.04})
  if hit then click(hit) end
else
  print(isolated.status .. ": " .. isolated.reason)
end

Copyable Progressive Examples

From foundation to combined usage, each level is provided as a separate code block so you can copy the level you need and adapt it directly.

Foundation

Shows the shortest direct way to call the API.

Foundation
lua
Try in Editor →
local reference = Capture.regionToBase64(Region(20, 100, 160, 160), "png", 100)
local isolated = Capture.isolateObject(reference, 24, 2000)
if isolated.ok then
  local template = Asset.image("data:image/png;mh-coordinate-space=screen;base64," .. isolated.image):mScore(0.8)
  local hit = Region(0, 300, 720, 600):find(template, {engine = "strict_image", rotationFallback = false, scaleMin = 0.5, scaleMax = 1.5, scaleStep = 0.04})
  if hit then click(hit) end
else
  print(isolated.status .. ": " .. isolated.reason)
end

Simple

Wraps the base call with minimal flow control.

Simple
lua
Try in Editor →
local stepOk = true
local reference = Capture.regionToBase64(Region(20, 100, 160, 160), "png", 100)
local isolated = Capture.isolateObject(reference, 24, 2000)
if isolated.ok then
  local template = Asset.image("data:image/png;mh-coordinate-space=screen;base64," .. isolated.image):mScore(0.8)
  local hit = Region(0, 300, 720, 600):find(template, {engine = "strict_image", rotationFallback = false, scaleMin = 0.5, scaleMax = 1.5, scaleStep = 0.04})
  if hit then click(hit) end
else
  print(isolated.status .. ": " .. isolated.reason)
end
if stepOk then
  wait(200)
end

Practical Flow

A practical pattern for real macros with pcall, logging, and guards.

Practical Flow
lua
Try in Editor →
local ok, err = pcall(function()
  local reference = Capture.regionToBase64(Region(20, 100, 160, 160), "png", 100)
  local isolated = Capture.isolateObject(reference, 24, 2000)
  if isolated.ok then
    local template = Asset.image("data:image/png;mh-coordinate-space=screen;base64," .. isolated.image):mScore(0.8)
    local hit = Region(0, 300, 720, 600):find(template, {engine = "strict_image", rotationFallback = false, scaleMin = 0.5, scaleMax = 1.5, scaleStep = 0.04})
    if hit then click(hit) end
  else
    print(isolated.status .. ": " .. isolated.reason)
  end
end)

if not ok then
  print("API step failed: Capture.isolateObject: " .. tostring(err))
  requestStop()
end

Detailed

This level packages the API into a reusable helper with error reporting.

Detailed
lua
Try in Editor →
-- In detection flow, refresh the screen with Snap.screenRefresh() first, then narrow the search area and confidence using Region and FindParam.
local function run_isolateobject_step()
  local reference = Capture.regionToBase64(Region(20, 100, 160, 160), "png", 100)
  local isolated = Capture.isolateObject(reference, 24, 2000)
  if isolated.ok then
    local template = Asset.image("data:image/png;mh-coordinate-space=screen;base64," .. isolated.image):mScore(0.8)
    local hit = Region(0, 300, 720, 600):find(template, {engine = "strict_image", rotationFallback = false, scaleMin = 0.5, scaleMax = 1.5, scaleStep = 0.04})
    if hit then click(hit) end
  else
    print(isolated.status .. ": " .. isolated.reason)
  end
end

local ok, err = pcall(run_isolateobject_step)
if not ok then
  toast("Step failed")
  print(err)
end

Combined

Combines the API with related structures to form a more realistic workflow.

Combined
lua
Try in Editor →
Snap.screenRefresh()
local reference = Capture.regionToBase64(Region(20, 100, 160, 160), "png", 100)
local isolated = Capture.isolateObject(reference, 24, 2000)
if isolated.ok then
  local template = Asset.image("data:image/png;mh-coordinate-space=screen;base64," .. isolated.image):mScore(0.8)
  local hit = Region(0, 300, 720, 600):find(template, {engine = "strict_image", rotationFallback = false, scaleMin = 0.5, scaleMax = 1.5, scaleStep = 0.04})
  if hit then click(hit) end
else
  print(isolated.status .. ": " .. isolated.reason)
end
toast("Detection step completed")