Macro Handler

Macro Handler

Android Automation Platform

Object MethodNetwork and Integration

AI.askVision(prompt, imageBase64, providerId?, mimeType?, apiKey?, timeoutMs?, localTask?, localMaxTokens?)

Sends a prompt with a Base64 image to the AI provider; returns a table with the same fields as AI.ask.

Documented for app version 1.0.51Updated:

Detailed Explanation

This section explains when to use the API, how to call it, and which structures it works best with in production flow.

How To Call It

AI is a global table: call its functions with a dot (AI.<method>(...)), not a colon.

When To Use It

Use this to analyze a screenshot with a cloud model. The local pixel mode of MH Visual Decision is a separate flow.

Parameters and Return

prompt and imageBase64 are required. A nil providerId uses app settings. mimeType must match the image. credentialReference accepts nil or a saved Visual Decision reference, never a raw API key. timeoutMs is bounded to 1000–60000 ms.

Best Combined With

Check ok, text, errorCode and error in the result table. This call does not tap the screen. Visual Decision retries, thresholds and outcome branches require additional Lua logic. Bound request frequency in loops.

Example Usage

The snippet below is a starter pattern that can be applied directly in runtime flow.

lua
Try in Editor →
-- Uses the configured provider and credential; do not embed API keys.
local image = Capture.regionToBase64(Region(), "jpeg", 60)
if image ~= nil and image ~= "" then
  local response = AI.askVision("Describe the visible targets", image, nil, "image/jpeg", nil, 30000)
  if response.ok then print(response.text) else print(response.errorCode, response.error) end
else print(System.lastScanError()) end

Copyable Progressive Examples

From foundation to combined usage, each level is provided as a separate code block so you can copy the level you need and adapt it directly.

Foundation

Shows the shortest direct way to call the API.

Foundation
lua
Try in Editor →
-- Uses the configured provider and credential; do not embed API keys.
local image = Capture.regionToBase64(Region(), "jpeg", 60)
if image ~= nil and image ~= "" then
  local response = AI.askVision("Describe the visible targets", image, nil, "image/jpeg", nil, 30000)
  if response.ok then print(response.text) else print(response.errorCode, response.error) end
else print(System.lastScanError()) end

Simple

Wraps the base call with minimal flow control.

Simple
lua
Try in Editor →
local stepOk = true
-- Uses the configured provider and credential; do not embed API keys.
local image = Capture.regionToBase64(Region(), "jpeg", 60)
if image ~= nil and image ~= "" then
  local response = AI.askVision("Describe the visible targets", image, nil, "image/jpeg", nil, 30000)
  if response.ok then print(response.text) else print(response.errorCode, response.error) end
else print(System.lastScanError()) end
if stepOk then
  wait(200)
end

Practical Flow

A practical pattern for real macros with pcall, logging, and guards.

Practical Flow
lua
Try in Editor →
local ok, err = pcall(function()
  -- Uses the configured provider and credential; do not embed API keys.
  local image = Capture.regionToBase64(Region(), "jpeg", 60)
  if image ~= nil and image ~= "" then
    local response = AI.askVision("Describe the visible targets", image, nil, "image/jpeg", nil, 30000)
    if response.ok then print(response.text) else print(response.errorCode, response.error) end
  else print(System.lastScanError()) end
end)

if not ok then
  print("API step failed: AI.askVision: " .. tostring(err))
  requestStop()
end

Detailed

This level packages the API into a reusable helper with error reporting.

Detailed
lua
Try in Editor →
-- Use configured credentials and handle failed responses before continuing.
local function run_askvision_step()
  -- Uses the configured provider and credential; do not embed API keys.
  local image = Capture.regionToBase64(Region(), "jpeg", 60)
  if image ~= nil and image ~= "" then
    local response = AI.askVision("Describe the visible targets", image, nil, "image/jpeg", nil, 30000)
    if response.ok then print(response.text) else print(response.errorCode, response.error) end
  else print(System.lastScanError()) end
end

local ok, err = pcall(run_askvision_step)
if not ok then
  toast("Step failed")
  print(err)
end

Combined

Combines the API with related structures to form a more realistic workflow.

Combined
lua
Try in Editor →
-- Uses the configured provider and credential; do not embed API keys.
local image = Capture.regionToBase64(Region(), "jpeg", 60)
if image ~= nil and image ~= "" then
  local response = AI.askVision("Describe the visible targets", image, nil, "image/jpeg", nil, 30000)
  if response.ok then print(response.text) else print(response.errorCode, response.error) end
else print(System.lastScanError()) end