AiRobot¶
AiRobot implements fully-automatic interactions with target window or screen region (keyboard and mouse). Should be used when AI is unable to find reasonable entries in other page objects. This way of interacting is last resort. It may be applied to complex, exploratory style actions.
Action Summary¶
| Action | Description |
|---|---|
| DoConfigure | Set common execution parameters and limitations. |
| DoFullScreen | Do fully automatic AI interactions with the full screen. |
| DoObject | Do fully automatic AI interactions with a screen region occupied by a given object. |
| DoScreenRegion | Do fully automatic AI interactions with a specified screen region. |
| DoWebBrowser | Do fully automatic AI interactions with currently active web browser window. |
| DoWindow | Do fully automatic AI interactions with a window matched by title. |
| SetSelfCheck | Enables self-check mode. |
Action Detail¶
DoConfigure¶
Set common execution parameters and limitations.
AiRobot.DoConfigure(system_prompt, timeout, n_last_images, max_tokens, token_limit, vendor)
Parameters:
| Name | Type | Description |
|---|---|---|
| system_prompt | string | Additional prompt to be used for all interactions. |
| timeout | number | Maximum time to wait, in milliseconds Optional, Default: "600000". |
| n_last_images | number | Number of images to remember. Optional, Default: "5". |
| max_tokens | number | Maximum amount of tokens per response. Optional, Default: "1000". |
| token_limit | number | Token limit to be used for whole prompt. Optional, Default: "1000000". |
| vendor | string | LLM provider to use for Robot: bedrock, openai, or qwen.Optional, Default: "bedrock". |
DoFullScreen¶
Do fully automatic AI interactions with the full screen.
AiRobot.DoFullScreen(prompt, timeout, n_last_images, max_tokens, token_limit)
Parameters:
| Name | Type | Description |
|---|---|---|
| prompt | string | Clear explanation of what you want AiRobot to do with a given target. |
| timeout | number | Maximum time to wait, in milliseconds Optional, Default: "600000". |
| n_last_images | number | Number of images to remember. Optional, Default: "5". |
| max_tokens | number | Maximum amount of tokens per response. Optional, Default: "1000". |
| token_limit | number | Token limit to be used for whole prompt. Optional, Default: "1000000". |
DoObject¶
Do fully automatic AI interactions with a screen region occupied by a given object.
AiRobot.DoObject(prompt, objectId, timeout, n_last_images, max_tokens, token_limit)
Parameters:
| Name | Type | Description |
|---|---|---|
| prompt | string | Clear explanation of what you want AiRobot to do with a given target. |
| objectId | objectid | SeSObject | Object whose screen rectangle will be shown to AI robot to interact with. |
| timeout | number | Maximum time to wait, in milliseconds Optional, Default: "600000". |
| n_last_images | number | Number of images to remember. Optional, Default: "5". |
| max_tokens | number | Maximum amount of tokens per response. Optional, Default: "1000". |
| token_limit | number | Token limit to be used for whole prompt. Optional, Default: "1000000". |
DoScreenRegion¶
Do fully automatic AI interactions with a specified screen region.
AiRobot.DoScreenRegion(prompt, x, y, w, h, timeout, n_last_images, max_tokens, token_limit)
Parameters:
| Name | Type | Description |
|---|---|---|
| prompt | string | Clear explanation of what you want AiRobot to do with a given target. |
| x | number | X coordinate of the screen region. |
| y | number | Y coordinate of the screen region. |
| w | number | Width of the screen region. |
| h | number | Height of the screen region. |
| timeout | number | Maximum time to wait, in milliseconds Optional, Default: "600000". |
| n_last_images | number | Number of images to remember. Optional, Default: "5". |
| max_tokens | number | Maximum amount of tokens per response. Optional, Default: "1000". |
| token_limit | number | Token limit to be used for whole prompt. Optional, Default: "1000000". |
DoWebBrowser¶
Do fully automatic AI interactions with currently active web browser window.
AiRobot.DoWebBrowser(prompt, timeout, n_last_images, max_tokens, token_limit)
Parameters:
| Name | Type | Description |
|---|---|---|
| prompt | string | Clear explanation of what you want AiRobot to do with a given target. |
| timeout | number | Maximum time to wait, in milliseconds Optional, Default: "600000". |
| n_last_images | number | Number of images to remember. Optional, Default: "5". |
| max_tokens | number | Maximum amount of tokens per response. Optional, Default: "1000". |
| token_limit | number | Token limit to be used for whole prompt. Optional, Default: "1000000". |
DoWindow¶
Do fully automatic AI interactions with a window matched by title.
AiRobot.DoWindow(prompt, window_title, timeout, n_last_images, max_tokens, token_limit)
Parameters:
| Name | Type | Description |
|---|---|---|
| prompt | string | Clear explanation of what you want AiRobot to do with a given target. |
| window_title | string | Exact window title or regex to match window title, i.e. Calculator or regex:Calc.* |
| timeout | number | Maximum time to wait, in milliseconds Optional, Default: "600000". |
| n_last_images | number | Number of images to remember. Optional, Default: "5". |
| max_tokens | number | Maximum amount of tokens per response. Optional, Default: "1000". |
| token_limit | number | Token limit to be used for whole prompt. Optional, Default: "1000000". |
SetSelfCheck¶
Enables self-check mode. After each step, AiRobot will take a screenshot and evaluate if the right outcome was achieved before moving on.
AiRobot.SetSelfCheck()