Computer Control

Computer control lets LanMate operate your desktop like a person: see the screen, recognize buttons, click, type, drag — and open and switch apps. On macOS it can even work in the background while you do something else, without taking over your mouse and keyboard.

Enable

Settings → Computer Control:

1. Turn on the “Enable Computer Control” master switch
2. On macOS, grant Accessibility (mouse and keyboard, interface elements) and Screen Recording permissions
3. Restart LanMate after granting for it to take effect

On Linux it needs xdotool and python3-pyatspi; if a dependency is missing, the settings page shows the install command.

What it can see

Three screenshot return modes can be chosen in settings:

The matching tools are computer_screenshot / computer_screenshot_window, computer_get_ui_tree, computer_get_app_state, and computer_get_active_app.

What it can do

Operations that change your computer, such as clicking and typing, ask you first by default; you can choose “Always allow desktop app operations” or approve app by app.

Background actions (macOS)

LanMate can operate another app while you keep using the computer: your mouse and keyboard are completely unaffected. Useful when you want it to fill in forms or tidy documents while you carry on with your own work.

Teaching recording

This is a highlight of 1.0.x: the flow you perform by hand is distilled into a reusable skill.

Settings → Computer Control → Teaching recording:

• Start manually: click “Start recording”, do the work as usual, then click “Stop and summarize”
• Automatic per task: LanMate records every computer operation and summarizes automatically when the task ends

When recording ends, only the desensitized steps are submitted (trigger, inputs, actions, verification); screenshots stay on this machine with the skill. If there is no reusable fixed sequence, it tells you plainly that no skill was generated.

Learned skills are managed on the same settings page: you can see replay counts, or “Forget skill” — the skill directory, steps, and keyframes are deleted together and cannot be recovered.

Taking over Office documents

With computer control enabled, LanMate can also take over the Word / Excel / PPT you currently have open:

• Reads what you’re currently editing and understands its context before acting
• Edits in place per your instructions (e.g., “add a version of the risk note after this paragraph”) — no closing windows, no exporting files
• The first call requires system automation permission; writes still go through approval

For the fuller human-machine co-editing experience (side conversation + tracked revisions), see Office Document Editing in Files & Artifacts.

Picture-in-picture monitor

Settings → Computer Control → PiP monitor: whichever app LanMate operates, its live view is mirrored into a pinned small window. You can see exactly what it’s doing, while your mouse and keyboard stay free.

Safety guardrails

• Master switch off by default, and you can turn it off at any time
• Changing operations ask first: confirmation appears before clicks, typing, and drags; authorization can be remembered per app
• Stricter for automated tasks: Lansenger channels and scheduled tasks may only operate apps on the allowlist; everything else is refused
• Password fields untouched: the agent stops at password entry and lets you type it yourself
• Operation telemetry off by default: it is used only as local diagnostic data; to turn operations into skills, use teaching recording

Platform support