ߩ^j1 UdZddlmZddlmZmZdddiddgd d d d dgd dd ddddddddddddddddddddddddd d!did"d"d#d$d%dgd&d'd d(d dgd)d*d+d,d-dd.dd/dd0dd1d d!did"d"d2d$d3d d!did"d"d4d$d5dgd6d7d d8dd9dd:dd;dddd?dd@dAdddBdCgdDd d@dEdd@dFddGdgdHdIZdJedK<dOdMZdNS)PuNSchema for the generic `computer_use` tool. Model-agnostic. Any tool-calling model can drive this. Vision-capable models should prefer `capture(mode='som')` then `click(element=N)` — much more reliable than pixel coordinates. Pixel coordinates remain supported for models that were trained on them (e.g. Claude's computer-use RL). ) annotations)AnyDict computer_useuDrive the desktop in the background via cua-driver — screenshots, mouse, keyboard, scroll, drag — without stealing the user's cursor or keyboard focus. Supported on macOS, Windows, and Linux. Preferred workflow: call with action='capture' (mode='som' gives numbered element overlays), then click by `element` index for reliability. Pixel coordinates are supported for models trained on them. Works on any window — hidden, minimized, or behind another app. Requires cua-driver to be installed.objectactionstring)captureclick double_click right_click middle_clickdragscrolltypekey set_valuewait list_apps list_windows focus_appu Which action to perform. `capture` is free (no side effects). All other actions require approval unless auto-approved. Use `set_value` for select/popup elements and sliders — it selects the matching option directly without opening the native menu (no focus steal).)renum descriptionmode)somvisionaxuCapture mode. `som` (default) is a screenshot with numbered overlays on every interactable element plus the AX tree — best for vision models, lets you click by element index. `vision` is a plain screenshot. `ax` is the accessibility tree only (no image; useful for text-only models).appuOptional. Limit capture/action to a specific app (by name, e.g. 'Safari', or bundle ID, 'com.apple.Safari'). If omitted, operates on the frontmost app's window. Pass app='screen' (or 'desktop') to capture the OS desktop/shell surface — e.g. to see the wallpaper or click the taskbar. Note: capture is per-window; a single image cannot span multiple monitors, so on a multi-screen setup capture one window or display at a time.)rrpidintegerzqOptional exact process target for action='capture'. Pair with window_id when discovery cannot resolve an X11 app. window_idzOptional exact native window target for action='capture'. Pair with pid when an external cua-driver list_windows lookup has already identified the window. max_elementsuOptional cap on the AX `elements` array returned by `action='capture'`. Default 100, hard maximum 1000. Dense UIs (Electron apps such as Obsidian or VS Code, JetBrains IDEs) can publish 500+ AX nodes — capping prevents a single capture from blowing session context. When the cap trims the response, `total_elements` and `truncated_elements` are surfaced in the result so you can re-call with `app=` to narrow scope or raise `max_elements` when the full tree is required. Has no effect on `mode='som'` / `mode='vision'` when a screenshot is included in the response; only the rare image-missing fallback returns an `elements` array and is subject to the cap.di)rrdefaultminimummaximumelementzoThe 1-based SOM index returned by the last `capture(mode='som')` call. Strongly preferred over raw coordinates. coordinatearrayrzPixel coordinates [x, y] relative to the captured window screenshot (top-left origin). Only use this if no element index is available.)ritemsminItemsmaxItemsrbutton)leftrightmiddlezMouse button. Defaults to left. modifiers) cmdshiftoptionaltctrlfnwinwindowssupermeta)rrz%Modifier keys held during the action.)rr,r from_elementzSource element index (drag). to_elementzTarget element index (drag).from_coordinatez3Source [x,y] (drag; use when no element available). to_coordinatez3Target [x,y] (drag; use when no element available). direction)updownr0r1zScroll direction.amountzScroll wheel ticks. Default 3.valuezFor action='set_value': the value to set on the element. For AXPopUpButton / select dropdowns, pass the option's display label (e.g. 'Blue'). For sliders and other AXValue-settable elements, pass the numeric or string value.z+Text to type (respects the current layout).zUKey combo, e.g. 'cmd+s', 'ctrl+alt+t', 'return', 'escape', 'tab'. Use '+' to combine.numberzSeconds to wait. Max 30.booleanuOnly for action='focus_app'. If true, brings the window to front (DISRUPTS the user). Default false — input is routed to the app without raising, matching the background co-work model. background foregrounduHow input is delivered, for the input actions (click, double_click, right_click, drag, scroll, type, key). `background` (DEFAULT) routes input to the target without raising it or stealing focus — the co-work model. `foreground` briefly fronts the window, acts, then restores the prior frontmost app. Only escalate to `foreground` when a background attempt did NOT land — i.e. a prior result had `effect: 'suspected_noop'`, `code: 'background_unavailable'`, or `escalation.recommended: 'foreground'`. Do not predict it from the app being Electron/Chromium; react to the returned signal. Foreground is a visible focus change and needs its own approval.zOptional, pairs with delivery_mode='foreground'. Keep the target fronted after the action instead of restoring the previous app, to avoid a per-call flash across a short sequence of foreground actions. Default false.zIf true, take a follow-up capture after the action and include it in the response. Saves a round-trip when you need to verify an action's effect.)textkeysseconds raise_window delivery_modebring_to_front capture_after)r propertiesrequired)namer parametersDict[str, Any]COMPUTER_USE_SCHEMAreturnctS)z2Return the generic OpenAI function-calling schema.)rW;/home/ice/.hermes/hermes-agent/tools/computer_use/schema.pyget_computer_use_schemar] s r[N)rXrV) __doc__ __future__rtypingrrrW__annotations__r]rZr[r\rbs#"""""  d  "H'd 8  ///-  9d P  7  Qd l !Omd z !@{d J ! *)Kd x !'yd H  ),*  Id ^  333@_d h $ G  id @ Y-KMMAd D 9+IKKEd H  ),1T   Id T  ),1T Ud b  7772cd l !?md v  Swd L!L !;!9 "=!%|4 . ("E"Byd d d JJOhhv'v'vvvvrr[