.cjy4dZddlmZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z mZmZejeZehdZdZdd eDZejd ezd zejZejd ezd zejZd;dZehdZehdZdd!Z!d?d$Z"d@d%Z#dAd&Z$ dBdddCd'Z%dddDd(Z&dEd+Z'ehd,Z(dFd.Z)dBdGd1Z*dHd2Z+ dBdId9Z,gd:Z-dS)JuRouting helpers for inbound user-attached images. Two modes: native — attach images as OpenAI-style ``image_url`` content parts on the user turn. Provider adapters (Anthropic, Gemini, Bedrock, Codex, OpenAI chat.completions) already translate these into their vendor-specific multimodal formats. text — run ``vision_analyze`` on each image up-front and prepend the description to the user's text. The model never sees the pixels; it only sees a lossy text summary. This is the pre-existing behaviour and still the right choice for non-vision models. The decision is made once per message turn by :func:`decide_image_input_mode`. It reads ``agent.image_input_mode`` from config.yaml (``auto`` | ``native`` | ``text``, default ``auto``) and the active model's capability metadata. In ``auto`` mode: - If the active model reports ``supports_vision=True`` (via config override or models.dev metadata), we attach natively — vision-capable main models should always see the original pixels, even when an auxiliary vision backend is configured. That auxiliary backend then acts as a *fallback* for sessions whose main model can't take images. - Otherwise, if the user has explicitly configured ``auxiliary.vision`` (provider/model/base_url not ``auto``/empty), we route through the text pipeline so the auxiliary vision backend can describe the image for the text-only main model. - Otherwise (non-vision model, no explicit override), we fall back to text via the default vision_analyze flow. This keeps ``vision_analyze`` surfaced as a tool in every session — skills and agent flows that chain it (browser screenshots, deeper inspection of URL-referenced images, style-gating loops) keep working. The routing only affects *how user-attached images on the current turn* are presented to the main model. ) annotationsN)Path)AnyDictListOptionalTuple>autotextnative) .png.jpg.jpeg.gif.webp.bmpz.tiffz.tifz.heic|c#@K|]}|dVdS).N)lstrip).0es 5/home/ice/.hermes/hermes-agent/agent/image_routing.py r?s,AAahhsmmAAAAAAz/(?\"']+?\.(?:z)(?:\?[^\s<>\"']*)?r strreturnTuple[List[str], List[str]]c t|tr|sggfSg tjd|tjD]=} ||f>tjd|D]=} ||f>d fd }g}t}t|D]}||r | d}tj |} tj |sun#t$rYwxYw||vr||||g}t} t"|D]y}||r | d} | d } | | vrO| | || z||fS) uScan free-form text for image references the model should see. Returns ``(local_paths, urls)``: * ``local_paths`` — absolute (``/``) or home-relative (``~/``) paths whose suffix is an image extension AND whose expanded form exists on disk as a file. Order-preserving, deduplicated. * ``urls`` — ``http(s)://…`` URLs whose path ends in an image extension (a ``?query`` is allowed after the extension). Order-preserving, deduplicated. Matches inside fenced code blocks (``` ``` ```) and inline backticks (`` `…` ``) are skipped so that snippets pasted into a task body for reference aren't mistaken for live attachments. This mirrors the behaviour of ``gateway.platforms.base.BaseAdapter.extract_local_files``. Local paths are validated against the filesystem; URLs are not (the provider fetches them at request time). z```[^\n]*\n.*?```z `[^`\n]+`posintrboolc<tfdDS)Nc3>K|]\}}|cxko|kncVdSN)rsrr s rrz7extract_image_refs.._in_code..rs;77DAq1<<<._in_codeqs'7777J777777rrz .,;:!?)]>)r r!rr") isinstancerrefinditerDOTALLappendstartendset_LOCAL_IMAGE_PATH_REgroupospath expanduserisfileOSErroradd _IMAGE_URL_RErstrip) r mr* local_paths seen_pathsmatchrawexpandedurls seen_urlsurlr)s @rextract_image_refsrFRss( dC 2v )+J [-tRY ? ?0017799aeegg.//// [t , ,0017799aeegg.////888888 K55J%..t44%% 8EKKMM " "  kk!nn7%%c** 7>>(++      H  z ! ! x   8$$$$D%%I''--   8EKKMM " "  kk!nnjj%% )    c C  sE66 FF>1onyestrue>0noofffalserArOptional[bool]c*t|tr|St|tr|dvrt|SdSt|tr<|}|t vrdS|tvrdSdS)z@Return True/False for recognised boolean values, None otherwise.)rNTF)r+r"r!rstriplower _TRUE_TOKENS _FALSE_TOKENS)rAr's r_coerce_capability_boolrVs#t #s &==99 t#s IIKK      4  5 4rrequested_providercfgOptional[Dict[str, Any]]providermodelrYc  t|tsdS|d}t|tr|ni}t|d}||St |dpd}g}|||fD]]} | s|| | dr.| tdd} | r|| ^|d} t| tr| ni} t |D]} | | }t|tr|ni}|d}t|tr|ni}||}t|tr|ni}t|d|d }||cS|d }t|trKt |D]/} |  }|D]}t|tst |d pd }||krh|d}t|tr|ni}||}t|tr|ni}t|d|d }||ccS1dS) uResolve user-declared vision capability from config.yaml. Resolution order, first hit wins: 1. ``model.supports_vision`` (top-level shortcut for the active model) 2. ``providers..models..supports_vision`` (named custom providers — ``provider`` may be the runtime-resolved value ``"custom"``, the runtime's originally requested provider, and/or the user-declared name under ``model.provider``; all are tried. For ``custom:`` syntax, the stripped ```` is also tried as a provider key.) 2b. ``custom_providers`` (legacy list form) ``.models.`` Under (2) and (2b), the per-model capability key may be written as either ``supports_vision`` or the shorter ``vision`` alias; both work. Returns None when no override is set, so the caller falls through to models.dev. Returns False explicitly only when the user wrote a recognised boolean false token. Nr]supports_visionr\rWcustom: providersmodelsvisioncustom_providersname) r+dictgetrVrrRr/ startswithlenfromkeyslistrS)rZr\r]rY model_cfg_raw model_cfgtopconfig_providerprovider_candidates candidatestripped_candidate providers_raw providers_cfgp entry_rawentry models_raw models_cfg per_model_raw per_modelcoercedrdcandidate_name entry_names r_supports_vision_overriders4 c4 tGGG$$M1;M41P1P X VXI !)--0A"B"B C CC  )-- 339r::@@BBO%'((OD??   ""9---    * * ?!*3y>>??!; ! ?#**+=>>>GGK((M5? t5T5T$\MMZ\M ]]. / /  !%%a(( -7 4-H-H P bYYx(( 3=j$3O3O%WZZUW "u-- 5? t5T5T$\MMZ\ ) MM+Y]]8-D-D E E    NNN  ww122"D))#':;; # #I&__..4466N- # # !)T22 v!6!6!<"==CCEEKKMM //&]]844 +5j$+G+GOZZR *u 5 5 -7 t-L-LTMMRT 1MM"3Y]]85L5LMM&"NNNNN' # 4rc* ddlm}t|dpd}t|dpd}t|pd}|r |r||kr|Sn#t $rYnwxYwt |tsdS|d}t |tr|ni}t|dpd}|r|St|dpd} t} td|| fD]} | | |  dr0| | d d d n| d| |d } t | trk| D]h} | | }t |trz._resolve_inference_base_url..As 66617799666rre)agent.auxiliary_clientrrrRrS Exceptionr+rfrgr2filterr:rhsplitrk)rZr\rruntimeruntime_providerrYrlrmrrocandidate_namesrurtrerwburdloweredrvr~s r_resolve_inference_base_urlrs  >>>>>>))*55;<<BBDD22:>>D"EEKKMMSSUU R006688>>@@  . 2DHX2X2XN       c4 rGGG$$M1;M41P1P X VXI9==,,23399;;H)-- 339r::@@BBO #O D8_5 6 6//A 7799   * * /   Q 2 3 3 3 3    ! . . . .GGK((M-&&#  D!%%d++E%&& :..4"55;;==IIIww122"D)) 66o666)  Ii.. Y]]6228b99??AAJ00Z5E5E5G5Gw5V5VY]]:..4"55;;==B    2sB0B44 CCrr"c|pd}|dkrdS|sdS ddlm}||dkS#t$rYdSwxYw)zBTrue when the active provider likely fronts a local Ollama server.rWollamaTFr)detect_local_server_type)rRrSagent.model_metadatarr)r\rrurs r_should_probe_ollama_visionrOs R  &&((AH}}t uAAAAAA''11X== uusA AAct|tsdS|}|tvr|SdS)z5Normalize a config value into one of the valid modes.r )r+rrRrS _VALID_MODES)rAvals r _coerce_moder^sG c3  v ))++    C l 6rcht|tsdS|dpi}t|tsdS|dpi}t|tsdSt|dpd}t|dpd}t|dpd}|dvr|s|sdSd S) ucTrue when the user configured a specific auxiliary vision backend. An explicit override means the user has a dedicated vision backend available; it's used as a *fallback* when the main model can't take images natively. In ``auto`` mode, native vision on a vision-capable main model still wins over this fallback — see issue #29135. F auxiliaryrcr\rWr]r>rWr T)r+rfrgrrRrS)rZauxrcr\r]rs r_explicit_aux_vision_overriderhs  c4 u ''+   $"C c4 u WWX   $"F fd # #u6::j))/R006688>>@@H  7##)r * * 0 0 2 2E6::j))/R006688H<hu 4rcb|s ddlm}t|dpd}t|dpd}t|pd}t|pd}||kr2||kr,t|dpd}n#t $rYnwxYwt ||||} | | S|r|sdSd} dd lm} | ||} n4#t $r'} t d ||| Yd} ~ nd} ~ wwxYw| t| j St||} | s.|pdd krd } t|| rL dd lm}||| }||Sn4#t $r'} t d||| Yd} ~ nd} ~ wwxYwdS)aReturn True/False if we can resolve caps, None if unknown. Consults the user's ``supports_vision`` override in config.yaml first (so custom/local models declared as vision-capable don't fall through to text routing in ``auto`` mode), then falls back to models.dev. rrr\rWr]rYrXN)get_model_capabilitiesu2image_routing: caps lookup failed for %s:%s — %srzhttp://localhost:11434/v1)query_ollama_supports_visionu:image_routing: ollama vision probe failed for %s:%s — %s)rrrrRrSrragent.models_devrloggerdebugr"r_rrrr)r\r]rZrYrr runtime_modellookup_provider lookup_modeloverridecapsrexcrr ollama_visions r_lookup_supports_visionrs$   B B B B B B"##J//52  eggeegg  3 3G < < BCCIIKKM!(.b117799??AAOu{++1133L?22} 7T7T%(''(<==C&&%''#    D )  - H  5t Da;;;;;;%%h66 aaa I8UZ\_````````a D()))*399H /R..006688HDD."8X66   I I I I I I88IIM($$)    LLL           4sBDD DD6E E:E55E:$G;; H,H''H,cld}t|trN|dpi}t|tr"t|d}|dkrdS|dkrdS|rt ||||}nt |||}|durdSt |rdSdS)a~Return ``"native"`` or ``"text"`` for the given turn. Args: provider: active inference provider ID (e.g. ``"anthropic"``, ``"openrouter"``). model: active model slug as it would be sent to the provider. cfg: loaded config.yaml dict, or None. When None, behaves as auto. requested_provider: provider identity before runtime canonicalization. r agentimage_input_moder r rXT)r+rfrgrrr)r\r]rZrYmode_cfg agent_cfgsupportss rdecide_image_input_modersH#tGGGG$$* i & & G#IMM2D$E$EFFH8x6v  A*   1    +8UC@@4x$S))v 6rbytes Optional[str]c||sdS|drdS|drdS|dddvrdSt|d kr|dd d kr|d d d krdS|drdSt|d kr$|d d dkr|d d }|dvrdS|dvrdS|dd dvrdS|dd dkrdS|dd}|ds|drd|vrdSdS)aDetect image MIME from magic bytes. Returns None if unrecognised. Filename-based detection (``mimetypes.guess_type``) is unreliable when upstream platforms lie about content-type. Discord, for example, can serve a PNG with ``content_type=image/webp`` for proxied/animated stickers, custom emoji previews, or images uploaded via certain bots. Anthropic strictly validates that declared media_type matches the actual bytes and returns HTTP 400 on mismatch, so we sniff to be safe. NsPNG  image/pngs image/jpeg>GIF87aGIF89a image/gif sRIFFsWEBP image/webpsBM image/bmpsftyp>avifavisz image/avif>heicheimheisheixhevchevxmif1msf1z image/heic>II*MM*z image/tiffsz image/x-iconisrrrrOptional[bytes]ct ddlm}n+#t$rtdYdSwxYw ddl}|n#t$rYnwxYw ddl}n#t$rYnwxYw ddl m }| ||5}|j dvr| d}|}||dd |cdddS#1swxYwYdS#t$r&}td |Yd}~dSd}~wwxYw) aDecode arbitrary image bytes with Pillow and re-encode as PNG. Returns None if Pillow isn't installed or can't decode the input (rare formats, corrupted bytes, missing optional decoder plugin for HEIC/AVIF, or vector formats like SVG). Caller falls back to skipping the image so the rest of the turn still works. HEIC/HEIF and AVIF need optional Pillow plugins; we try to register them on demand and swallow ImportError so a missing plugin just looks like 'Pillow can't decode this' rather than crashing. r)Imagezimage_routing: Pillow not installed; cannot transcode non-standard image format to PNG. Install with `pip install Pillow` (and `pillow-heif` / `pillow-avif-plugin` for those formats).N)BytesIO>LPLARGBRGBArPNGF)formatoptimizez> " " " " " " " " " " " " " " " " " "  JC   ttttt sl $11A AAA$$ A10A15$DAC:- D:C>>DC>D D7D22D7r6rc|t|}|r|Stjt|\}}|r|dr|S|j}ddddddd|dS) zReturn image MIME type for *path*. If *raw* bytes are provided, magic-byte sniffing wins (authoritative). Otherwise we fall back to ``mimetypes`` then suffix-based defaults. Nzimage/rrrrr)rrr rrr)r mimetypes guess_typerrhsuffixrSrg)r6rAsniffedmime_rs r _guess_mimers  (--  N"3t99--GD! )) [   F    c&, rc ddlm}|t|n?#t$r'}td||Yd}~dSd}~wt $rYnwxYw |}n4#t $r'}td||Yd}~dSd}~wwxYwt||}|tvrTt|}|td||dSt d|j ||}d }tj|d }d |d |S) uWEncode a local image as a base64 data URL at its native size. Size limits are NOT enforced here — the agent retry loop (``run_agent._try_shrink_image_parts_in_messages``) shrinks on the provider's first rejection. Keeping this simple means providers that accept large images (OpenAI 49 MB+, Gemini 100 MB) don't pay a silent quality tax just because one other provider is stricter. Format compatibility IS handled here: if the sniffed MIME isn't one of ``_UNIVERSALLY_SUPPORTED_MIMES`` (i.e. it's something like AVIF, HEIC, BMP, TIFF, or ICO that some providers reject outright), we transcode to PNG with Pillow before declaring media_type. This fixes the user-visible "Could not process image" HTTP 400 from Anthropic on Discord-attached AVIF/HEIC/BMP files. Returns None if the file can't be read OR if the format isn't universally supported AND Pillow can't transcode it (Pillow missing, HEIC/AVIF plugin missing, vector format like SVG, corrupt bytes). The caller reports those paths in ``skipped`` and the rest of the turn proceeds. r)raise_if_read_blockedz6image_routing: blocked local image attachment %s -- %sNu'image_routing: failed to read %s — %s)rAzimage_routing: %s is %s which is not accepted by all major vision providers and could not be transcoded to PNG; skipping this attachment.zIimage_routing: transcoded %s (%s) -> image/png for provider compatibilityrasciizdata:z;base64,)agent.file_safetyrr ValueErrorrwarningr read_bytesr_UNIVERSALLY_SUPPORTED_MIMESrrrebase64 b64encodedecode)r6rrrAr transcodedb64s r_file_to_data_urlrs,  ;;;;;;c$ii(((( OQUWZ[[[ttttt      oo @$LLLttttt t % % %D ///&s++   NN,d     4 W It     3   & &w / /C &4 & & & &&s2! AA  AA!A66 B'B""B' user_text image_paths List[str] image_urlsOptional[List[str]]&Tuple[List[Dict[str, Any]], List[str]]cg}g}g}g}|D]}t|}|r|s#|t |\t |} | s#|t ||dd| id|t ||pgD]J} | pd} | s|dd| id|| K|pd} |s|r~| pd} g} | d|D| d|D| dd | z}d |d g}||||fSg}| r|d | d ||fS) uBuild an OpenAI-style ``content`` list for a user turn. Shape: [{"type": "text", "text": "...\n\n[Image attached at: /local/path]"}, {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}, {"type": "image_url", "image_url": {"url": "https://example.com/a.png"}}, ...] Local paths are read from disk and embedded as base64 ``data:`` URLs. Remote URLs (``http(s)://``) are passed through verbatim — the provider fetches them server-side. The model still sees the pixels either way. For each successfully attached image, a hint is appended to the text part: * local path → ``[Image attached at: ]`` * URL → ``[Image attached: ]`` The hint gives the model a string handle so MCP/skill tools that take an image path or URL argument can be invoked on the same image without an extra round-trip. This parallels the text-mode hint produced by ``Runner._enrich_message_with_vision`` (``vision_analyze using image_url: ``) so behaviour is consistent across both image input modes. Images are attached at their native size. If a provider rejects the request because an image is too large (e.g. Anthropic's 5 MB per-image ceiling), the agent's retry loop transparently shrinks and retries once — see ``run_agent._try_shrink_image_parts_in_messages``. Returns (content_parts, skipped). Skipped entries are local paths that couldn't be read from disk; URLs are never skipped (they're not validated here). image_urlrE)typer rWzWhat do you see in this image?c3"K|] }d|dV dS)z[Image attached at: ]Nr&)rrus rrz-build_native_content_parts..#s.NN!5555NNNNNNrc3"K|] }d|dV dS)z[Image attached: r Nr&)rus rrz-build_native_content_parts..$s.JJq2a222JJJJJJrz  r )r r ) rexistsis_filer/rrrRextendjoin)rrrskipped image_partsattached_paths attached_urlsraw_pathrudata_urlrEr base_text hint_lines combined_textpartss rbuild_native_content_partsrs\LG(*K "N!M - - NNxxzz   NN3x== ) ) ) $Q''  NN3x== ) ) ) *      c(mm,,,,R""yb!!        S!!!! O " " $ $D<< " NN~NNNNNNJJMJJJJJJ$***TYYz-B-BB 06 'N'N&O [!!!g~ E 5 fd33444 '>r)rrrF)r rrr)rArrrO) rZr[r\rr]rrYrrrO)rZr[r\rrr)r\rrrrr")rArrr)rZr[rr"r%) r\rr]rrZr[rYrrrO) r\rr]rrZr[rYrrr)rArrr)rArrr)r6rrArrr)r6rrr)rrrrrrrr).__doc__ __future__rrloggingrr5r,pathlibrtypingrrrrr getLogger__name__r frozensetr _IMAGE_EXTSr_IMAGE_EXT_PATTERNcompile IGNORECASEr3r;rFrTrUrVrrrrrrrrrrrrr__all__r&rrr,s$$L#"""""  33333333333333  8 $ $y33344  XXAA[AAAAA "rz69KKfTM !$669OOM BBBBZy33344  55566 ,! ]]]]]]@8888v    <%)G ! GGGGGG^! ------~0000| )y***   1111h     48'8'8'8'|'+VVVVVr   r