.cj+^dZddlmZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z mZddlmZmZmZmZddlmZddlmZdd lmZdd lmZmZejeZdFdZ dGdZ!dHdZ"dIdZ#dJdZ$dKdZ%dLd$Z&dMd)Z'dNd+Z(dOd/Z)dPd7Z*eGd8d9Z+ dQd:d;dRdEZ,dS)Su?Per-turn setup for ``run_conversation`` (the turn prologue). ``run_conversation`` opened with ~470 lines of straight-line setup before the tool-calling loop ever started: stdio guarding, runtime-main wiring, retry-counter resets, user-message sanitization, todo/nudge-counter hydration, system-prompt restore-or-build, session-row creation (before compression, whose DB writes reference the row), preflight context compression, the ``pre_llm_call`` plugin hook, external-memory prefetch, and crash-resilience persistence (last, so the user row is written once with its final ``api_content`` sidecar). All of that is *prologue* — it runs once per turn, has no back-references into the loop, and produces a fixed set of values the loop then consumes. ``TurnContext`` captures those produced values; ``build_turn_context`` performs the setup work and returns one. ``run_conversation`` is left to unpack the context and run the loop, shrinking the orchestrator by the full prologue. The builder still mutates ``agent`` heavily (counters, thread id, cached prompt, session DB) exactly as the inline code did — those side effects are the point. The ``TurnContext`` it returns carries only the *locals* the loop reads back. Behavior is identical to the original inline prologue; this is a pure move-and-name refactor with no semantic change. ) annotationsN) dataclass)AnyDictListMappingOptional)IDLE_COMPACTION_STATUS_TEMPLATE%PREFLIGHT_COMPRESSION_STATUS_TEMPLATEcompression_skipped_due_to_lock&conversation_history_after_compression)#automatic_compaction_status_message)IterationBudget)build_memory_context_block)estimate_messages_tokens_roughestimate_request_tokens_roughcontentrext_prefetch_cachestrplugin_user_contextreturn Optional[str]ct|tsdSg}|r&t|}|r|||r|||sdS|dzd|zS)ueCompose the API-bound content of the current turn's user message. Sources: memory-manager prefetch + ``pre_llm_call`` plugin context with target="user_message" (the default). Both are appended to the *API copy* of the user message only — the stored content stays clean. This is the single source of that composition. The prologue stamps the result onto the live message as ``api_content`` (persisted alongside the clean content) and the ``api_messages`` build in ``conversation_loop`` sends the same helper's output, so the persisted sidecar can never drift from the bytes on the wire — which is the whole prompt-cache invariant: what turn N sends must be what turn N+1 replays. Returns ``None`` when nothing is injected (multimodal/non-string content, or no ephemeral context), meaning the message is sent as-is. N ) isinstancerrappendjoin)rrr injectionsfenceds 4/home/ice/.hermes/hermes-agent/agent/turn_context.pycompose_user_api_contentr!3s* gs # #tJ&+,>??  &   f % % %/-... t V fkk*55 55api_msgDict[str, Any]c|dd}t|tr|r|ddvr||d<|S)a}Pop the ``api_content`` sidecar and substitute it into ``content``. Used at every API-bound message-build site (the ``api_messages`` build in ``conversation_loop``, the max-iterations summary in ``chat_completion_helpers``, the chat-completions transport). The sidecar carries the exact bytes previously sent to the API for this message when they differ from the clean stored content; substituting it here keeps the provider prompt-cache prefix byte-stable across turns. Returns the popped sidecar string (for callers that need the value for current-turn composition logic) or ``None`` when absent. api_contentNrole)user assistantr)poprrget)r#sidecars r substitute_api_contentr-VsZkk-..G7C  % % KK  #8 8 8$  Nr"msgNonec2|dddS)uDrop the ``api_content`` sidecar from a message whose content was rewritten. Called from every content-rewrite path (historical image strip, merge-summary-into-tail, consecutive-user repair merge, stale-confirmation redaction). Replaying the pre-rewrite sidecar would resend exactly what the rewrite removed, so it must be dropped — the cost is one cache boundary miss, never wrong content. r&N)r*)r.s r drop_stale_api_contentr1msGGM4     r"Mapping[str, Any]c^|d}t|tr|ndS)zExtract the ``api_content`` sidecar from a message dict for persistence. Shared by the gateway/branch forwarding sites that copy the sidecar into a new row. Returns the string sidecar or ``None`` when absent/non-string. r&N)r+rr)r.vs r extract_api_content_sidecarr5ys.  A1c"" ,11,r"agentct|ddpd}t|dr d|_n#t$rYnwxYwt |t r|ndS)aMPop the gateway's per-turn must-deliver notes off the agent (one-shot). The gateway relocates volatile per-turn facts OUT of the ephemeral system prompt (auto-reset notes, the first-contact intro, voice-channel changes) and delivers them on the current user message via the api_content sidecar instead, so the composed system prompt stays byte-stable turn-over-turn. It stages the rendered notes on ``agent._gateway_turn_context_notes`` right before ``run_conversation``; this consumes them so a cached agent can never replay a stale note on a later turn. _gateway_turn_context_notes)getattrhasattrr8 Exceptionrr)r6notess r "consume_gateway_turn_context_notesr>sw E8" = = CEu344 02E - -    D uc** 2552s - ::r=boolc|rt|tsdS |d|ddS#t$rYdSwxYw)uDeliver must-deliver notes on a multimodal (list) user message. ``compose_user_api_content`` returns ``None`` for non-string content, so sidecar-borne facts would silently drop on image/attachment turns. For gateway must-deliver notes we instead append a text part to the content list in place — the part becomes durable message content (persisted and replayed as-is), which keeps the wire and the transcript byte-identical. Returns ``True`` when a part was appended. Ftext)typerAT)rlistrr<)rr=s r "append_notes_to_multimodal_contentrDsh  7D11u66777t uus5 AAmessages List[Any] user_messageintc d}tt|dz ddD]^}||}t|tr|ddks9|dkr|}|d|kr|cS_|S)uLocate this turn's user message after compaction rebuilt ``messages``. Compression replaces list entries with fresh copies (and may append a todo-snapshot user message or a restored user turn AFTER the surviving copy of the current turn's message), so a pre-compression index is meaningless. Prefer the LAST user message whose content exactly matches this turn's text — the surviving copy in the common case — so the injection stamp and the #48677 persist override can't land on a todo-snapshot or historical row. Fall back to the last user message when no exact match survives (merge-summary-into-tail rewrites the content but the trackers still need a live anchor). Returns -1 when the list has no user message at all. r'r(rr)rangelenrdictr+)rErGfallbackir.s r reanchor_current_turn_user_idxrQsH 3x==1$b" - -qk3%% #''&//V*C*C  a<<H 779   - -HHH . Or"orig_lennew_len orig_tokens new_tokensc0||krdS|dko||dzkS)uFReturn ``True`` if a compression pass materially reduced the request. Compression can succeed by summarising message contents — reducing the estimated request token count — without reducing the message row count. Treating row count as the sole progress signal false-positives on size-only wins and surfaces a misleading "Cannot compress further" failure even when post-compression tokens are well below the model context window. See issue #39548 for an observed case: 220 → 220 messages, ~288k → ~183k tokens on a 1M-context model still triggered auto-reset. The token reduction must be *material* (>5%) to count as progress — the same floor the overflow-handler retry path uses (conversation_loop.py, #39550) — so a sub-5% wobble doesn't keep the multi-pass loop spinning. Trffffff?)rRrSrTrUs r _compression_made_progressrYs-$t ? >zK$,>>>r"threshold_tokensc,||ko|dko||dzkS)awWhether an over-threshold request merits another immediate summary. Row-count progress is enough to prove that a compression boundary was real, but not enough to justify another expensive pass before trying the provider. Continue only when the request remains over threshold *and* the previous pass materially reduced its estimated token pressure (>5%). rrWrX)rTrUrZs r ,_compression_warrants_another_preflight_passr\s0 && , !O , t+ +r"List[Dict[str, Any]]protect_first_nprotect_last_nc^t|||zdzkrdSt||kS)uCheap gate for the (expensive) full preflight token estimate. Returns ``True`` when either: (a) message count exceeds the protected ranges (the historical gate), or (b) a cheap char-based estimate already crosses the configured threshold — the few-but-huge case from issue #27405 that the count-only gate would silently skip (a handful of very large messages never trips the count condition, so compression was never attempted and the turn hit a hard context-overflow error). Branch (b) uses ``estimate_messages_tokens_rough`` (the shared char-based estimator) so a single large base64 image isn't mistaken for ~250K tokens. It intentionally undercounts vs. the full request estimate — it omits the system prompt and tool schemas — because it is only a *hint* deciding whether to pay for the authoritative ``estimate_request_tokens_rough``, which (together with ``should_compress``) makes the real decision. rKT)rMr)rEr^r_rZs r _should_run_preflight_estimateras8. 8}}7!;;;t )( 3 37G GGr"enabledidle_after_secondsidle_gap_secondsfloattokens floor_tokenscooldown_activec:|r|dkrdS||krdS|rdS||kS)ahDecide whether an idle-triggered compaction should run this turn. Idle compaction is opt-in (``idle_after_seconds <= 0`` disables it). It fires when a session resumes after a wall-clock gap of at least ``idle_after_seconds`` since its last activity, so a long-lived thread that is paused and later resumed compacts its accumulated history up front instead of re-reading it on every subsequent turn. It is orthogonal to the token-threshold trigger: it does NOT require the context to exceed ``threshold_tokens``. It still skips work when the context is at or below ``floor_tokens`` (the size compaction would reduce *to*), so a small idle thread never pays for a summarisation that saves nothing, and it defers to an active compression-failure cooldown. Pure predicate so the policy is unit-testable without a live agent. rFrXrbrcrdrfrgrhs r _should_idle_compactrk sE2 (A--u,,,uu L  r"ceZdZUdZded<ded<ded<ded <d ed <ded <ded <ded<dZded<dZded<dZded<dZded<dS) TurnContextzCValues produced by the turn prologue and consumed by the turn loop.rrGroriginal_user_messager]rEOptional[List[Dict[str, Any]]]conversation_historyractive_system_prompteffective_task_idturn_idrHcurrent_turn_user_idxFr?should_review_memoryr9rrpreflight_compression_blockedN) __name__ __module__ __qualname____doc____annotations__rurrrvrXr"r rmrm+sMM""""8888''''LLL!&&&&&!!!!!     */!//////r"rmF) moa_activesystem_messagerprotask_idpersist_user_message Optional[Any]persist_user_timestampOptional[float]r|c.*^| | j| tdd ddlm}|tddpdtddpdtddpdtd dpdtd dpdtd dpdtd dpd n#t $rYnwxYw tdds,ddl}d|jvrddlm }m }|r |dn,#t $rt ddYnwxYwt|tr | |}t|tr | |}|_d_|_|_|ptt'j}|_jpdd|dt'jjdd}|_d_ddlm}||d_d_d_d_d_d_ d_!d_"d_#d_$d_%j&'d_(tj)dd}tU|r |d_+j,dkr; -r.dn#t $rYnwxYwj/r0d_/tcj2_3| |}ti|dkr |dddzn|}|5d d!}t6d"jpd#j7j8pd$j9pd$tipg|rtung^td%d}||n|}t|tvr!|<d&|kr|}||d&<n!d'|d(}t|tvrd_=r.j>?s@rVjAdkrKtd)D}|dkr,|_AjCdkrjDdkr|jCz_D^E|ti^d*z }|_xjAd*z c_Ad_Ftd+d}||Gtd,d} | | G||n|}!d}"jCdkr9d-jHvr0j)r)xjDd*z c_DjDjCkr d}"d_Dtd.d}#|#0 dd/lImJ}$|$|!}%|%r |#|%n#t $rYnwxYwjKsC| |}&Ld0|&dd1ti|&d1krdndd2jM ||jM}'td3d}( |(Nn.|(5Ndddn #1swxYwYn4#t $r'tOd4jpd#dYnwxYwt|tvr|<d5rd_=n6#t|tvr|<d5rd_=wxYwtd6d})jPr|)dkr^rtjQtd7tjQz }*|*|)kr_jR}+t^|'pdjTpd8},t|+jV|+jWz}-t|+d9d:}.tjP|)|*|,|-t|.;rt6dtj\t|*|,?|,t|*j7@}/|/r.|/^}0]^||,|A\^}'^|0ur(t^t^|}|_d}1d}2d_`d_ajPrt^jRjcjRjdjRjVrt^|'pdjTpd8}3jR}+t|+dBd}4tU|4r;|4}5t|5trt|5ts|5_at|+dCdD}6|6|3}7td ddEko3ttdFdGpdGedHv}8|7s|+jf}9|9dkr |3|9kr|3|+_ft|+d9dI}:d};d}<|7r.t6dJ|3d=|+jVd=|+jgd=n|:rlt6dKt|:<dLdMjpd#|3|+jVkr|:<dLdM}=dN|=dO}tU|>r% |>|3d*}z%build_turn_context.. s?  quuV}}/F/FA/F/F/F/F  r"rK_stream_context_scrubber_stream_think_scrubbermemoryreaction_callback)detect_reactionu💬 Starting conversation: '<'_session_persist_lockz5Turn-start session row creation failed for session=%s _db_persisted&compression_idle_compact_after_seconds_last_activity_ts) system_prompttools'get_active_compression_failure_cooldowncdSNrXrXr"r z$build_turn_context..sPTr"rjzDIdle compaction: %ss idle >= %ss, ~%s tokens > %s floor (session %s),idle) idle_secondsrf)phasedefault_message approx_tokensrr)rr~!snapshot_preflight_display_tokens$should_defer_preflight_to_real_usagecdS)NFrX)_tokenss r rz$build_turn_context..sEr"codex_app_server codex_app_server_auto_compactionnative>offrcdSrrXrXr"r rz$build_turn_context..sDr"zpSkipping preflight compression: rough estimate ~%s >= %s, but last real provider prompt was %s after compressionz`Skipping preflight compression: same-session cooldown active (~%s seconds remaining, session %s)remaining_secondsgz cooldown:z.0fzsSkipping Hermes preflight compression for codex app-server (mode=%s); Hermes will not start thread compaction here.should_compress_info_clear_context_overflow_warnzDPreflight compression: ~%s tokens >= %s threshold (model %s, ctx %s) preflight)rf threshold)rrrrZcontext_lengthrmax_compression_attemptszRPreflight compression deferred: compression lock held by another path (session %s)zgPreflight compression made insufficient progress: ~%s -> ~%s request tokens; skipping additional passesshould_compress_preflightzQshould_compress_preflight raised %s; skipping engine-driven preflight maintenancez_Engine-driven preflight maintenance: %s requested compress() at ~%s tokens (below %s threshold)namerZ) invoke_hook pre_llm_callplatform_user_id) session_idr~rsrGrp is_first_turnrr sender_id)get_spill_configspill_if_oversizedcontextz plugin hook)rsourceconfigzhook context spill failed: %srzpre_llm_call hook failed: %sr'r&_last_compaction_in_place _session_dbz>in-place compaction api_content backfill failed for session=%srr/c\dSr)_ensure_db_session_persist_session)r6rprEsr _ensure_and_persistz/build_turn_context.._ensure_and_persists2   """ x)=>>>>>r"z:Early turn-start session persistence failed for session=%s) rGrnrErprqrrrsrtrurrrv)rr/)rr:_restore_primary_runtimeagent.auxiliary_clientrr<sysmodulestools.mcp_toolrrloggerdebugrr_stream_callback_persist_user_message_idx_persist_user_message_override_persist_user_message_timestampuuiduuid4_current_task_idhex_current_turn_id_current_api_request_idagent.agent_runtime_helpersr_invalid_tool_retries_invalid_json_retries_empty_content_retries_incomplete_scratchpad_retries_codex_incomplete_retries_thinking_prefill_retries_post_tool_empty_retried_last_content_with_tools$_last_content_tools_all_housekeeping_mute_post_response_unicode_sanitization_passes_tool_guardrailsreset_for_turn_tool_guardrail_halt_decision _memory_storecallable_vision_supportedr_cleanup_dead_connections _emit_status_compression_warning_replay_compression_warningrmax_iterationsiteration_budgetrMreplaceinforrrrCrNr+r _todo_store has_items_hydrate_todo_store_user_turn_countsum_memory_nudge_interval_turns_since_memoryr_is_user_initiated_turnresetvalid_tool_namesagent.reactionsrr _safe_print_cached_system_promptrwarningcompression_enabledtimecontext_compressorrrrHrZsummary_target_ratiorkr?rr format_compress_contextr rQ _turn_received_provider_response _turn_preflight_display_snapshotrar^r_lowerlast_prompt_tokenslast_real_prompt_tokensshould_compressrr maxrLr rYr\_warn_context_overflow_blockedrBrwhermes_cli.pluginsrtools.hook_output_spillrrstriprr>rD_turn_failed_file_mutationsset_turn_file_mutation_paths_verification_stop_nudges_pre_verify_nudges threadingcurrent_threadident_execution_thread_id_set_interrupt_interrupt_requested _interrupt_thread_signal_pending_interrupt_message_memory_manager on_turn_start prefetch_allr!set_latest_user_api_contentrm)_r6rGr}rpr~stream_callbackrrrestore_or_build_system_promptinstall_safe_stdiosanitize_surrogatessummarize_user_message_for_logset_session_contextset_current_write_originrar|r_sysrrrrrsr _reset_consol _preview_text _msg_previewpending_cli_messageexpected_persist_contentuser_msgprior_user_turnsrtscrubberthink_scrubberrnrurrkind_print_previewrq persist_lock _idle_after _idle_gap _compressor _idle_tokens _idle_floor_idle_cooldown _idle_status _idle_input_preflight_compressed_preflight_compression_blocked_preflight_tokens _snapshot_fn _snapshot_val_defer_preflight_preflight_deferred_codex_native_auto_last_compression_cooldown_should_compress_now_compress_block_reason_cooldown_secs_info _clear_warn_preflight_status_max_preflight_passes_pass _orig_len _orig_tokens_preflight_input_engine_preflight_wants_engine_preflight_preflight_exc _engine_inputr _invoke_hook _pre_results _ctx_parts _spill_cfg_spill_if_oversized_spill_config_cachedr_piece _spill_excexc_gateway_notes_gw_turn_content _turn_msgr_query_turn_user_msg _api_content_dbrrEs_` ` @r build_turn_contextrHs4()))WU,BDTUUVVV ""$$$  ;;;;;; E:r * * 0b E7B ' ' -2&u.BBGGM2UJ339rE9b117RUJ339re["55;           Nu1599 D    4<//\\\\\\\\++--D++EdCCCC NNN = MMMMMN,$$9**<88 &,,I223GHH-E&*E#+?E(,BE)43tz||#4#4.E!.Y[[1B[[TZ\\EUVXWXVXEY[[G$E$&E!<;;;;;OE7####$E"#E#$E +,E(&'E#&'E#%*E"%)E"16E. %E)*E& ))+++*.E'E/1OQUVVM  "E ~--- ..00 """     D  !* ))+++%)"-U-ABBE32<@@M36}3E3E3J3JM#2#&..P]L''c22L KKZ "FEK1L9 #)S)=)C%D%D .BIt()))rH"%)DdKK 4 @l &-- 3  # #I . .2J J J&+"|<< )4 0 0 3.2E +8E$5$?$?$A$A8 !!"6777\ 6! ; ;  +      a  %5E "+a//E4MQR4R4R,RSSS 65"94@@L3    $ $ & & & & + +((*** + + + + + + + + + + + + + + +     C   &       -t44 38K8O8OP_8`8` 3.2E +-t44 38K8O8OP_8`8` 3.2E + 2 2 2 2%!I1MMK  CL[1___IKK'%1Ddikk"R"RR  # #2K828bk)TL,{/OOKWF N$1#.!*#( $^ 4 4 2 L # NN#''"&&$. C $C$J%(^^L%%%#/!$Y+     5&&|444& 161H1HnL-2I22..;..+Qx)=,,( -K ,--)7LE3 "%*"-2E*-1E*  h6%C  0  /  1 &&h6 : .4"+%    .  #>?P#Q#Q ' 6 -CTJJE??66167H1I1I!1L..$66615...6 A 6$( ! "%)GNNK $$  KKV$((/33 -11    !D! E L,):!!!0!,!=*9k ! ! ! ! 6""#4555%(3wu&@!DDIJJ%% !455> > MM 0 #+ 161H1Hn>1KK<(2F E %B"6"<"+-%%%! 2s8}}l& /4++++++4' 6 DKk1B1B1KLL(,,{,>BBFF !) 161H1Hn05E- @!? l! ! +@'1<BBBBBB#| '%.!%h#$8999+UJ55;eZ66<"    !#  (        $.:<<  ( ( ("& #'  ( & &AF!T"" quuY'7'7 Qy\**As##   ".P00#(#3,3 FF !PPPNN#BJOOOOOOOOP   f % % % %  :"(++j"9"9  <<<5s;;;;;;;;<8>>N)9999CMM999998$9:DAA: * + / / : : :  & - -  ./? P P P P'$#f,~==# )+E%&)eeE#&'E# E"+!9!;!;!AEBDDu9::: !7 D%"<===16..#' 16.  1;[Z8, [8Z<<[?Z<[\*.[52\*4[55\**3] n n+*n+y!! z+z  zA$ACAC$!AC#$$A8ACAAA2A1ACA2 AB!A<ABBACBAB!B!0ACC ADCAC<C<ADI 9AJJ AJJAJJ!5AKK AK$K#AK$O/AO0O0.AP!P AP!P-AQP< AQQ AQQAQQAQQAQQAQQASQ.ARR ASRARRASS3AS8)rrrrrrrr)r#r$rr)r.r$rr/)r.r2rr)r6rrr)rrr=rrr?)rErFrGrrrH) rRrHrSrHrTrHrUrHrr?)rTrHrUrHrZrHrr?) rEr]r^rHr_rHrZrHrr?)rbr?rcrHrdrerfrHrgrHrhr?rr?r)rGrr}rrpror~rrrrrr|r?rrm)-rz __future__rloggingr.rr dataclassesrtypingrrrrr agent.conversation_compressionr r r r agent.context_engineragent.iteration_budgetragent.memory_managerragent.model_metadatarr getLoggerrwrr!r-r1r5r>rDrQrYr\rarkrmrrXr"r rs0#""""" !!!!!!55555555555555 EDDDDD222222;;;;;;  8 $ $ 6 6 6 6F. ! ! ! !----3333((4????."HHHH8!!!!D 0000000 0H/3I"#IIIIIIIIr"