[j5%BdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z ddl mZddlmZmZmZmZddlmZddlmZejeZd Zd Zd Zd Zd8dZd9d:dZ d9d:dZ!dddddd;dZ"eddd2Z%edddd3d?d7Z&dS)@uOut-of-loop shutdown backstop + event-loop liveness heartbeat (#66892). When the asyncio loop freezes mid-drain, every asyncio-based recovery path is structurally unable to fire: the drain deadline, status rewrites, and forensics all need the same loop that is stuck. launchd/systemd KeepAlive only restarts a *dead* process, so a wedged-but-alive gateway sits as a zombie until manual SIGKILL. This module provides: 1. A plain OS-thread shutdown watchdog armed at ``stop()``. If shutdown has not completed within ``restart_drain_timeout + grace``, it dumps all-thread stacks via ``faulthandler`` plus a metadata snapshot, then ``os._exit`` so the service manager can revive the process. 2. An event-loop heartbeat file at ``/state/gateway.heartbeat`` so external supervision can distinguish "process alive" from "loop frozen" (``gateway_state.json`` alone can't — it only rewrites on transitions/turns). ) annotationsN)datetimetimezone)Path)AnyCallableDictOptional)get_hermes_home)atomic_json_writegN@g>@)statezgateway.heartbeat)logszgateway-shutdown-watchdog.logreturnrctjdd}|rt |St S)zHHERMES_HOME for process-level identity files (ignore profile overrides). HERMES_HOME)osenvirongetstriprr )vals ;/home/ice/.hermes/hermes-agent/gateway/shutdown_watchdog.py_process_hermes_homer/sC *.. + + 1 1 3 3C Cyy   homeOptional[Path]cD||n t}|jtS)z1Return ``/state/gateway.heartbeat``.)rjoinpath_HEARTBEAT_RELATIVErbases rget_loop_heartbeat_pathr"7s'#44)=)?)?D 4=- ..rcD||n t}|jtS)zBReturn the faulthandler / metadata dump path for a fired watchdog.)rr_WATCHDOG_DUMP_RELATIVEr s rget_shutdown_watchdog_dump_pathr%=s'#44)=)?)?D 4=1 22r)pid start_timerextrar& Optional[int]r'Optional[float]r(Optional[Dict[str, Any]]ct|}t||ntjt jt jtj d}|t||d<|r| | t||dn,#t$rtddYnwxYw|S)uAtomically rewrite the loop-liveness heartbeat file. ``start_time`` is the gateway process start (``time.time()`` epoch seconds) so supervisors can detect PID reuse. Best-effort — never raises. N)r& updated_at monotonicr')indentz&Failed to write gateway loop heartbeatTexc_info)r"intrgetpidrnowrutc isoformattimer.floatupdater Exceptionloggerdebug)r&r'rr(pathpayloads rwrite_loop_heartbeatr?Cs #4 ( (D#/33ry{{;;l8<00::<<^%%G  %j 1 1  uN$55555 NNN = MMMMMN Ks#B66&CC)grace_s drain_timeoutr8r@c tt|d}n#ttf$rd}YnwxYw tt|d}n#ttf$r t}YnwxYw||zS)z=Return the wall-clock leash for the shutdown watchdog thread.)maxr8 TypeError ValueError!DEFAULT_SHUTDOWN_WATCHDOG_GRACE_S)rAr@draingraces rresolve_shutdown_watchdog_delayrJ`s E-((#.. z "2E'NNC(( z "22212 5=s 66AA32A3 dump_pathdelay_ssnapshotNonec |jddn#t$rYdSwxYwdtj|t jtj |pid} t|dd5}| tj |t d z| d | t!j|d n%#t$$r| d YnwxYw| d|dddn #1swxYwYn#t$$rYnwxYw t&j d|ddtjdt&jt!jddS#t$$rYdSwxYw)z:Best-effort faulthandler + metadata dump before hard-exit.T)parentsexist_okNshutdown_watchdog_fired)eventr&rLfired_atrMazutf-8)encoding)default z(--- faulthandler dump (all threads) --- )file all_threadsz%(faulthandler.dump_traceback failed) z--- end dump --- z&Gateway shutdown watchdog fired after z.0fzs (pid=z); dumping all thread stacks. )rZ)parentmkdirOSErrorrr3rr4rr5r6openwritejsondumpsstrflush faulthandlerdump_tracebackr:sysstderr)rKrLrMheaderfhs r_write_watchdog_dumprjqsvtd;;;; +y{{L..88::N F  )S7 3 3 3 r HHTZ444t; < < < HH@ A A A HHJJJ C+FFFFF C C CABBBBB C HH) * * * HHJJJ                        AW D A AIKK A A A    #555555      sz --:E% AE(C?>E?D!E D!!,E E%EE% E!E%% E21E26A+G## G10G1zgateway-shutdown-watchdog) done_event snapshot_fn exit_coderKnamerlOptional[threading.Event]rm&Optional[Callable[[], Dict[str, Any]]]rnr2rorbthreading.Eventc||ntj tt|dn#tt f$r t YnwxYwdkrSd fd } tj|d|n,#t$rt d d YnwxYwS) aArm a daemon-thread hard-exit backstop for a wedged shutdown path. If ``done_event`` is set before ``delay_s`` elapses, the thread exits quietly (normal / progressing shutdown completed). Otherwise it dumps diagnostics and calls ``os._exit(exit_code)``. Never raises. Returns the ``done_event`` (creating one when omitted) so the caller can disarm on successful completion. NrCrrrNctj z}tj|krS|tjz } t|drdStj|kS rdSd} 4  }n(#t $r}dt |i}Yd}~nd}~wwxYw  n t}t| | t d |n#t $rYnwxYwtj tj fD]'} |#t $rY$wxYw ddlm}m}||n#t $rYnwxYw ddlm}|dn#t $rYnwxYwt)j dS) N?)timeoutsnapshot_error)rLrMuhShutdown watchdog fired after %.0fs — forcing process exit (asyncio drain path appears wedged; see %s)r)remove_pid_filerelease_gateway_runtime_lock)drain_log_queue)r7r.waitminis_setr:reprr%rjr;criticalrfstdoutrgrcgateway.statusrxryhermes_loggingrzr_exit)deadline remainingrMexctargetstreamrxryrzdelaydonerKrnrms r _watchdogz(arm_shutdown_watchdog.._watchdogsj>##e+n)) 4>#3#33IyyY!4!4y55 n)) ;;==  F-1  " 9&;== 9 9 9,d3ii8 9(39X9Z9ZVUXFFFF  OO>          D z3:.  F         T T T T T T T T O    ( ( * * * *    D   6 6 6 6 6 6 OC ( ( ( ( (    D  s` B(( C 2CC 5D DD=E EE#F F  F F$$ F10F1T)rdaemonrozFailed to arm shutdown watchdogr0)rrN) threadingEventrDr8rErFrGThreadstartr:r;r<) rLrlrmrnrKrorrrs ``` @@rarm_shutdown_watchdogrs$$/::Y_5F5FD2E'NNC(( z "22212 zz 3333333333jG $TBBBHHJJJJ GGG 6 FFFFFG Ks!<AA-)B&C?C) interval_sr'rshould_continuerrOptional[Callable[[], bool]]c4K tt|d}n#ttf$r t}YnwxYwt || | |sdSt j|d{V| |sdSt ||H)uRewrite the loop heartbeat file on a cadence until cancelled / gated off. Runs as an asyncio task on the gateway loop — if the loop freezes, this task stops and the file mtime/updated_at goes stale for external monitors. ru)r'rTN)rDr8rErFDEFAULT_HEARTBEAT_INTERVAL_Sr?asynciosleep)rr'rrintervals rloop_heartbeat_foreverrs0uZ((#.. z "000/0 JT::::?  &/@/@ & FmH%%%%%%%%%  &/@/@ & F >>>> ?s "==)rr)N)rrrr) r&r)r'r*rrr(r+rr)rAr8r@r8rr8)rKrrLr8rMr+rrN)rLr8rlrprmrqrnr2rKrrorbrrr) rr8r'r*rrrrrrN)'__doc__ __future__rrrdr`loggingrrfrr7rrpathlibrtypingrrr r hermes_constantsr utilsr getLogger__name__r;rGrrr$rr"r%r?rJrjrrrrrs&&#"""""   ''''''''000000000000,,,,,,######  8 $ $%)!#4C///// 33333"&&* @7"+ + + + b-1:> $+TTTTTTr5"&48 ????????r