'jQUddlmZddlmZmZmZmZmZmZm Z m Z ddl m Z m Z ddlmZmZmZmZddlmZmZmZmZ ddlZn#e$rddlZYnwxYw ddlmZn#e$r ddlmZYnwxYw d Zd Zd Zd Z d Z!dddddZ"dddddZ#e e e$dfee$e$fZ%ee&d<eGddZ'eGddZ(GddZ)dS)) dataclass)AnyDictIterableListLiteralOptionalTupleUnion)BayesianNetworkget_possible_values)get_header_networkget_headers_orderget_browser_helper_fileget_input_network) get_browserget_user_agentpascalize_headerstuplifyN) TypeAlias)chromefirefoxsafariedge)windowsmacoslinuxandroidios)desktopmobile)12z*MISSING_VALUE*z same-sitenavigatez?1document)zSec-Fetch-ModezSec-Fetch-DestzSec-Fetch-SitezSec-Fetch-User)zsec-fetch-modezsec-fetch-destzsec-fetch-sitezsec-fetch-user. ListOrStringcveZdZUdZeed<dZeeed<dZ eeed<dZ e eefed<dZ dS) BrowserzORepresents a browser specification with name, min/max version, and HTTP versionnameN min_version max_versionr$ http_versionc<t|jtrt|j|_t|jtrJt|jtr2|j|jkr$t d|jd|jddSdSdS)Nz Browser min version constraint (z) cannot exceed max version ()) isinstancer-intstrr+r, ValueErrorselfs b/home/ice/.hermes/hermes-agent/venv/lib/python3.11/site-packages/browserforge/headers/generator.py __post_init__zBrowser.__post_init__7s d' - - 7 #D$5 6 6D  t' - - 4+S11  4#333u43Cuubfbruuu      43) __name__ __module__ __qualname____doc__r2__annotations__r+r r1r,r-r r7r8r6r)r).stYY III!%K#%%%!%K#%%%$'L%S/'''     r8r)cneZdZUdZeeed<eedfed<eed<eed<e dZ dS) HttpBrowserObjectzWRepresents an HTTP browser object with name, version, complete string, and HTTP versionr*.versioncomplete_stringr-c|jdkS)Nr$)r-r4s r6is_http2zHttpBrowserObject.is_http2Os C''r8N) r9r:r;r<r r2r=r r1propertyrDr>r8r6r@r@Fsqaa 3- 38_ ((X(((r8r@cjeZdZUdZdZeedfed<ee Z ee Z e eedddfdeeeefd ed ed ed ed def dZddddddddddeeeeefd eed eed eed eed deedeedeeeeffdZ d/deeeefdeeededeeeffdZdeeeffdZdeeeefd edeefdZdeedeefdZdeeefdeeeffdZ deeefdeeeeffdZ!de"defd Z#d!edefd"Z$deeeeffd#Z%dee"fd$Z&d%eeeefd&eeefd'eeefdeeeeffd(Z'e(d)ed&eeefd'eeefdefd*Z)e(d)ed&eeefd'eeefd+edef d,Z*d-ede"fd.Z+dS)0HeaderGeneratorz4Generates HTTP headers based on a set of constraints)localesdevicesoperatingSystemsbrowsers.relaxation_orderzen-USFbrowserosdevicelocaler-)rrMstrictc4t|}|t||t|t|t|||d|_||_||_dS)a Initializes the HeaderGenerator with the given options. Parameters: browser (Union[ListOrString, Iterable[Browser]], optional): Browser(s) or Browser object(s). os (ListOrString, optional): Operating system(s) to generate headers for. device (ListOrString, optional): Device(s) to generate the headers for. locale (ListOrString, optional): List of at most 10 languages for the Accept-Language header. Default is 'en-US'. http_version (Literal[1, 2], optional): Http version to be used to generate headers. Defaults to 2. strict (bool, optional): Throws an error if it cannot generate headers based on the input. Defaults to False. )rKrOrIrHr-rRN)r2_prepare_browsers_configroptions_load_unique_browsersunique_browsers_load_headers_order headers_order)r5rNrOrPrQr-rRhttp_vers r6__init__zHeaderGenerator.__init__]s(L))55gg6F6FQQ"++vv$     $99;;!5577r8N)rNrOrPrQr- user_agentrRrequest_dependent_headersr\r]c bt|t|t|t||rt|nd|t||d} |jdid| D} | dp |jddkrt | S| S)a Generates headers using the default options and their possible overrides. Parameters: browser (Optional[Iterable[Union[str, Browser]]], optional): Browser(s) to generate the headers for. os (Optional[ListOrString], optional): Operating system(s) to generate the headers for. device (Optional[ListOrString], optional): Device(s) to generate the headers for. locale (Optional[ListOrString], optional): Language(s) to include in the Accept-Language header. http_version (Optional[Literal[1, 2]], optional): HTTP version to be used to generate headers. user_agent (Optional[ListOrString], optional): User-Agent(s) to use. request_dependent_headers (Optional[Dict[str, str]], optional): Known values of request-dependent headers. strict (Optional[bool], optional): If true, throws an error if it cannot generate headers based on the input. N)rKrOrIrHr-rRr\r]ci|] \}}||| SNr>.0kvs r6 z,HeaderGenerator.generate..sAAA11=q!===r8r-r$r>)rr2 _get_headersitemsrUr) r5rNrOrPrQr-r\rRr]rU generateds r6generatezHeaderGenerator.generates6 (("++vv1=GC ---4!*--)B   %6D$5% % AA AAA% %  N # Ct|N'C K K$Y// /r8rUreturnc |i}dvsdvr,d|jdkr|i|j}||}|rZt|tt fst |}t |jd|it |jd|i}}nii}}||||}|j |} | s|ddkr5|j ||fiddi} | t| Stfdt|jDd } |d s| d krt#d i} | |j| =|j ||fi| S|j| } || d }|jrd nd}||d| |<||r<|jr| t0n| t2d| D} | i| |S)a Generates HTTP headers based on the given constraints. Parameters: request_dependent_headers (Dict[str, str], optional): Dictionary of request-dependent headers. user_agent (Iterable[str], optional): User-Agent value(s). **options (Any): Additional options for header generation. Returns: Dict[str, str]: Dictionary of generated HTTP headers. NrKr-z User-Agentz user-agentr#r$c3*K|] \}}|v |VdSr`r>)rbikeyrUs r6 z/HeaderGenerator._get_headers..s+TTvq#SG^^^^^^TTr8rRzoNo headers based on this input can be generated. Please relax or change some of the requirements you specified. *BROWSER_HTTPzaccept-languagezAccept-LanguagerHci|]F\}}|dkr|dks#|ds|tkC||GS) connectionclose*)lower startswithMISSING_VALUE_DATASET_TOKENras r6rez0HeaderGenerator._get_headers..sf    1 \))LL<<$$!333 q 433r8)rU_update_http_version_get_possible_attribute_valuesr0tuplelistr header_generator_network_prepare_constraintsinput_generator_network(generate_consistent_sample_when_possiblerf order_headersrnext enumeraterLr3generate_sample_prepare_http_browser_objectrD_get_accept_language_header_should_add_sec_fetchupdateHTTP2_SEC_FETCH_ATTRIBUTESHTTP1_SEC_FETCH_ATTRIBUTESrg)r5r]r\rUheader_optionspossible_attribute_values http1_values http2_values constraints input_sampleheaders2relaxation_indexrelaxed_optionsgenerated_samplegenerated_http_and_browseraccept_language_field_names ` r6rfzHeaderGenerator._get_headerssE" % ,(* %  g % %'4<+GGG  % %g . . .4DL4G4$($G$G$W$W!  0j5$-88 /":.. #D$ALR\C]^^#D$ALR\C]^^'LL *,R,L// %|\  3\\    _n-44,4,-z=DSV))*;H*E*EFFF#TTTT4+@!A!ATTTVX   h' +;r+A+A F*kO 56F GH$4$%> ^^o^^ ^8HHVV%)%F%F _ -& & " "z8HeaderGenerator._update_http_version..$sw # # #"%11 HG"Z$)$5$)$5%1 !e,GGG # # #r8N)rUrT)r5rUr-s @r6ryz$HeaderGenerator._update_http_versions W $ $">2LL<7L  "&"?"? @SUa"b"bGJ    # # # #"\*5 # # #GJ   r8rKc fd|DS)aU Prepares the browser configuration based on the given browsers and HTTP version. Parameters: browsers (Iterable[Union[str, Browser]]): Supported browsers or Browser objects. http_version (str): HTTP version ('1' or '2'). Returns: List[Browser]: List of Browser objects. c`g|]*}t|trt|n|+S)r)r0r2r))rbrNr-s r6rz.?sO   gs++W<@@@@    r8r>)r5rKr-s `r6rTz(HeaderGenerator._prepare_browsers_config2s0    $     r8c fd|DS)a Retrieves the browser HTTP options based on the given browser specifications. Parameters: browsers (Iterable[Browser]): Iterable of Browser objects. Returns: List[str]: List of browser HTTP options. cg|]t}jD]j}|j|jkrX|jr|j|jdkr;|jr|j|jdkr|jr|j|jkc|jkuS)r)rWr*r+rAr,r-rB)rbrNbrowser_optionr5s r6rz=HeaderGenerator._get_browser_http_options..Rs   "&"6  |~222(3,3,?>CYZ[C\,\,\(-],3,?>CYZ[C\,\,\)-]-4-A^E`-`-`  * .a-`-`-`r8r>)r5rKs` r6_get_browser_http_optionsz)HeaderGenerator._get_browser_http_optionsHs.    #    r8headersct}|tdt|}|td|j|}|rfd|DnS)z Orders the headers based on the browser-specific header order. Parameters: headers (Dict[str, str]): Dictionary of headers. Returns: Dict[str, str]: Ordered dictionary of headers. Nz/Failed to find User-Agent in generated responsez$Failed to find browser in User-Agentc*i|]}|v||Sr>r>)rbrnrs r6rez1HeaderGenerator.order_headers..qs$ I I I3#..S'#,...r8)rr3rrYget)r5rr\ browser_name header_orders ` r6rzHeaderGenerator.order_headers\s$G,,  NOO O":..  CDD D)--l;;   I I I I, I I I I r8rc||dd|dd}||}||dtd}d|vr |d|d<|S) a Retrieves the possible attribute values based on the given header options. Parameters: header_options (Dict[str, Any]): Dictionary of header options. Returns: Dict[str, List[str]]: Dictionary of possible attribute values. rKr>r-r$rO)rqz*OPERATING_SYSTEMrIz*DEVICE)rTrrSUPPORTED_OPERATING_SYSTEMS)r5rrKbrowser_http_optionsrs r6rzz.HeaderGenerator._get_possible_attribute_valuesvs00   z2 . .   ~s 3 3   $==hGG2!/!3!3D:U!V!V% % !  & &3A)3L %i 0((r8c|jdkr|jddkrdS|jdkr|jddkrdS|jdkr|jddkrdSd S) a  Determines whether Sec-Fetch headers should be added based on the user agent. Parameters: browser (HttpBrowserObject): Browser object. Returns: bool: True if Sec-Fetch headers should be added, False otherwise. rrLTrZrOF)r*rA)r5rNs r6rz%HeaderGenerator._should_add_sec_fetchsr <8 # #(:b(@(@4 <9 $ $);r)A)A4 <6 ! !goa&8B&>&>4ur8rHcZddt|DS)z Generates the Accept-Language header based on the given locales. Parameters: locales (ListOrString): Locale(s). Returns: str: Accept-Language header string. z, c38K|]\}}|dd|dzz dVdS)z;q=g?g?z.1fNr>)rbindexrQs r6roz>HeaderGenerator._get_accept_language_header..sQ  6CeVv 1 1# + 1 1 1      r8)joinr)r5rHs r6rz+HeaderGenerator._get_accept_language_headers?yy  GPQXGYGY      r8cftjtS)z Loads the headers order from the headers-order.json file. Returns: Dict[str, List[str]]: Dictionary of headers order for each browser. )jsonloadsr read_bytesr4s r6rXz#HeaderGenerator._load_headers_orders'z+--88::;;;r8ctjt}fd|DS)z Loads the unique browsers from the browser-helper-file.json file. Returns: List[HttpBrowserObject]: List of HttpBrowserObject instances. cPg|]"}|tk |#Sr>)rxr)rb browser_strr5s r6rz9HeaderGenerator._load_unique_browsers..s=   999  - -k : :999r8)rrrr)r5unique_browser_stringss` r6rVz%HeaderGenerator._load_unique_browserssS"&,C,E,E,P,P,R,R!S!S    5    r8rrrcLfd|DS)a Prepares the constraints for generating consistent samples. Parameters: possible_attribute_values (Dict[str, List[str]]): Dictionary of possible attribute values. http1_values (Dict[str, Any]): Dictionary of HTTP/1 values. http2_values (Dict[str, Any]): Dictionary of HTTP/2 values. Returns: Dict[str, Iterable[str]]: Dictionary of constraints for each attribute. c `i|])\}ttfd|*S)cndkr|n|S)Nrq)filter_browser_httpfilter_other_values)xrrrnr5s r6zAHeaderGenerator._prepare_constraints...sE/1100L,OOO!55a|UXYYr8)r{filter)rbvaluesrnrrr5s @r6rez8HeaderGenerator._prepare_constraints..sq    V         r8)rg)r5rrrs` ``r6r~z$HeaderGenerator._prepare_constraintssH"        9>>@@    r8valuec|d\}}|dkr| p||ddvn| p||ddvS)a Filters the browser HTTP value based on the HTTP/1 and HTTP/2 values. Parameters: value (str): Browser HTTP value. http1_values (Dict[str, Any]): Dictionary of HTTP/1 values. http2_values (Dict[str, Any]): Dictionary of HTTP/2 values. Returns: bool: True if the value should be included, False otherwise. |r#z*BROWSERr>)splitr)rrrrr-s r6rz#HeaderGenerator.filter_browser_httpsp&+[[%5%5" ls""  Q1A1A*b1Q1Q!Q""Vll6F6FzSU6V6V&V r8rncn|s|r0|||dvp|||dvSdS)a Filters the other attribute values based on the HTTP/1 and HTTP/2 values. Parameters: value (str): Attribute value. http1_values (Dict[str, Any]): Dictionary of HTTP/1 values. http2_values (Dict[str, Any]): Dictionary of HTTP/2 values. key (str): Attribute key. Returns: bool: True if the value should be included, False otherwise. r>T)r)rrrrns r6rz#HeaderGenerator.filter_other_valuessQ  \< \L,,S"555[,BRBRSVXZB[B[9[ [tr8http_browser_stringc*|d\}}|tkrtddtdS|d\}}|d}td|D}t||||S) z Extracts structured information about a browser and HTTP version from a string. Parameters: http_browser_string (str): HTTP browser string. Returns: HttpBrowserObject: HttpBrowserObject instance. rNr>)r*rArBr-/.c34K|]}t|VdSr`)r1)rbparts r6roz?HeaderGenerator._prepare_http_browser_object..#s(<r8r6rGrGTs>>(^eCHo^^^.o.?.?.A.ABB./A/A/C/CDD;M60&&' 8 8|Xg%667 8  8 8  8 dm 8 8 8 8 8J<@%))-)-04-1!%>B)))(5g#678) \ " ) & ) & )wt}-)\*))$,DcN#;))))Z?C.2eUeU#+DcN#;eUXc]+eU eU c3h eUeUeUeUNc3hB  sG|!45 EH g    , (72C S     ( T#s(^ S#X    4)"38n) c49n ))))6->4$  <  C       "&sCx. @DS#X    \ *"&sCx.@DS#XUX \&  HY      r8rG)* dataclassesrtypingrrrrrr r r browserforge.bayesian_networkr r apify_fingerprint_datapointsrrrrutilsrrrrorjsonr ImportErrorrtyping_extensionsrrrSUPPORTED_HTTP_VERSIONSrxrrr2r'r=r)r@rGr>r8r6rs!!!!!!!MMMMMMMMMMMMMMMMMMMMNNNNNNNNKJJJJJJJJJJJKKKKK,       ,,,++++++++,<M)$/!  "    c3hcC ?@ i@@@  .  ( ( ( ( ( ( (  (U U U U U U U U U U s#A A AA A'&A'