Hij08dZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z mZe r dd lmZd d lmZiZeed GddZGddZedZddZ ddZdddZej4ZddZddddZy) z Contains the logic for automatic additional output capture with our forward decorators. This mostly describe the hooks used and the logic to make capture thread/context safe. ) annotationsN) ContextVar) dataclasswraps) TYPE_CHECKING)is_torchdynamo_compilingrequires)nnPreTrainedModel)torch)backendscFeZdZUdZded<dZded<dZded <dZded <y) OutputRecordera Configuration for recording outputs from a model via hooks. Attributes: target_class (Type): The class (e.g., nn.Module) to which the hook will be attached. index (Optional[int]): If the output is a tuple/list, optionally record only at a specific index. layer_name (Optional[str]): Name of the submodule to target (if needed), e.g., "transformer.layer.3.attn". class_name (Optional[str]): Name of the class to which the hook will be attached. Could be the suffix of class name in some cases. ztype[nn.Module] target_classrintindexN str | None layer_name class_name)__name__ __module__ __qualname____doc____annotations__rrre/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/transformers/utils/output_capturing.pyrr's,"!E3N!J !!J !r rc(eZdZdZdZdZdZdZy)CompileableContextVara Convenience wrapper around a ContextVar for usage with `torch.compile`. This behaves exactly as a `ContextVar`, except when compilation is triggered in which case it behaves as a simple global variable. This is useful as `torch.compile` cannot trace the `get` method of `ContextVar`. This however means that the access to the underlying variable is not thread-safe when compilation is triggered. cDt|d|_d|_d|_y)N)defaultF)r context_var global_var compiling)selfnames r!__init__zCompileableContextVar.__init__Bs%dD9r cf|jr |jS|jjSN)r(r'r&get)r)s r!r.zCompileableContextVar.getGs( >>?? "##'') )r cjtr||_d|_y|jj |S)NT)r r'r(r&set)r)values r!r0zCompileableContextVar.setNs0 # %#DO!DN##''. .r ct|js|d|_d|_y|jj|y)NF)r(r'r&reset)r)tokens r!r3zCompileableContextVar.resetVs/ >>U]"DO"DN    " "5 )r N)rrrrr+r.r0r3rr r!r#r#:s */*r r#output_collectorc6fd}|j|y)zaInstall the forward hook needed to capture the output described by `key` and `index` in `module`.c6tj}||jvrydk(r(t|dk(r|j |dt |t s|j |y||j |yy)N hidden_statesr)_active_collectorr.keyslenappend isinstancetuple)moduleargsoutputcollected_outputsrkeys r!output_capturing_hookz.output_capturing_hookes-113  $3D3I3I3K(K  / !c*;C*@&AQ&F c " ) )$q' 2&%( c " ) )& 1 E] & c " ) )&- 8'r N)register_forward_hook)r?rCrrDs `` r!install_output_capturing_hookrFbs 9   !67r cddlm}|jD]6\}}t||st ||d||%t ||d|8|D]\}}|j t||j s)|j5|j|jsQ|j|j|vrlt|||jy)a Recursively install all output capturing hooks on all submodules of `parent_module`. Note that we need to use this recursive approach instead of simply iterating over all modules, because we want to respect the `capture_tasks` of all individual submodels (`PreTrainedModel` instances) in the graph. That is, once we reach a submodel in the graph, its children should use this submodel's `capture_tasks`, but other parts of the graph should not. r r.)prefixN) modeling_utilsrnamed_childrenr=recursively_install_hooks"install_all_output_capturing_hooksrrendswithrrFr) parent_module module_name capture_tasksrr*r?rCspecss r!rLrLvs1&446W f&/2 %f Qtf.E} U /v Qtf>U V W$K U    *z-I[I[/\    ([-A-A%BRBR-S+0@0@ 0S )-ekk JKr ctjt|jxsi}g}|j D]\}}t |t s|g}|D]c}t |ts>d|vrdnd}t |tsdn|}t |ts|nd} t| ||}|j||fe||nd}t|||t|ddy) z Install the output recording hooks on all the modules in `model`. This will take care of correctly dispatching the `_can_record_outputs` property of each individual submodels in case of composite models. r8rr N)rrr!_output_capturing_hooks_installedT) _CAN_RECORD_REGISTRYr.str __class__itemsr=listrr<rLsetattr) modelrI capture_flagsrQrC layer_specsrRrrrs r!rMrMs ),,S-ABHbMM)//1 /[+t,&-K  /Ee^4,3)3E3)?TU ,6uc,Bu &LZde  #u .  / /)VrFeV]; E6=r ct|ddryt5t|ddr dddyt|dddy#1swYyxYw)z Check if the model already has output capturing hooks installed, and install them if it is not already the case. Note that this is thread-safe, in case 2 (or more) threads want to install them concurrently. rUFN)getattr_hook_installation_lockrM)r\s r!maybe_install_capturing_hooksrbsRu95A 2 5=u E  22 +51 222sA AA T)tie_last_hidden_statesc&fd}|||S|S)a Decorator to intercept specific layer outputs through hooks. The hooks are installed only once and lazily, the first time output capture is requested with the `output_xxx` kwargs/config. The implementation is fully context/thread safe, except when using `torch.compile`, as dynamo is unable to trace through `ContextVar` methods. Args: tie_last_hidden_states (`bool`, *optional*, defaults to `True`): Whether to overwrite `out.hidden_states[-1]` with the `out.last_hidden_state`. This is true for all language models and should be toggled off only if `out.hidden_states[-1]` has to be the hidden state before last layer norm, which is needed for some vision models (e.g. CLIP, SigLIP) c2tfd}|S)Nc x|jdt|jdd}tj t |j xsi}|Dcic]3}d||j d|t|jd|d5}}d|vr*|j dt|jdd|d<d|vr*|j dt|jdd|d <|jDcic]\}}|s |jdd g}}}t|d kDr t|tj|}  |g|i|} tj| |D]} | d k(r snkt| d r*|| dd|| <|| j| j n5t| dr)|| dd|| <|| j| j"t%|| | | <| dk(rht'|| t(rCt|| dk(r2t%|| d dd| | <t%|| ddd| d| z<t%|| | | <t%|| | | < |dur| j+} | Scc}wcc}}w#tj| wxYw)N return_dictToutput_Fcross_attentionsoutput_attentionsoutput_cross_attentionsmask_decoder_attentionsoutput_mask_decoder_attentionsrTrr8vision_hidden_stateslast_hidden_state attentionsr r cross_)popr`configrVr.rWrXrYreplacer;rbr9r0r3hasattrr<rnrpr>r=rZto_tuple)r)r@kwargsrgcapturable_flagskrecordable_keysvrB output_tokenoutputsrCfuncrcs r!wrapperz4capture_outputs..wrapped_fn..wrappers!**]GDKKX\4]^K 477DNN8KLRPR *! vzzGA3-PWXYWZm]b9cddO "%55=CZZ'>QSX)Y> 9:),<QSX)YE @AKZJ_J_Ja g$!Qef9b!92!= g  g$%)-d3,001BCL 6t5d5f5"'' 5) A/)1 *@A1B31G1L)#.)#.55g6R6RS *=>1B31G1L)#.)#.55g6O6OP#():3)?#@GCLL(!"23"7>3GWX[G\C]abCb',->s-CADqD-I'J 278I#8NqtRSt8T2U3/',->s-C'D #():3)?#@GCL) A,e#!**,No!h"'' 5s8J J J J""J9r)rrrcs` r! wrapped_fnz#capture_outputs..wrapped_fns! t=  = ~r r)rrcrs ` r!capture_outputsrs#AF $ r )r? nn.ModulerCrWrrreturnNone)rOrrPrWrQz list[tuple[str, OutputRecorder]]rrr-)r\rrIrrr)r\rrr)r __future__r threading contextvarsr dataclassesr functoolsrtypingr import_utilsr r rr rJrrVrr#r9rFrLrMLockrarbrrr r!rs #"! <0  :"" ""!*!*J**<=8(KK+.K?_K K@>:))..*2&TTr