GiV) 8UdZddlZddlZddlmZddlmZmZddlZddlm Z ddl m Z m Z ddl mZmZmZmZdd lmZdd lmZer dd lmZdd lmZej4eZiZeeee eee e jBfffe"d <GddZ#GddZ$Gdde#Z%dddddede#deee e jBff dZ&dddddede#ddf dZ'de%iZ(eee#fe"d<deee)eeefzfde*efd Z+ d"dddeee)eeefzfdzddfd!Z,y)#zrFusion registration helpers. See `docs/source/en/fusion_mapping.md` for the design overview and extension guide. N)Mapping) TYPE_CHECKINGAny)nn)!get_checkpoint_conversion_mapping®ister_checkpoint_conversion_mapping)Conv3dToLinearWeightConverterWeightRenamingWeightTransform)register_patch_mapping)logging)PretrainedConfig)PreTrainedModel_FUSION_DISCOVERY_CACHEceZdZUdZdZeedfed<dedefdZde jde fd Z d e e jde e jfd Zd d deefdZy)ModuleFusionSpeczBase recipe for a fusion family. A fusion spec decides which modules are eligible for a fusion, how to build the runtime replacement class, and which weight transforms are needed to map checkpoints between the original and fused layouts. .target_modules_patterns model_namereturnc8dt|jd|S)zDReturn the log message emitted when no compatible modules are found.zNo compatible z classes found to fuse for )type__name__)selfrs ]/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/transformers/fusion_mapping.py get_empty_logzModuleFusionSpec.get_empty_log6s"T 3 344OPZ|\\modulect)z>Return whether `module` is compatible with this fusion family.NotImplementedError)rr s r is_fusablezModuleFusionSpec.is_fusable:!!r original_clsct)zBBuild the runtime replacement class for a compatible module class.r")rr&s rmake_fused_classz!ModuleFusionSpec.make_fused_class>r%rconfigrct)zMBuild the weight transforms needed to load and save the fused runtime layout.r")rr)s rmake_transformsz ModuleFusionSpec.make_transformsBr%rN)r __module__ __qualname____doc__rtuplestr__annotations__rrModuleboolr$rr(listr r+rrrrr,s02U38_1]]]""t""T"))_"bii""&8"T/=R"rrcVeZdZfdZdej dej fdZxZS)_FusedPatchEmbeddingMixinct||i||jjt j |jj z|_tj|j|jj|jjdu|jjj|jjj|_|`y)N)biasdevicedtype)super__init__proj in_channelsmathprod kernel_size patch_volumerLinear out_channelsr8weightr9r: linear_proj)rargskwargs __class__s rr<z"_FusedPatchEmbeddingMixin.__init__Hs $)&) II11DIIdii>S>S4TT99    II " "t+99##**))""((   Ir hidden_statesrc|jjj}|jd|j}|j|j |}|jd|j S)N)r:)rFrEr:viewrBto embed_dim)rrJ target_dtypes rforwardz!_FusedPatchEmbeddingMixin.forwardWsf''..44 %**2t/@/@A (()9)9 )9)MN !!"dnn55r)rr,r-r<torchTensorrQ __classcell__)rIs@rr6r6Gs# 6U\\6ell6rr6ceZdZdZdZdej defdZde ej de ej fdZ dd de e fd Z y ) PatchEmbeddingsFusionSpeczJFuse compatible Conv3d patch embeddings into flattened Linear projections.)z(^|\.)patch_embed$r rctt|ddx}tjsy|j|j k(xr1|j dk(xr |jdk(xr|jdk(S)Nr=F)rrr)rrrr) isinstancegetattrrConv3dstriderApaddingdilationgroups)rr r=s rr$z$PatchEmbeddingsFusionSpec.is_fusablecso'&&$"??$K KK4++ + ! ) ! * ! q  rr&cntd|jt|fi}d|j|_|S)NFused)rrr6r-)rr& fused_clss rr(z*PatchEmbeddingsFusionSpec.make_fused_classosB5!6!6 78;TVb:cegh #()B)B(C!D rr)rct|d|}|j}t|tr||f}|jgt |}|j }tddt||gtddgS) N vision_configzpatch_embed\.proj\.weight$z!patch_embed\.linear_proj\.weight$)r>rA)source_patternstarget_patterns operationszpatch_embed\.proj\.bias$zpatch_embed\.linear_proj\.bias$)rdre) rY patch_sizerXinttemporal_patch_sizer/r>r r r )rr)rcrgrAr>s rr+z)PatchEmbeddingsFusionSpec.make_transformsts@ "-- j# &$j1J$88M5;LM #//   = D"$/$/   ; B   rN)rr,r-r.rrr2r3r$rr(r4r r+rrrrVrV^s^T6    t  T"))_bii  &8 T/=R rrVclsztype[PreTrainedModel]r)r fusion_namespecrc<tj|i}||vr||Stjd5||}dddt }i}|j r.t jdj|j nd}jD]j\} } t| } | |vr||j| *|j| s<|j| |j| || j<l|||<|S#1swYxYw)arDiscover compatible module classes for one fusion family on a meta-initialized model. This function: - instantiates `cls(config)` on the meta device - scans `named_modules()` for candidate modules - optionally pre-filters them with `target_modules_patterns` - uses `is_fusable(...)` as the final structural check - builds the class-level patch mapping used by monkey patching Results are cached per `(fusion_name, cls)` to avoid repeated meta-initialization. This matches the current class-level fusion behavior, where one compatible module class maps to one fused replacement class. metaN|)r setdefaultrRr9setrrecompilejoin named_modulesrsearchr$addr(r) rjr)rkrlcachemodel seen_classes patch_mappingtarget_module_pattern module_namer module_clss r_discover_fusable_modulesrs"( $ . .{B ?E e|Sz f F 5LM>B>Z>Z 388D889:`d %224 O V&\  %  ,1F1M1Mk1Z1b v& $-1-B-B:-N j))* OE#J +s DDc zt||||}|s/tj|j|jyt |dt |drt |jdstd|jd|jj}|j|}t|}|l|Dcic]}t|j|} }|D]?} t| j} | j| } | ,td |d |d | d ||z}t||dycc}w) aRegister one fusion family for `cls`. This function updates the two global registries used by fused loading: - the monkey-patching registry, so compatible module classes are replaced before initialization - the checkpoint conversion mapping, so fused runtime modules still load from the original checkpoint layout Notes: - conflicting checkpoint transforms fail fast )rkrlNT) overwrite config_class model_typezModel z" has no config class or model typezFusion z for model type zC conflicts with an existing conversion mapping for source patterns .)rloggerinforrrhasattrr ValueErrorrr+rr/rdgetr ) rjr)rkrlfusable_classesr convertersexisting_convertersexistingexisting_converter_sources converterrdexisting_converters r_register_module_fusionrsO0V[_`O  D&&s||45?d; 3 'ws7G7G/V6#,,/QRSS!!,,J%%f-J;JG&at%tT\eH,D,D&Ex&O%t"%t# I#I$=$=>O!;!?!?!P !- k]*::,G++:*;1>  ):5 *:zTR&usD8patch_embeddings_FUSION_REGISTRY fusion_configcg}|jD]T\}}|tvrtd||dur!|durt|tstd|d|j |V|S)zQValidate `fusion_config` and return enabled fusion names in user-specified order.zUnknown fusion type: FTzInvalid fusion config for z4: expected `True`, `False`, or a mapping of options.)itemsrrrXrappend)renabled_fusionsrkfusion_optionss r_iter_enabled_fusionsrsO'4':':'< ,# ^ . .4[MBC C U "   %j.Q,[M9mn  {+ , rcT|syt|D]}t|||t|y)aRegister requested runtime fusions for `cls`. This function: - validates `fusion_config` against `_FUSION_REGISTRY` - resolves the enabled fusion families in user order - registers monkey patches and checkpoint transforms before model instantiation N)rrr)rjr)rrks rregister_fusion_patchesrs4 ,];Y V[:J;:WXYr)N)-r.r?rrcollections.abcrtypingrrrRrconversion_mappingrr core_model_loadingr r r r monkey_patchingrutilsrconfiguration_utilsrmodeling_utilsr get_loggerrrrdictr0rr2r1rr6rVrrrr3r4rrrrrrs  #% i``35/   H %MOc4d3RYY3G.H(H#IIJO""666.- 0- `- - --  -  #tBII  -`*S *S*<*SKN*SVf*S *SZ2DE^E`0a$s,,-adWS#X=N6N1N)OTXY\T]$jnY Y9@dWUXZ]U]M^F^A^9_bf9fY Yr