GiqKUddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl mZmZddlmZd d lmZej(eZ dd lmZmZmZmZmZdd lmZdd lmZ  ddlm!Z! ddlm#Z$dZ%ejLjOddjQZ)dZ*e)evZ+de,fdZde,fdZ#deddideddiejZeddiejZeddiejZed d!iejZed"d!iejZeddid#ded$d%iej\ed&d'ejZed&d'iejZed(d'iejZed&d'iejZed&d)id*dejZej^zed+d,d -iidejZej^zed+d.d -iidejZej^zed+d/d -iidejZej^zed+d0d -iidejZej^zed+d1d -iidejZej^zed+d2d -iid3 Z0e1e,e1ee,zee1eefzffe2d4<e!DejZe!d5d67iej\e!d5d67ejZe!d5d67id8e0d9<d:Z3d\d;Z4dCd dDdEd dDdEd dDdFd dDdGd dDdHd dDdIZ5e1e,e1e,e,ffe2dJ<iZ6e1e,e dzfe2dK<dLe,dzdMe7fdNZ8 d]dLe,dOedzdPe7dMe dzfdQZ9e6fdRe,dSe1e,e dzffdTZ: d^dRe,dUe,dzdVe;e,zdzdPe7dMe f dWZdXeedZZ?gd[Z@y#e"$rdZ!Y-wxYw#e"$rdZ%Y2wxYw#e"$r dZ*dZ+d<Zd=Z#Gd>d?Zd@ZdAZdBZ4YwxYw)_N)Callable)contextmanager) ModuleType)version)ENV_VARS_TRUE_VALUESlogging)is_kernels_available)flash_attention_forward)DeviceLayerRepositoryModeregister_kernel_mappingreplace_kernel_forward_from_hub) get_kernel)use_kernel_forward_from_hub)FuncRepository)use_kernel_func_from_hubTFUSE_HUB_KERNELSYES layer_namecbtr t|StjdtdS)NFkernels hub usage is disabled through the environment USE_HUB_KERNELS=c|SNclss g/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/transformers/integrations/hub_kernels.pyz-use_kernel_forward_from_hub..Fss)_kernels_enabled$_kernels_use_kernel_forward_from_hublogger warning_once_TRANSFORMERS_USE_HUB_KERNELS)rs r rr?s2 7 C C   XYvXwx # "r" func_namectrtr t|Ststj ddStj dt dS)Nzwuse_kernel_func_from_hub is not available in the installed kernels version. Please upgrade kernels to use this feature.rc|Srrfuncs r r!z*use_kernel_func_from_hub..Usr")r#_has_use_kernel_func_from_hub!_kernels_use_kernel_func_from_hubr%r&r')r(s r rrHsV  =4Y? ?0##B% $##\]z\{|% $r"cudaz!kernels-community/deformable-detrMultiScaleDeformableAttention)repo_idrzkernels-community/moe Llama4TextMoezkernels-community/liger_kernels LigerRMSNormzkernels-community/rmsnormRMSNormzkernels-community/mlx_rmsnorm)r/rocmxpumpsnpuzmedmekk/triton-llama-mlpTritonLlamaMLPzkernels-community/megablocksMegaBlocksMoeMLPzahadnagy/megablocksCPUMegaBlocksMoeMLP)r/r5r6cpuzkernels-community/activationFastGELU)r1rr QuickGELUNewGELUSiluGeluGeluTanh) r0r2r4MLPr:r=r>r?SiLUGeLUrB_KERNEL_MAPPINGzkernels-community/rotaryapply_rotary_transformers)r1r()r6r/rotary_pos_embcT|vxs"tfd|jDS)Nc3ZK|]"}t|txr t|$ywr) isinstancedicthas_key).0vkeys r zhas_key..s&[1z!T2Fwq#F[s(+)anyvalues)drPs `r rMrMs'ax[3[PQPXPXPZ[[[r"cp|t}t|drtds tdt |y)Nr6z0.10.2) MIN_VERSIONzfkernels uses an incompatible version. Please install the latest version with `pip install -U kernels`.)rFrMr ImportErrorr)mappings r $register_kernel_mapping_transformersrYs8 ?%G 7E "+?H+Ux  (r"c d}|S)Nc|Srrrs r decoratorz.use_kernel_forward_from_hub..decoratorsJr"rargskwargsr\s r rrs r"c d}|S)Nc|Srrr+s r r\z+use_kernel_func_from_hub..decoratorsKr"rr]s r rrs r"ceZdZdZy)rctd)NzNLayerRepository requires `kernels` to be installed. Run `pip install kernels`. RuntimeError)selfr^r_s r __init__zLayerRepository.__init__ sop pr"N)__name__ __module__ __qualname__rgrr"r rr s qr"rctd)Nz^replace_kernel_forward_from_hub requires `kernels` to be installed. Run `pip install kernels`.rdr^r_s r rrs l  r"ctd)NzVregister_kernel_mapping requires `kernels` to be installed. Run `pip install kernels`.rdrls r rrssttr"ctd)Nzcregister_kernel_mapping_transformers requires `kernels` to be installed. Run `pip install kernels`.rdrls r rYrYs q  r"zkernels-community/causal-conv1d)r1rzkernels-community/mamba-ssmz!kernels-community/finegrained-fp8zkernels-community/deep-gemmzkernels-community/sonic-moe)z causal-conv1dz mamba-ssmzfalcon_mamba-ssmzfinegrained-fp8z deep-gemmz sonic-moe_HUB_KERNEL_MAPPING_KERNEL_MODULE_MAPPINGattn_implementationreturnc>|duxrtjd|duS)zJCheck whether `attn_implementation` matches a kernel pattern from the hub.Nz'^[^/:]+/[^/:]+(?:@[^/:]+)?(?::[^/:]+)?$)research)rqs r is_kernelrv*s- 4' c II@BU V^b br"attention_wrapperallow_all_kernelsctddlm}ddlm}d|vr|j ddn|}t |syt s tdd|vr%|j d\}}|j}n|}d}|j}|jd \}}} |j}| r| jnd} t|| | } t| dr |t}|} n| t| |} |j!| |j!||d| S#t$r} td |d | d d} ~ wwxYw)a Load and register the kernel associated to `attn_implementation`. Args: attn_implementation: A string, usually a kernel repo like "kernels-community/flash-mla". attn_wrapper: a callable for the wrapper around the attention implementation. In `transformers` we have a wrapper around the `flash_attn_var_len` call, and the same goes for `sdpa` and `eager`. They just prepare the arguments properly. This is mostly used for continious batching, where we want the `paged` wrapper, which calls the paged cache. allow_all_kernels (`bool`, optional): Whether to load kernels from unverified hub repos, if it is a custom kernel outside of the `kernels-community` hub repository. r)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS|r N`kernels` is either not installed or uses an incompatible version. Please install the latest version with `pip install -U kernels`.:@)revisionrxz-An error occurred while trying to load from 'z': .flash_attn_varlen_funcflash_attention_2) masking_utilsrzmodeling_utilsr{splitrv_kernels_availablerWstrip partitionr Exception ValueErrorhasattrr getattrregister) rqrwrxrzr{actual_attn_namer1 kernel_name_revkernelekernel_functions r load_and_register_attn_kernelr2si =8 ; - %--c37/**5  '/ //0F TI"))Y"E  ' (-@-B ,000O3DF'- $ $(GK ; =! (#'GK : ; 9 (#'GK 4 ; 9 ( '"&  ' ,'+ $ ;   ,s= AD>$E5EE- EE EEE*)E*rrcVddlm}ts td|j dd}|dk7r |s t dd||d }t jjd }tj|tjd k\rt|||| St||| S)Nr) __version__r}/rzkernels-communityziYou need to specify `allow_all_kernels=True` to use kernels outside of the `kernels-community` repository transformers) frameworkrr1kernelsz0.10.4)rr user_agentr) rrrWrrrmetadatar pkg_versionparseget_kernel_hub)rrrrxr repo_parentrkernels_versions r rrs   -  ##C(+K))2C w   .+R]^J((00;O)[->->x-HHkHgZdeekHgNNr" module_namesc8ttrgfd}|S)aO This decorator attaches the target function within the module as a plain attribute (not as a submodule). Keep in mind that this registration is only meant for `kernelize` to recognize its target modules (i.e. function exchanged for a weightless `nn.Module` with the same forward) to then exchange to the kernel variation (in-place) if the conditions are met. We cache each of these function-based registrations: After proper registration and exchange it is removed from the module's `_modules` dict as it does not really act as `nn.Module` but a base function. c<|jfd}||_|S)Nc|g|i||jjdi}D]B}t|ddxst|ddxs t|dd}|td||||<Dy)N_hidden_kernelsrhkernel_layer_namer(z)Could not infer kernel function name for )__dict__ setdefaultrr)rfr^r_hidden_kernelsfnnamer orig_inits r new_initz8use_kernelized_func..decorator..new_inits d ,T ,V ,"]]556GLN" *B D16r#6=6r;5 <$'PQSPV%WXX(*t$ *r")rg)rrrrs @r r\z&use_kernelized_func..decoratorsLL  *"   r")rKr)rr\s` r use_kernelized_funcrs#,)$~ . r"c#,K dadday#dawxYww)z Context manager used to adjust the value of the global `ALLOW_HUB_KERNELS`. This is needed, as this argument cannot be forwarded directly to the `__init__` of the models, where we set the attention implementation. TNF)ALLOW_ALL_KERNELSrr"r allow_all_hub_kernelsrs""  "Es ) rrrrrYrrrrr)NF)NNF)Aimportlib.metadatarosrtcollections.abcr contextlibrtypesr packagingrrutilsrr utils.import_utilsr flash_attentionr get_loggerrhr%rr rrrrrrrr$rrWrr.r-environrupperr'rr#str INFERENCETRAINING TORCH_COMPILErFrL__annotations__rMrYrorpboolrvrrintlistrrr__all__rr"r rs $%,154   H %}  * .Y(,%%'JJNN3De$L$R$R$T!48LL## %C %" O;:*  O/* =-!=-!7(!;(!=-!5  D O2+  :1 :1! 11!:1!:4!/ > !3!33_:)6  !3!33_:*6  !3!33_:(6  !3!33_:vWX6  !3!33_:vWX6  !3!33_:z[\6 wBeOT#tFSL/DI^D_2_$_``aBJ!6B]!  ~6B] 6B]!  - () \)^"CqQ:qI$AaP#FSTU:qI:qI 2T#tCH~-.8:S*t"33493:$di::19D:\`:$:zPf, #, S*t:K5K0L, b $# OODjO3Y O O  O:$d8nx&?$P " " O  .(-%.D!   qq u =! sU N-(N/N7I:N-NN-NN-N*&N-)N**N--"OO