Hi(8ddlmZdZdZdZGddeZy))PushToHubMixincd} t|j}|jj }|dk(r(ddlm}|rddl}t|djy |S#t$rtd|wxYw) z Infers the device type from the model parameters. Args: model: The model instance. Returns: The device type. z { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } zPCannot determine model device, please provide a device to the mapping. Example: cudar)is_torch_availableNversionrocm) next parameters StopIteration ValueErrordevicetypeutilsrtorchgetattrhip)modelEXAMPLE_MAPPINGparamdev_typerrs b/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/transformers/utils/kernel_config.py infer_devicers O U%%'( ||  H6.   ui(,,8 O!  ^_n^o p   s A""A:cddlm}|dvrtd||jdd}|jdd}|||||ii||<y)NrLayerRepositoryrr xpunpuneuron=Only cuda, rocm, xpu, npu and neuron devices supported, got: :)repo_id layer_name)kernelsrr split)r%r repo_namemodecompatible_mappingrrepo_layer_namer$s radd_to_mappingr,:sn' ==XY_X`abbooc*1-Oooc"1%G /* &z"cddlm}ddlm}|dvrt d||j dd}|j dd}|j dd } |||||| | ii||<y) Nr)Path)LocalLayerRepositoryrr!r"r#/) repo_path package_namer%)pathlibr/r&r0r r') r%rr(r)r*r/r0r+r3repo_package_names radd_to_mapping_localr7Ks, ==XY_X`abbooc*1-O$Q'I!,R0 &y/.* &z"r-c4eZdZdZddZd dZdZdZd dZy) KernelConfigze Kernel configuration class. This class is used to configure the kernel mapping for a model. Nc6||ni|_i|_||_yN)kernel_mappingregistered_layer_namesuse_local_kernel)selfr<r>s r__init__zKernelConfig.__init__es"0>0JnPR&(# 0r-cHddlm}||||||ii|j|<y)Nrr)r$r%revision)r&rr<)r?r$registered_namer%rr)rBrs r update_kernelzKernelConfig.update_kerneljs4+ o#)%0 O,r-c|jD]+\}}t|ds|j|j|<-y)Nkernel_layer_name) named_moduleshasattrrFr=)r?rnamemodules rstore_registered_layer_namesz)KernelConfig.store_registered_layer_nameswsA!//1 MLD&v234:4L4L++D1 Mr-c|d}|j|t|jts$t d|dt |j|jj D]\}}||jjvrt d|dt|trd|vsd|vsJt d|d |t|trT|j D]@\}}|d vrt d |t|tr d|vsd|vs1t d|d |t d |d|y )aJ Validates the kernel_mapping to ensure that: 1. Each layer_name in the mapping is registered in the model (i.e., the model contains a module with a matching kernel_layer_name). 2. Each kernel value is either a string of the form 'org/repo:layer_name' or a dict mapping device types ("cuda", "rocm", "xpu", "npu") to such strings. 3. Each device key in a dict is one of "cuda", "rocm", "xpu", or "npu". 4. Each repo_name is a valid repository and layer name in the format 'org/repo:layer_name' (i.e., a string containing both a slash and a colon). 5. If a local path is detected, it should be in the format '/abs/path:layer_name'. The absolute path must include the `package_name`, like "/home/user/layer_norm". Args: model: The model instance whose modules are checked for registered kernel_layer_name attributes. Raises: ValueError: If a layer_name is not registered in the model, if a device is not supported, or if a repo_name is not a valid 'org/repo:layer_name' string. a For single device form remote { "RMSNorm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, For multiple devices form remote { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", "rocm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } For single device form local { "RMSNorm": "/abs/path:LlamaRMSNorm", ... }, For multiple devices form local { "RMSNorm": { "cuda": "/abs/path:LlamaRMSNorm", "rocm": "/abs/path:LlamaRMSNorm", ... }, ... } z7Kernel mapping must be a dict of the following format: z, got: zLayer z[ is not registered in the model, please register it first using use_kernel_forward_from_hubr1r"zKernel mapping for 'zl' must be a valid repo name with a layer name (e.g., 'org/repo:layer_name' or '/abs/path:layer_name'), got: rr!z'Kernel mapping must follow the format: N) rK isinstancer<dictr ritemsr=valuesstr)r?rMAPPING_FORMATr%kernelrr(s rsanitize_kernel_mappingz$KernelConfig.sanitize_kernel_mapping|s # H ))%0$--t4I.IYY`aefjfyfyaz`{| #'"5"5";";"= l J!JZZabhai!jkk- lr-c ddlm}i}t|}|jj D]\}}|j r |j n |j}|r||jz}t|tr.|} |jst||| ||it||| ||yt|ts|j D]6\} } | |k7r |jst|| | ||(t|| | ||8||_y)aW Transforms a simple kernel_mapping of the form: { "RMSNorm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, or for local path: { "RMSNorm": "/home/user/liger_kernels:LigerRMSNorm", ... }, into a nested mapping: { "RMSNorm": { "cuda": { Mode.INFERENCE: LayerRepository( repo_id="kernels-community/layer_norm", layer_name="LlamaRMSNorm", ) } } } or for local path: { "RMSNorm": { "cuda": { Mode.INFERENCE: LocalLayerRepository( repo_path=Path("/home/user/liger_kernels"), package_name="liger_kernels", layer_name="LigerRMSNorm", ) } } } that's compatible with the kernels library. The device is inferred from the model's parameters if not provided. The Mode is inferred from the model's training state. r)ModeN)r&rVrr<rOtrainingTRAINING INFERENCE TORCH_COMPILErMrQr>r,r7rN) r?rcompilerVr*current_devicer%rSr)r(rs rcreate_compatible_mappingz&KernelConfig.create_compatible_mappingsb !%e,"&"5"5";";"= f J$)NN4==Dd000&#&" ,,":~y$Pbc(^YPTVhiFD))/f%FI/ 00&z69dL^_,ZDRde f f*1r-)NFr;)F) __name__ __module__ __qualname____doc__r@rDrKrTr]r-rr9r9`s%1  M QlfJ1r-r9N)rrrr,r7r9rbr-rrcs(#%P"*y1>y1r-