Gi) ~dZddlmZmZddlmZddlmZmZer ddl Z ddl m Z eje ZdadZGdd e j"Z dd eedzd efd Zd e j.dedefdZde j.de j.de j.dedef dZGddeZGddeZy)a Metal affine quantization integration for transformers. This module provides: - ``MetalLinear``: a drop-in replacement for ``nn.Linear`` that stores weights as affine-quantized uint32 packed tensors and uses the ``quantization-mlx`` Metal kernels for the forward pass. - ``replace_with_metal_linear``: walks a model and swaps every eligible ``nn.Linear`` with ``MetalLinear``. - ``MetalQuantize`` / ``MetalDequantize``: weight conversion operations that participate in the new ``WeightConverter`` pipeline. Weight layout (transposed, matching ``affine_qmm_t``): - ``weight``: ``[N, K_packed]`` (``uint32``) -- K is the packed dimension. - ``scales``: ``[N, K // group_size]`` (``float16 / bfloat16``) - ``qbiases``: ``[N, K // group_size]`` (same dtype as scales) The kernel call is ``affine_qmm_t(x, weight, scales, qbiases, group_size, bits)`` which computes ``y = x @ dequant(weight).T``, identical to ``nn.Linear``. ) ConversionOps _IdentityOp)should_convert_module)is_torch_availableloggingNct ddlm}|datStS#t$r}t d|d|d}~wwxYw)z>Lazily load the quantization-mlx kernel from Hugging Face Hub.N) get_kernelz0kernels-community/mlx-quantization-metal-kernelsz9Failed to load the quantization-mlx kernel from the Hub: zm. Make sure you have `kernels` installed (`pip install kernels`) and are running on an Apple Silicon machine.) _metal_kernel hub_kernelsr Exception ImportError)r es n/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/transformers/integrations/metal_quantization.py_get_metal_kernelr3s`  /&'YZM =  KA3O??   s" A;Ac eZdZdZdej ddfdedededed ef d Zd ejd ejfd Z y) MetalLinearz A quantized linear layer that stores weights in affine uint32 packed format and uses the ``quantization-mlx`` Metal kernels for the forward pass. Parameters match ``nn.Linear`` with additional quantization metadata. F in_features out_featuresbiasbits group_sizec:tjj|||_||_||_||_d|z}||z}||z} |tjk(rAtjtj||tjd|_ n2tjtj|||d|_ |tjk(rtjnd} tjtj|| | d|_ tjtj|| | d|_|r.tjtj||_y|j!ddy)N )dtypeF) requires_gradr)nnModule__init__rrrrtorchuint32 Parameterzerosweightfloat32scalesqbiasesrregister_parameter) selfrrrrrr elems_per_intk_packedn_groups scales_dtypes rr"zMetalLinear.__init__Qs' 4 &( $d -/*, ELL ,,u{{<QVQ]Q]'^nstDK,,u{{<TY'ZjopDK(-(=u}}4 ll5;;|X\#Zjop ||EKK hl$[kpq  U[[%>?DI  # #FD 1inputreturnc|jjtjk7r5tj j ||j|jSt}|j||j|jj|j|jj|j|j|j}|j||jz}|SN)r'rr#r$r functionallinearrr affine_qmm_tr)tor*rr)r,r2kerneloutputs rforwardzMetalLinear.forwardss ;;   ,==''t{{DIIF F"$$$  KK KKNN5;; ' LLOOEKK ( OO II   99 dii'F r1N) __name__ __module__ __qualname____doc__r#r$intboolr"Tensorr<r1rrrIsjll 2 2 2 2  2 2DU\\ellr1rmodules_to_not_convert pre_quantizedc |jr|S|j}|j}d}|jD]z\}}t ||st |t js.|rinddi} td|j|j|jdu||d| } |j|| d}||stjd|S)a` Replace every eligible ``nn.Linear`` with ``MetalLinear``. Args: model: the ``PreTrainedModel`` (on the meta device at this point). modules_to_not_convert: module names to leave untouched. quantization_config: the ``MetalConfig`` instance. pre_quantized: ``True`` when loading from a quantized checkpoint. FrN)rrrrrTzYou are loading a model with Metal quantization but no nn.Linear modules were found. Please double check your model architecture.rD) dequantizerr named_modulesr isinstancer Linearrrrr set_submoduleloggerwarning) modelrEquantization_configrFrrhas_been_replaced module_namemodule module_kwargs new_modules rreplace_with_metal_linearrVs%%  # #D$//J$224% V$[2HI  fbii ("/Bgt_M$"..#00[[,%   J    Z 8 $ !%$  ; Lr1r'rrc |j\}}d|z}d|zdz }||z}|jj|||}|jdj} |j dj} | | z |z j d} | } || jdz | jdz } | jj d|jtjj||} ||z}tj||tj|j}t|D]}|| d d |d |f||zzz}|jtj| | fS) aP Quantize a 2-D float weight ``[N, K]`` into packed uint32 + scales + biases. Returns ``(w_packed, scales, biases)`` with: - ``w_packed``: ``[N, K // (32 // bits)]`` uint32 - ``scales``: ``[N, K // group_size]`` float32/float16/bfloat16 - ``biases``: ``[N, K // group_size]`` float32/float16/bfloat16 rr )dimg:0yE>)minrrdeviceN)shapefloatreshaperZvaluesmaxclamp unsqueezeroundr9r#int32r&r\ranger$)r'rrNKr-max_valr/ w_groupedw_minw_maxr)biasesw_intr.w_packedis r_affine_quantize_tensorrqsm <r?r@r"dictrrDr1rrxrxs) $ T r1rxcDeZdZdZdZd dededzdefdZed dZ y) MetalDequantizez Dequantize (weight, scales, qbiases) back to a full-precision tensor. Used when ``dequantize=True`` is set in the config to fall back to a normal ``nn.Linear`` on devices without MPS. c||_yr5rzr|s rr"zMetalDequantize.__init__r}r1Nr~full_layer_namer3c 4|jjj}|jjj}t |dkr||diS|dd}|dd}|dd}t |||||} || j |jiS)Nrzweight$rr)r*)r{rPrrlenrvr9r) r,r~rrrr quantizedr)r*rus rrzMetalDequantize.converts  4499&&::EE z?Q #Z %:; ;y)!, H%a(Y'*))VWjRVW&,,!788r1ctSr5)r)r,s r reverse_opzMetalDequantize.reverse_op*s }r1r5)r3r) r=r>r?r@r"rstrrpropertyrrDr1rrrs?) 9$ 9t 9Y] 9r1r)NNF)r@core_model_loadingrrquantizers.quantizers_utilsrutilsrrr#torch.nnr get_loggerr=rMr rrKrrrrBrVrCrArqrvrxrrDr1rrs*<?/   H % ,;"));@04 / I,/ /d5ELL5c55Bll$)LL:?,,TW_b. M @mr1