GiddlmZddlmZddlmZmZddlmZddl m Z m Z eje Zer ddlZddlmZ d ddZGd d eZGd d eZy)) annotations)Any)is_torch_availablelogging) ConversionOps)get_module_from_nameshould_convert_moduleNc ddlm}|g}t|jD]\}}t |t j s!t||s.|jd\} } } | r|j| n|} ||s |jnd|s |jnd|s|jdund|||d} t| | | |S)a Replace nn.Linear modules with empty SINQLinear modules. Args: model: The model to modify modules_to_not_convert: List of module names to skip quant_config: SINQ quantization config dict (None for pre-quantized models) compute_dtype: Computation dtype for the quantized layers device: Device string for the quantized layers pre_quantized: Whether loading a pre-quantized checkpoint Returns: The modified model with SINQLinear modules r) SINQLinearN.FT) in_features out_featuresbias quant_config compute_dtypedeviceuse_unpack_kernel)sinq.sinqlinear_hfr list named_modules isinstancennLinearr rpartition get_submodulerrrsetattr)modelmodules_to_not_convertrrr pre_quantizedr full_namemodule parent_path_ child_nameparent sinq_layers `/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/transformers/integrations/sinq.pyreplace_with_sinq_linearr) s,.%!#!%"5"5"780 6&")), $Y0FG %.%9%9#%>" Q 5@$$[1e2?**T4A,,t2?&++T)U%'"   J/'0* Lc4eZdZdZdZ d ddZy) SinqQuantizea' Param-level ConversionOp for SINQ (from FP weights). At load time, for each `Linear.weight` that should be quantized: - The SINQLinear module already exists (created in _process_model_before_weight_loading) - We just call quantize() on it with the loaded weight tensor c||_yN hf_quantizerselfr0s r(__init__zSinqQuantize.__init__\ (r*Nc tt|j\}}t|tr|dn|}t ||\} } | j |||j|d| _iS)NrT) nextiteritemsrrr quantizediscard_is_hf_initialized) r2 input_dictrfull_layer_name missing_keyskwargsr$values weight_tensorr" tensor_names r(convertzSinqQuantize.convert_srj..012 6%/%=q 6 25/J  &  #   1$(! r*)NNNr<zdict[str, Any]rztorch.nn.Module | Noner=z str | Nonereturnzdict[str, torch.Tensor]__name__ __module__ __qualname____doc__r3rCr*r(r,r,SsB) )-&* "&$  !r*r,c2eZdZdZdZ d ddZy)SinqDeserializea0 ConversionOp for loading *pre-quantized* SINQ checkpoints. Checkpoint layout (what `SINQLinear.state_dict` produces) is, per module: .W_q .bias .meta WeightConverter in the quantizer is configured so that: - we group ".W_q", ".meta", ".bias" as input_dict - conceptually treat them as belonging to ".weight" - and call this SinqDeserialize.convert to load the state into the existing SINQLinear. The returned dict is {} because we load directly into the module. c||_yr.r/r1s r(r3zSinqDeserialize.__init__r4r*Nc t|jD]\}}t|ts|d||< |jd}|jd}|jd} ||;t t |j }t|tr|d}||iSt||\} } ||d} | | | d<| j| d| _ iS)Nrz.W_qz.metaz.bias)W_qmetarT) rr8rgetr6r7r@r load_state_dictr;) r2r<rr=r?kvrPrQrr"r$states r(rCzSinqDeserialize.converts))+, %DAq!T" !! 1  %nnV$~~g&~~g& ;$,T*++-./A!T"aD#Q' '(@     E&Mu%$(! r*)NNrDrFrKr*r(rMrMvs? ) )-&* #"#&#$ # ! #r*rM)NNNzcuda:0F)rtorch.nn.Modulerzlist[str] | Nonerz dict | Nonerz torch.dtyperstrr boolrErW) __future__rtypingrtransformers.utilsrrcore_model_loadingrquantizers.quantizers_utilsr r get_loggerrGloggertorchtorch.nnrr)r,rMrKr*r(rcs#:.U   H % 04 $!% 0 0,00 0  0  00f = F7m7r*