GiUddlZddlZddlmZddlmZddlmZmZm Z ddl m Z m Z e jeZe rddlZerddlmZdZ d!d ed d ed d edzdedzdedeff dZ d"d ed d ed d edzdedzdededeff dZ d!d ed d ed d edzdedzdedeff dZ d#d d d ed d edzdedzdedeff dZ d#d d d ed d edzdedzdedeff dZ d#d d d ed d edzdedzdedeff dZeeeeeedZeeededefffe d<Gdde Z!GddZ"d$d e"de#dzfd Z$y)%N)Callablewraps) TYPE_CHECKINGOptional TypedDict)is_torch_availablelogging)PreTrainedConfigcHddddtdfd }|S)ad Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. cPtj|dz}|4|j}|j}d}|jj d}n?|j|}t ||d}|d}|jj |d}||kDr\t||ds%t|} | |j||dz|\} } |j|d  d t||d | y|j|}|j|d |d t||d |y)zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.r N original_max_position_embeddings_original_inv_freq__long_inv_freqseq_len layer_typeinv_freqF persistent long_inv_freqoriginal_inv_freq) torchmax rope_typerconfigrope_parametersgetattrhasattrROPE_INIT_FUNCTIONSregister_buffersetattrto) self position_idsdevicerrrrprefixr rope_init_fnrrs b/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_updatez6dynamic_rope_update..longrope_frequency_update/sY))L)A-  I $ 6 6 F/3{{/J/JKm/n ,z2I ' ?29= #/KK.dynamic_frequency_updateRs| ))L)A-  I!%!8!8  $ 6 6 Fz2I!(*=P/QSWSjSj!k  ' W>W X 4b .r.c| |jn|j|}|d|ini}d|vr||fd|ji|n|dk(r||fd|ji||||fi|S)Nrdynamicr)longrope)rr)) r'xr(rrkwargsr4r- rope_forwards r,wrapperz$dynamic_rope_update..wrapperxs&0&8DNNdnnZ>X /9/E, +2  ! $T< S SF S * $ %dL T TV TD!\tj| tj| tj| fd }n| }||z}|| fS) a Computes the inverse frequencies with proportional RoPE. Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): The proportion of the embedding dimension to apply rotary positional encoding, e.g., [0.0, 0.25, 0.5, 0.75, 1.0]. Unlike other RoPE functions that use this parameter, proportional RoPE will always return an encoding that is the size of `head_dim`. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrAr@rCrBrErrFrHrGr))rS)rIr r!rKrLrJrMrrNrOr&rPcatzerosfloat32)rr)rrrVrQrDrRr@rope_proportionrT rope_anglesinv_freq_rotated nope_anglesrs r,%_compute_proportional_rope_parametersr`sKJ ""$AKAW611*=]c]s]sv|T2ff6H6HFLfLf6fH  -D ! % %h 4F*../FLOo0A56K LLA OQekk B E EV[`[f[f E gjr r t a-+-KQ99  Ku}}VL   $ H % %%r.c|j||j|n |j}|d}|jdd}t|d|j|j z}t ||z}|d} d} | |j}n{t|tjrKtj|tj|j|j|j}nt||j}|| |z|jz | dz z ||dz z zz}d|tj d |dtj" j%|tj& |z zz } | | fS) a Computes the inverse frequencies with NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The default sequence length used to update the dynamic RoPE at inference time * rope_parameters (`dict[str, float]`): The standard RoPE scaling parameters, from which `factor` will be accessed. The value of `factor` is used to determine the new base frequency, along with the current sequence length (seq_len), the maximum positional embeddings (max_position_embeddings), and the computed dimensionality (dim) of the rotary embeddings. If seq_len <= max_position_embeddings, this factor has no effect. If seq_len <= max_position_embeddings, this factor effectively stretches the context window using an exponent derived from `dim`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length, used to update the dynamic RoPE at inference time. If `None` or shorter than max_position_embeddings, this value will be overridden by max_position_embeddings. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rArBrCrDr@rXr rErrFrH)rIr rJr!rKrLrMmax_position_embeddings isinstancerTensormaximumtensorrGr)rrNrOr&rP) rr)rrrQrRrBrDrSr@rTrs r,_compute_dynamic_ntk_parametersrgswV ""$AKAW611*=]c]s]s  -D0445LcRvz6+=+=A[A[+[\H h.. /C !( +F00 GU\\ *--  LL77w}}U\UcUc d  gv==> FW$v'E'EE&ST*U[^behibi[jk kDdu||AsAU[[ILLTZbgbmbmLnqttuvH % %%r.c|j||j|n |j}|d}|jdd}t|d|j|j z}t ||z}|d} |jd} |jd} |jd} |d } | |j| z } dd }| )| r| rt|| | || | z } n|| } |jd xsd }|jdxsd }dfd}d}|tjd|dj|tj|z z}d|z }d| |zz }|jjdd}|||||| |\}}d ||||dzj|tjz }|d |z z||zz}|| fS)aD Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://huggingface.co/papers/2309.00071) Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * rope_parameters (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied to the computed cos/sin. If None, the value is inferred from `factor`, `mscale`, and `mscale_all_dim` as available. * `beta_fast` (`float`, *optional*, defaults to 32): Parameter to set the boundary for extrapolation (only) in the linear ramp function. * `beta_slow` (`float`, *optional*, defaults to 1): Parameter to set the boundary for interpolation (only) in the linear ramp function. * `factor` (`float`, *optional*): The scaling factor applied when interpolating the position IDs to extend the possible context length. Additionally, if `attention_factor` is None, the log of this value is used to compute a value for `attention_factor`, possibly in conjunciton with `mscale` and `mscale_all_dim`, if provided. * `mscale` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale` acts scalar augmenting `log(factor)` when computing the numerator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `mscale_all_dim` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale_all_dim` acts scalar augmenting `log(factor)` when computing the denominator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `original_max_position_embeddings` (`int`): The original max position embeddings used during pretraining. * `truncate` (`bool`, *optional*): Whether to truncate the correction range. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rArBrCrDr@rTmscalemscale_all_dimrr cJ|dkryd|ztj|zdzS)Nr rCg?)mathlog)scaleris r, get_mscalez,_compute_yarn_parameters..get_mscales( A:V|dhhuo-33r. beta_fast beta_slowc|tj||dztjzz zdtj|zz S)zPInverse dimension formula to find the dimension based on the number of rotationsrE)rlrmpi) num_rotationsrSrRrbs r,find_correction_dimz5_compute_yarn_parameters..find_correction_dimsBdhh6-!:Kdgg:UVWW\]`d`h`him`n\noor.c||||}||||}|r*tj|}tj|}t|dt ||dz fS)z.Find dimension range bounds based on rotationsrr )rlfloorceilrmin) low_rothigh_rotrSrRrbtruncatelowhighrvs r,find_correction_rangez7_compute_yarn_parameters..find_correction_ranges^!'36MN"8S$8OP **S/C99T?D3{CcAg...r.c||k(r|dz }tj|tj|z ||z z }tj|dd}|S)NgMbP?rFrr )rrNr[clamp)rzrrS linear_func ramp_funcs r,linear_ramp_factorz4_compute_yarn_parameters..linear_ramp_factorsL #: 5LC||Cu}}=Cc R KK Q2 r.rrErHr}T)r ) rIr rJr!rKrLrMrbrPrrNr&)rr)rrrQrRrBrDrSr@rTrirjrrorprrrr pos_freqsinv_freq_extrapolationinv_freq_interpolationr}r~rinv_freq_extrapolation_factorrrvs @r,_compute_yarn_parametersrGs9t ""$AKAW611*=]c]s]s  -D0445LcRvz6+=+=A[A[+[\H h.. /C !( +F+//0BC ! % %h /F)--.>?N';<^'_$ ~//2RR4  n$Z%?*VUcBd%de )&1 %((5;I$((5:Ip/aa03363UX[[\I 9_ FY$67%%))*d;H%iCGgiqrIC%&(:3cQh(O(R(RZ`hmhshs(R(t$t!!&C"CD #@ @ A  % %%r.c@|j||j|n |j}|d}|jdd}t|d|j|j z}t ||z}|d} |d} |jd} |jd} |d } | |j| z } | I| dkrd} nAtjd tj| tj| z z} |r,|| kDr'tj| tj| }n&tj| tj| }tjd |d tj| j!|z }d|||zzz }|| fS)a Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * original_max_position_embeddings (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, defaults to `max_position_embeddings`. * rope_parameters (`dict[str, float]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, inferred from the value of `factor`. * `factor` (`float`, *optional*): The scaling factor to apply to the RoPE embeddings. If both `max_position_embeddings` and `original_max_position_embeddings` are provided, this value will be overridden s the ratio between those values. * `long_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is provided and greater than `original_max_position_embeddings`. * `short_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rArBrCrD long_factor short_factorr@rTrr rXrrE)rIr rJr!rKrLrMrbrlsqrtrmrrfr[rNrOrP)rr)rrrQrRrBrDrSrrr@rTr ext_factorsinv_freq_shapers r,_compute_longrope_parametersrsd ""$AKAW611*=]c]s]s  -D0445LcRvz6+=+=A[A[+[\H h.. /C&}5K'7L ! % %h /F+//0BC';<^'_$ ~//2RR S=" #yyTXXf-=Ii@j-j)jk 7==ll;emmFS ll?*+MNO&8'*::$''kH$G[[+;!;X=NPXYN$w.@EUXgEghM]*n