Gi9ddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z m Z mZmZddlZddlmZddlmZddlmZmZmZmZmZdd lmZmZmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9ddl:m;Z;mZ>m?Z?ddl@mAZAddlBmCZCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQmRZRmSZSmTZTmUZUmVZVmWZWmXZXmYZYmZZZm[Z[m\Z\ddl]m^Z^m_Z_m`Z`maZambZbmcZcmdZde rddlemfZfddlgmhZhddlimjZjddlkmlZle,jenZoe+rddlpmqZqmrZrgdZse?jde?jde?jde?jde?jde?jde?jd e?jd!e?jd"i Z}e Gd#d$e)Z~e Gd%d&e)Ze Gd'd(e)Ze Gd)d*e)Ze~ezZeezZeezZGd+d,eAZd-Zd/d.Zy)0N)Callable)contextmanager) dataclass) TYPE_CHECKINGAnyOptionalcast)nn)Cache DynamicCacheEncoderDecoderCacheQuantizedCache StaticCache)check_python_requirementsget_cached_module_fileget_class_in_moduleresolve_trust_remote_code)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_masks_for_generate)ExtensionsTrie) ModelOutputTransformersKwargsis_accelerate_availablelogging)is_flash_attention_requested) AssistantVocabTranslatorCacheAssistedCandidateGenerator-AssistedCandidateGeneratorDifferentTokenizersCandidateGeneratorEarlyExitCandidateGeneratorPromptLookupCandidateGenerator%UniversalSpeculativeDecodingGenerator_prepare_attention_mask_prepare_position_ids_prepare_token_type_ids) ALL_STATIC_CACHE_IMPLEMENTATIONS'DEPRECATED_STATIC_CACHE_IMPLEMENTATIONSSTATIC_CACHE_IMPLEMENTATIONSGenerationConfigGenerationMode)ContinuousMixin)#EncoderNoRepeatNGramLogitsProcessor'EncoderRepetitionPenaltyLogitsProcessorEpsilonLogitsWarperEtaLogitsWarperExponentialDecayLengthPenaltyForcedBOSTokenLogitsProcessorForcedEOSTokenLogitsProcessorInfNanRemoveLogitsProcessorLogitNormalizationLogitsProcessorListMinLengthLogitsProcessor!MinNewTokensLengthLogitsProcessorMinPLogitsWarperNoBadWordsLogitsProcessorNoRepeatNGramLogitsProcessor PrefixConstrainedLogitsProcessor RepetitionPenaltyLogitsProcessorSequenceBiasLogitsProcessor$SuppressTokensAtBeginLogitsProcessorSuppressTokensLogitsProcessorTemperatureLogitsWarperTopHLogitsWarperTopKLogitsWarperTopPLogitsWarperTypicalLogitsWarper.UnbatchedClassifierFreeGuidanceLogitsProcessor)ConfidenceCriteriaEosTokenCriteriaMaxLengthCriteriaMaxTimeCriteriaStoppingCriteriaStoppingCriteriaListStopStringCriteria)GenerativePreTrainedModel)PreTrainedModel)PreTrainedTokenizerBase) BaseStreamer)AlignDevicesHookadd_hook_to_module)past_key_values cache_paramsstatememspast_buckets_states_sample _beam_search_assisted_decodingztransformers-community/dolaz)transformers-community/contrastive-searchz(transformers-community/group-beam-searchz.transformers-community/constrained-beam-searchceZdZUdZej ed<dZeejdzed<dZ eejdzed<dZ eeejdzed<dZ eeejdzed<dZ edzed<y) GenerateDecoderOnlyOutputa\ Outputs of decoder-only generation models, when using non-beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. past_key_values (`Cache`, *optional*, returned when `use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. sequencesNscoreslogits attentions hidden_statesrV)__name__ __module__ __qualname____doc__torch LongTensor__annotations__ratuple FloatTensorrbrcrdrVr _/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/transformers/generation/utils.pyr_r_s4.2FE%## $t +2.2FE%## $t +29=JeE--./$6=<@M5u0012T9@$(OUT\(ror_ceZdZUdZej ed<dZeejdzed<dZ eejdzed<dZ eejdzed<dZ eejdzed<dZ eeejdzed<dZeeejdzed <dZeeejdzed <dZedzed <y) GenerateEncoderDecoderOutputa Outputs of encoder-decoder generation models, when using non-beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Processed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer of the decoder) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. past_key_values (`Cache`, *optional*, returned when `use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. r`Nrarbencoder_attentionsencoder_hidden_statesdecoder_attentionscross_attentionsdecoder_hidden_statesrV)rerfrgrhrirjrkrarlrmrbrsrtrurvrwrVr rnrorprrrrs!F.2FE%## $t +2.2FE%## $t +2:>e//047>=A5!2!23d:AAEeE$5$567$>E?CeE%"3"345<CDH5u'8'8!9:TAH$(OUT\(rorrcXeZdZUdZej ed<dZejdzed<dZ e ejdzed<dZ e ejdzed<dZ ej dzed<dZ e e ejdzed<dZe e ejdzed <dZedzed <y) GenerateBeamDecoderOnlyOutputa Outputs of decoder-only generation models, when using beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`torch.FloatTensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Beam transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. beam_indices (`torch.LongTensor`, *optional*, returned when `output_scores=True`): Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. past_key_values (`Cache`, *optional*, returned when `use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. r`Nsequences_scoresrarb beam_indicesrcrdrV)rerfrgrhrirjrkrzrmrarlrbr{rcrdrVr rnrorpryrys@15e''$.5.2FE%## $t +2.2FE%## $t +2,0L%""T)09=JeE--./$6=<@M5u0012T9@$(OUT\(roryceZdZUdZej ed<dZejdzed<dZ e ejdzed<dZ e ejdzed<dZ ej dzed<dZ e ejdzed<dZe ejdzed <dZe e ejdzed <dZe e ejdzed <dZe e ejdzed <dZedzed <y) GenerateBeamEncoderDecoderOutputa Outputs of encoder-decoder generation models, when using beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`torch.FloatTensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Beam transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. beam_indices (`torch.LongTensor`, *optional*, returned when `output_scores=True`): Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer of the decoder) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size*num_beams*num_return_sequences, sequence_length, hidden_size)`. decoder_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, num_heads, generated_length, sequence_length)`. cross_attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. decoder_hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. past_key_values (`Cache`, *optional*, returned when `use_cache=True`): Returns the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. r`Nrzrarbr{rsrtrurvrwrV)rerfrgrhrirjrkrzrmrarlrbr{rsrtrurvrwrVr rnrorpr}r}s(T15e''$.5.2FE%## $t +2.2FE%## $t +2,0L%""T)0:>e//047>=A5!2!23d:AAEeE$5$567$>E?CeE%"3"345<CDH5u'8'8!9:TAH$(OUT\(ror}c$eZdZdZdZ d~dZ ddeejzdzde dzd e fd Z dddd e jd edzd edzde jdzde j dzde dzfdZddde j$dzde j$dzdeee j$fd ee j$edzeee j$fff dZddde j$dzde j$dzdeee j$fd e jf dZdZde j$dedeeefd e jfdZddde j$dedzded eeeff dZ ddddededeee j$fde j$de j8dzd ee jeee j$fffd Ze dd!ed"e d e jdzd ee jeeefffd#Z dd$e deeefd"e d%ed eeeff d&Z! dddded e jde j$d'e"deeefd(e#d)d*e#d+d,e#d+d e$fd-Z% dddded.edzd/e jdzd0e ee j$ge&efdzd'e"dzdedzdeeefdzd1e j$dzd2e j$dzd e"fd3Z' dddded4e(dzd5e#d+d e(f d6Z)d7e"e(zd8e"e(zd e"e(zfd9Z* dddd:e j$d;ee j$dZ+ d~d?Z,dddeeeffd@Z- d~dAZ. d~dBZ/dddedzdCed eeeffdDZ0dddEededFed ef dGZ1e2dHe3dd e fdIZ4dddededJe5dedKed e fdLZ6ddd e fdMZ7 ddddedNe dzde j8ezdzfdOZ8dddeeefded e fdPZ9e:d~dQZ; ddJe5de dRedzd edzfdSZ1` and `do_sample=False` - *beam-search multinomial sampling* if `num_beams>1` and `do_sample=True` - *assisted decoding* if `assistant_model` or `prompt_lookup_num_tokens` is passed to `.generate()` To learn more about decoding strategies refer to the [text generation strategies guide](../generation_strategies). )textselfrPc .|jr1|/|jj|j|_y|jrq|n||||| | | d| } t j |f||d||_t|dr6| r3 |j|fd| i|}tj|| |_ yyyyy#t $r8tjdt j |fd||dd||_YwxYw#t $rYywxYw) N) cache_dirforce_downloadproxieslocal_files_onlytokenrevision subfolder) _from_auto_from_pipelinezZGeneration config file not found, using a generation config created from the model config.z config.jsonT)config_file_namerr_from_model_configload_custom_generatetrust_remote_code)model) can_generategeneration_config from_dictto_dictr,from_pretrainedOSErrorloggerinfohasattrr functoolspartialgenerate)rrfrom_auto_class from_pipelinepretrained_model_name_or_pathrrrrrrrrkwargsrepo_loading_kwargscustom_generates rpadjust_generation_fnz$GenerationMixin.adjust_generation_fnrsm    #4#@%)%;%;%E%EFWF_F_Fa%bD "    %B%N&"0"$4$& # #  )9)I)I1*.#0** *&.t349J&?d&?&?5'IZ'^q'O%.$5$5oT$RDM :K4I&O &  p*:)I)I1*%2.#0'+ * * *& .s$!C1D>DD DDNrrreturnc t|fddi|}tjj |}d|d}t |||| |t |fdd i|td |}|S#t$rtd|dwxYw) at Loads and returns a custom generate function, given a model repo. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. trust_remote_code (`bool`, *optional*): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. **kwargs: Additional keyword arguments for remote code loading. Raises: OSError: If `pretrained_model_name_or_path` does not contain a `custom_generate` subdirectory. Returns: A callable that can be used to generate text. module_filezcustom_generate/generate.py`zw` does not contain a `custom_generate` subdirectory with a `generate.py` file, can't load the custom generate function.zThe repository `zS` contains custom generation code that will override the default `generate` method.)has_local_codehas_remote_code error_messagerequirements_filez custom_generate/requirements.txtr)rrospathexistsrrr)rrrrmodule is_local_codercustom_generate_functions rprz$GenerationMixin.load_custom_generates< +-;X\bF'DE <=>- -  "  )( --'   " ) =_ ci $7z6#J ''3 123OO  s A""A; input_idsnext_sequence_lengthrVattention_mask inputs_embedsis_first_iterationc i}|jjrdnd} |jjsR|P|rNd|| <||dd| dddfn|} | jtj|d<| j dd\} } nE| |dd| dfn|}|jtj|| <|j dd\} } |||d<|jjrdnd } |j | dx}||| <|j d dx}||d <| d d fD]V}|j|}||j d | k7s*|d | dfjtj}|||<X|jjr|nd}|jjrdnd}|jjr|j ddn|}t|tr|jr||jdk(rt|dt}||jtj| | df|j|j ||jd|j| |jd |jd |}||||<|||d<d}|j#D]\}}||vs ||vs|||<|j%rdt't)j*|j,j.vrRt0j3d||j5nd}tj6| |j |z}||d<|S)a_ Prepare the model inputs for generation. Notable steps include selecting the correct input key and cloning when appropriate, creating position_ids from the attention_mask when missing, slicing inputs and converting 2D attention masks to 4D for compilable caches, and finally forwarding all additional keyword arguments unchanged to the model's forward pass. See the forward pass in the model documentation for expected arguments (different models might have different requirements for e.g. `past_key_values`). This function should work as is for most LLMs. decoder_input_idsrN) memory_formatrr rVdecoder_position_ids position_idstoken_type_idsmm_token_type_ids.decoder_attention_maskrrrdtypedevice)configrrrVrrrr)labelsrcache_positiona9The remote code model you are currently using seems to expect `cache_position`. This arg has been removed from the Transformers library, and will stop being created in `generate` even for remote code models in a future release. Please open a PR on the remote code hub repo to remove any usage of `cache_position`.r)ris_encoder_decoderclonericontiguous_formatshapepopget isinstancer is_compileablendimgetattrremptyrritemsis_remote_codesetinspect signatureforward parametersr warning_onceget_seq_lengtharange)rrrrVrrrr model_inputs input_ids_key prompt_embeds batch_sizesequence_lengthposition_ids_keyrrmodel_input_name model_inputencoder_attention_maskattention_mask_keycausal_mask_creation_functionkwargs_to_avoid_forwardingkeyvaluepast_seen_tokensrs rpprepare_inputs_for_generationz-GenerationMixin.prepare_inputs_for_generations& 04{{/M/M+S^ {{---2KPb*.L '?S?_ a"6!6!7:;er -:,?,?eNeNe,?,fL )*7*=*=bq*A 'JAU@` !&:%:%;"; >L K-9L) *$jj)94@ @N M-;L) *"23CEX Y = &**+;++:X:XFJJ/ 6^l   ...*##q(-4D:UWp,q ):{{#kk:*JRVR\R\eneueuv- , 0 01B C)--.>?+//0@A"."2"23F"G#5 N  %/=L+ , ! --CL) *&H" ,,. *JC,&36P+P$) S! *    %5W=N=Nt||=\=g=g9h%h   }  DSC^==?de "\\/)BRBRSVffN-;L) *roinputs bos_token_id model_kwargsc |jjrFt|dr:|jj|jk7r|jj}n |j}|j |d}||t d|d|d|d|d ||}|dk(rd |vr|d |j d n|jjsd ttj|jjjv}|s#t d |jjd |j||| |d<|d d }}n| t d |d d }}|j|||}|||fS)zT This function extracts the model-specific `inputs` for generation. encoderNz `inputs`: z` were passed alongside z0 which is not allowed. Make sure to either pass z or z=...rrzAYou passed `inputs_embeds` to `.generate()`, but the model class z doesn't have its forwarding implemented. See the GPT2 implementation for an example (https://github.com/huggingface/transformers/pull/21405), and feel free to open a PR with it!)rzMYou passed `inputs_embeds` and `input_ids` to `.generate()`. Please pick one.)rrrrmain_input_namer ValueErrorrrrrrkeys __class__re*_maybe_initialize_input_ids_for_generation)rrrr input_name inputs_kwarghas_inputs_embeds_forwardings rp_prepare_model_inputsz%GenerationMixin._prepare_model_inputsZs KK * *i( ,,0D0DD55J--J$'' D9  #(:VH$#%%d&H&HITTYY[C0,4$[\`\j\j\s\s[tuxx-1,[,[L|-\- [)&2/%BO %$%tuu%1/%BO @@Wcdz<//roc||S|jd}t|dd}|jjrH|F|j dd}t j |t j|jdzSd}|jD]-}t|t js|jd}nd |vr_t j |dft j|jjd k7r|jS|d jS| td t j |dft j|j|zS) z3Initializes input ids for generation, if necessary.Nencoder_outputslast_hidden_staterrirrrmetazB`bos_token_id` has to be defined when no `input_ids` are provided.)rrrrsizerioneslongrvaluesrTensorrtyper) rrrrrrrrrs rprz:GenerationMixin._maybe_initialize_input_ids_for_generationsN  M&**+<=#O5H$O ;; ) ).?.K%**,Sb1E::e5::dkkJTQ Q !((* E%."[[^   l *::Qjj'+kk&6&6&&@t{{   GSSbFcFjFj    ab bzz:q/DKKPS___rocd|vr|djddkDr|d}|jd}|jdx}9|jjddz }|j |dk(d}|Sd}|jdx}|j }t j||zt j|j}|jd}|S)z Tries to infer position ids given attention mask and past kv cache length. All instances when `position_ids=None` should call this method. rrrrrrVr) rrrcumsum masked_fillrrirr unsqueeze)r inputs_tensorr seq_lengthrr past_lengthcaches rp$_prepare_position_ids_for_generationz4GenerationMixin._prepare_position_ids_for_generations , &< +D+J+J1+MPQ+Q(5M"((+ *../?@ @N M)..077;a?L'33Na4GKL K%))*;<<I#224  << [(@ [h[o[opL'11!4Lrorrc|j}|j}d|vr|djddkDr|d}tj|jddtj |j }||St|jdk(xr,|jtjtj fv}|s|S|duxr$tj||j}|duxs%tj||j} || z} |j|j } | | z|| zz} | S)Nrrrr r) _pad_token_tensor_eos_token_tensorrrirrrlenrintisinanyne) rrrr pad_token_id eos_token_iddefault_attention_mask is_input_idsis_pad_token_in_inputs&is_pad_token_not_equal_to_eos_token_idcan_infer_attention_maskattention_mask_from_paddingrs rp&_prepare_attention_mask_for_generationz6GenerationMixin._prepare_attention_mask_for_generationsa ):: (::  , &< +D+J+J1+MPQ+Q(5M"'M,?,?,C5::^k^r^r!s  ) )=../14g9L9LQVQZQZ\a\f\fPg9g ) )".d":!oM[gAhAlAlAn2>$2F2 JJ|\ 2 6 6 8L .$:zQGenerationMixin._prepare_encoder_decoder_kwargs_for_generation..sI!x**1-Isrroutput_attentionsoutput_hidden_states return_dictrrn) get_encoderrr r!rUrTrrrrrrrr,r-r) rrrrrrirrelevant_prefixr*rencoder_kwargsencoder_signatureencoder_accepts_wildcards ` rp._prepare_encoder_decoder_kwargs_for_generationz>GenerationMixin._prepare_encoder_decoder_kwargs_for_generationsy""$ 4 )w +26  /"7,7P7P &') s*!D9 D?,D?rdecoder_start_token_idrc|d|vr|jd}nd|vr|dk7r|jd}nd}| |j}|jdk(rC|jd|k7rt d|d|jd|j dd}n+t j|dft j| |z}||}||fSd |jjjvsI|jjd k(r5d |jjjjvr ||fS|jjd k(r ||fS|dddf|dddfk7jj!r\t j"||gd }d|vr?|d}t j"t j$|ddddf|fd }||d<||fS)zGPrepares `decoder_input_ids` for generation with encoder-decoder modelsNrrrrz1`decoder_start_token_id` expected to have length z but got rrdonutzvision-encoder-decoderwhisperdimr)rrrrrviewrirrrrelowerr model_typerallitemcat ones_like)rrrrr5rrrs rp)_prepare_decoder_input_ids_for_generationz9GenerationMixin._prepare_decoder_input_ids_for_generation&s(  #(;|(K , 0 01D E  L (-=-L , 0 0 =  $  >[[F ! & &! +%++A.*< G |S\]s]y]yz{]|\}~&<%@%@Q%G " J?%**VLOee #  $ 6 ,!,..%//557 7 KK " "&> >7dkkNaNaNlNlNrNrNtCt !,..[[ # #y 0 !,.. 1%)?1)EE J J L Q Q S % +ACT*U[] ^ '<7)56N)O&).__%;._expand_dict_for_generationns^% d!#&2z.QTBUW\WcWc7d*8*=*O*OP[ab*O*cN3' d" !rorr9rzMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)rFrr)rCrrrrHs` rp_expand_inputs_for_generationz-GenerationMixin._expand_inputs_for_generationas ! l* * "  !33KQ3GI2<@  12: !pqq.I,WhJi.jL* +,&&rooutputsnum_new_tokensc:tD] }||vs|dvrd}n|}t||||<n|jdx}&tj||dd| dfgd|d<|jdx}:tj||j |j d|fgd|d<|sd nd } |j| x} d g| jd z zdgz} tj|| j| j j| | d ddfzd z} tj| | gd} | || <|sdnd} |j| x}:tj||j|j d|fgd|| <|S)am Update the model kwargs to account for the `num_new_tokens` new tokens that were just generated. That is, update the `attention_mask`, `position_ids`, and `token_type_ids` to account for the new tokens of the total sequence. Note that this function never slices inputs, this is performed in `prepare_inputs_for_generation`. )rZrYrVrNrr9rrrrrr.rr) ALL_CACHE_NAMESrrrir@ new_zerosrr:rrrr;new_ones)rrJrrrKpossible_cache_name cache_namerrrr required_dimnext_position_idsrrs rp#_update_model_kwargs_for_generationz3GenerationMixin._update_model_kwargs_for_generations$3  "g-&*II!2J!4J+27I_(,,-=> >L K3,"2"2"4q"89RD@Lg ^<3E3ElNaNabggiuvsBCx()  !& <9J*KQS T ->L) *6H-Me*../AB BN O/4yy!8!8.:N:Nq:QSa9b!cdjl0L+ ,rologits_processorassistant_modelrQtarget_tokenizerrRassistant_tokenizerc 4td|||fD} |jt||||||} | S|jat |j |j|j xsd|j||jjj} | S| rtd|}td|}td|}|jdurctj|||jjj|d } d|j_t#||||||||| } | S|jd urt%|||||||| } | St'd t)|jj*t-||||||} | S)zU Returns the candidate generator to be used in `assisted_generation` c3$K|]}|du ywr&rn)r(vs rpr+z;GenerationMixin._get_candidate_generator..s"sQ1D="sN)rrVrrrrUr )rnum_output_tokensmax_matching_ngram_size max_lengthrU vocab_sizerQrRT)rVassistant_prune_lm_head) rrVrrrrUrWrXatm_translatorF)rrVrrrrUrWrXz7Invalid value for `do_sample`: expected a boolean, got )r>assistant_early_exitr#prompt_lookup_num_tokensr$rr^r_rget_text_configr`r do_samplerget_translatorrrepetition_penaltyr%r!rrrer ) rrrrrUrrVrWrXdifferent_tokenizerscandidate_generatorrbs rp_get_candidate_generatorz(GenerationMixin._get_candidate_generators #"s?L\^q:r"ss  1 1 ="=# $"3)+!1 # D#"u 7 7 C"@.@@"3"L"L(9(Q(Q(VUV,77!1;;668CC # r#"c""#4oFO#$=?OP "&'@BU"V  **d2!>!M!M$'KK//1<<$3,0 "HL11D&K'$3&7!-"/%5%5(;#1 '#F#"1#,,5&S'$3&7!-"/%5%5(; '#.#"!MdSdSnSnNoNxNxMyz#=# /"3)+!1 # #"roinput_ids_seq_lengthencoder_input_idsprefix_allowed_tokens_fnnegative_prompt_idsnegative_prompt_attention_maskc t} |g}|jB|jdk7r3| jt|j||| |j|j %| jt |j |jj|jdk7r[|?t|jdk(r'| jt|j|ntjdt|j4|jdk7r%| jt|j |j 3|j d kDr$| jt#|j |j$i|j$d kDrZ|>t|jdk(r&| jt'|j$|ntjd t|j(/| jt+|j(|j,|j.Mt1|d d@|j.d kDr1| jt3|j.|j,| |j4Nt1|d dA|j4d kDr2| jt7||j4|j,| |%| jt9||j:|j<$| jt?|j<|j@1| jtC|jD|j@| |jFdur| jtI|jJ0| jtM|jJ|j,||jN&| jtQ|jN| |jRA|} |dkDs |j<| n| dz} | jtU|jR| | |jW| |} |jXr|j:|j:dkDrwt[|j,t\rt|j,dz} nFt[|j,t^j`r|j,jd dz} nd} nd} |jb3|jbdk7r$| jte|jb|jf%| jti|jf|jj5|jjd k7r&| jtm|jj| |jn5|jndkr&| jtq|jn| |jr&| jtu|jr| |jv5|jvdkr&| jty|jv| |jz>d|jzcxkrdkr)nn&| jt}|jz| |j~?d|j~cxkrdkr*nn'| jt|j~| ||jM| j|jj|jjj||jdur| jt| S)z This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsProcessor`] instances used to modify the scores of the language model head. Nr)unconditional_idsunconditional_attention_maskr$ sequence_bias?r )penaltyrmzyPassing `encoder_repetition_penalty` requires some form of `input_ids` to be passed to `generate`, ignoring the argument.)rwrz{Passing `encoder_no_repeat_ngram_size` requires some form of `input_ids` to be passed to `generate`, ignoring the argument.rrT)top_h)top_kmin_tokens_to_keep)top_prz)min_prz)massrz)epsilonrz)rrzr)Hr8guidance_scaleappendrHr$rur@encoder_repetition_penaltyrrr0warningswarn UserWarningrhr?no_repeat_ngram_sizer=encoder_no_repeat_ngram_sizer/ bad_words_idsr<r min_lengthrr9min_new_tokensr:r> num_beamsforced_bos_token_idr4forced_eos_token_idr5r_remove_invalid_valuesr6 exponential_decay_length_penaltyr3suppress_tokensrBbegin_suppress_tokensrA_merge_criteria_processor_listrfrlistrir temperaturerCrxrDryrEr{rFr|r; typical_prGepsilon_cutoffr1 eta_cutoffr2watermarking_configconstruct_processorrrer`renormalize_logitsr7) rrrlrmrnrUrrrorp processors begin_indexrzs rp_get_logits_processorz%GenerationMixin._get_logits_processor s")*  #!   + + 7%44&91O/99    * * 6   9HYHgHgh i  8 8 D!<<C ,5F5L5L1MQR1R!!; 1 L L*; 9  / / ;@Q@d@dhk@k   >GXGkGkl m  1 1 =BSBhBhklBl   :;L;a;ab c  : : F!>>B ,5F5L5L1MQR1R!!7)FF) 9  * * 6   )%33%77   ( ( 4)+>EQ!,,q0   (%00%77!   , , 8)+>EQ!0014   1(%44%77!   $ /   0,%//   0 0 <   -%99   0 0 <   -%00%99!   2 2d :   9; <  = = I   -%FF%77(   , , 8   -%55!   2 2 >.K)1,0A0U0U0] 1_    4%;;! 88EUV   & &!**6;L;V;VYZ;Z/AA4H),->-P-P)QTU)U& 1 C CU\\R):)L)L)R)RST)UXY)Y&)*&%&"!,,8=N=Z=Z^a=a!!"9:K:W:W"XY &&2!!"29J9P9P"QR &&27H7N7NRS7S!!$+<+B+BWij!&&27H7N7NQT7T!!$+<+B+BWij!&&2!!$+<+B+BWij!**6;L;V;VY\;\!!'->-H-H]op!//;FWFfFf@lil@l!!' 1 @ @Ug !++7CBSB^B^1` at generate-time. normalize_logits (`bool`, *optional*, defaults to `False`): Whether to normalize the logits (which, for legacy reasons, may be unnormalized). Return: `torch.Tensor`: A `torch.Tensor` of shape `(batch_size*num_return_sequences, sequence_length)` containing the transition scores (logits) Examples: ```python >>> from transformers import GPT2Tokenizer, AutoModelForCausalLM >>> import numpy as np >>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2") >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> tokenizer.pad_token_id = tokenizer.eos_token_id >>> inputs = tokenizer(["Today is"], return_tensors="pt") >>> # Example 1: Print the scores for each token generated with Greedy Search >>> outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True) >>> transition_scores = model.compute_transition_scores( ... outputs.sequences, outputs.scores, normalize_logits=True ... ) >>> # input_length is the length of the input prompt for decoder-only models, like the GPT family, and 1 for >>> # encoder-decoder models, like BART or T5. >>> input_length = 1 if model.config.is_encoder_decoder else inputs.input_ids.shape[1] >>> generated_tokens = outputs.sequences[:, input_length:] >>> for tok, score in zip(generated_tokens[0], transition_scores[0]): ... # | token | token string | log probability | probability ... print(f"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}") | 262 | the | -1.414 | 24.33% | 1110 | day | -2.609 | 7.36% | 618 | when | -2.010 | 13.40% | 356 | we | -1.859 | 15.58% | 460 | can | -2.508 | 8.14% >>> # Example 2: Reconstruct the sequence scores from Beam Search >>> outputs = model.generate( ... **inputs, ... max_new_tokens=5, ... num_beams=4, ... num_return_sequences=4, ... return_dict_in_generate=True, ... output_scores=True, ... ) >>> transition_scores = model.compute_transition_scores( ... outputs.sequences, outputs.scores, outputs.beam_indices, normalize_logits=False ... ) >>> # If you sum the generated tokens' scores and apply the length penalty, you'll get the sequence scores. >>> # Tip 1: recomputing the scores is only guaranteed to match with `normalize_logits=False`. Depending on the >>> # use case, you might want to recompute it with `normalize_logits=True`. >>> # Tip 2: the output length does NOT include the input length >>> output_length = np.sum(transition_scores.numpy() < 0, axis=1) >>> length_penalty = model.generation_config.length_penalty >>> reconstructed_scores = transition_scores.sum(axis=1) / (output_length**length_penalty) >>> print(np.allclose(outputs.sequences_scores, reconstructed_scores)) True ```Nrrrr9)rirrr;torexpandrstackreshape transposerrer`r functional log_softmaxrsummaxrgather) rr`rar{rstacked_scoresbeam_indices_maskmax_beam_lengthbeam_sequence_indicescut_idxindicestransition_scoress rpcompute_transition_scoresz)GenerationMixin.compute_transition_scores7sh   <<q (:;@@QGJJ9K[K[\L'..r3v;?L(-{{6':'B'B3v;PR'S']']^_ab'c +33DKK//1<R>RSU>VN#XX00<<^QR<SN+33B8L8LR8PQN)1,05577<.s4 \`GDKK.'/:P:PRV2WW s7:zThe main model and the assistant don't have compatible encoder-dependent input shapes. Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper.zc(see https://huggingface.co/docs/transformers/en/generation_strategies#universal-assisted-decoding)rXz`assistant_tokenizer` is not required when the main and assistant models use the same tokenizer. Please omit `assistant_tokenizer` from `generate()` rz~The main and assistant models have different tokenizers. Please provide `tokenizer` and `assistant_tokenizer` to `generate()` )r- BEAM_SEARCHrASSISTED_GENERATIONnum_return_sequences _is_statefulrrerrrdirr>rer`) rgeneration_modergeneration_mode_kwargsattributes_to_checkr are_equal doc_referencerVs ` @rp_validate_generation_modez)GenerationMixin._validate_generation_modes n88 8ZKa=al  n@@ @ 559 /DDEQH  !YZ^ZhZhZqZqYrs 699:KL LO Y{{--o6L6L6_6_&f#8;O>Bkk>d>d>f=ghBB !1 2 !MM*;<00OOQ   s4#9#9#A#A#CsSWnrs   G?G$G0(//9&9  1 1X =59  2 &#fkkm*>LL 1215;;A>?  ++55d5C!.++ ' &D(CDDK{{--"kk99$9G%12C%DQ%G%M%Ma%P"/0, 2$++{?jMi?jk {{ KK   {{roclscld}djjvxstfd|DS)z{ Return `True` if current model can use a `DynamicCache` instance when initializing the `past_key_values`. )reformerminimaxxlnet olmohybridrwkvxlstm minimaxm2c3VK|] }|jjv"ywr&)rer<)r(unsupported_namers rpr+zBGenerationMixin._supports_default_dynamic_cache..s): =M CLL$6$6$8 8: s&))rer<r>)runsupported_model_namess` rp_supports_default_dynamic_cachez/GenerationMixin._supports_default_dynamic_caches@ # cll0022 c: Qh: 7  rormax_cache_lengthchd|jjjv}|sdnd}|j|}|7|jt d|dt |tr t dy|jdury|js0|j#tjd |jd y|tjtjfvr6|j#tjd |jd d |_i} tdt!|j"j%ddgxsgD} |jd k7s| r|j"j%d| d<|jdk(rd| d<|jt&vr|jt(vr*tjd|jdt*d|j-|jt/|j0|j2|z||||<n|jdk(r|j"j4s|js t d|j6 |j6ni} | j9d|j"j%d| j;dd} t=dd| i| ||<nt?di| ||<|j"j4r5||vr0t ||t@stA||t?di| ||<yyyy)z Prepares the cache for generation (if applicable), given `generate`'s parameterization. If a cache is instantiated, writes it to `model_kwargs`, under the name expected by the model. mambarVrWNzMPassing both `cache_implementation` (used to initialize certain caches) and `zB` (a Cache object) is unsupported. Please use only one of the two.z]Passing a tuple of `past_key_values` is not supported anymore. Please use a `Cache` instance.FzNThis model does not support `Cache` instances. `cache_implementation` (set to z) will be ignored.zRAn assistant model is provided, using a dynamic cache instead of a cache of type='z'. dynamic_fullc3$K|]}|dv yw))rconvlinear_attentionNrn)r(rs rpr+z@GenerationMixin._prepare_cache_for_generation..Es"  6 6" r\Tr layer_typesrrrzUsing `cache_implementation='zE' is deprecated and will be removed in v5.13. Please only use one of z9, and the layer structure will be inferred automatically.)rrrr quantizedzThis model does not support the quantized cache. If you want your model to support quantized cache, please open an issue and tag @zucchini-nlp.backendquantorn)!rrer<rrrrrlr$rrrr-rCONTRASTIVE_SEARCHrrrrer)r*r+r rrrr cache_config setdefaultrrr r) rrrrrris_linear_attn_cacherQuser_defined_cachedynamic_cache_kwargsis_linear_attentionr'r$s rp_prepare_cache_for_generationz-GenerationMixin._prepare_cache_for_generation s '$..*A*A*G*G*II.B& *--j9  ) 55A cdncopTT,e4 s   & &% / 335 55A##d(==>>PR  ~AA>CdCde e 55A##)>>?rC6D  2!!" dkk99$9GXZ[a_a"    1 1^ CGZ-1[[-H-HQU-H-V  *  1 1[ @15  .  1 15U U 559``##34E4Z4Z3[\LLhKijNN (,'A'A%6%K%K0::O=[=[=g,99moL  # #Hdkk.I.IRV.I.W X"&&y(;G'5'Vg'V'VL $(4'K6J'KL $ KK * *l*|J79LM':Z(434(L $N+ +rocdttj|jjj vS)z Return True if the current model supports the keyword argument `logits_to_keep` in forward() to save memory. Checking it in this way allows to avoid using a new model attribute. logits_to_keep)rrrrrr)rs rp_supports_logits_to_keepz(GenerationMixin._supports_logits_to_keepvs2  3w'8'8'F'Q'Q'V'V'X#YYYrokwargs_has_attention_maskc&d fd }||j|}||j|}||j|}||j|}jj r||n|}| |j dk(r|jd}|9|7||stjd|d}tjd|djj r | td|=tj||jr||stjd |Atj|s|dkjrtjd |d ||_||_||_||_y) a Prepares the special tokens for generation, overwriting the generation config with their processed versions converted to tensor. Note that `generation_config` is changed in place and stops being serializable after this method is called. That is no problem if called within `generate` (`generation_config` is a local copy that doesn't leave the function). However, if called outside `generate`, consider creating a copy of `generation_config` first. Nc||S||n j}t|tjr|j |Stj ||tj S)N)rr)rrrirrtensorr)rrrs rp_tensor_or_nonez@GenerationMixin._prepare_special_tokens.._tensor_or_nonesQ} %1Vt{{F%.xx''<<fEJJG GrorrzThe attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.z)Setting `pad_token_id` to `eos_token_id`:z for open-end generation.z\`decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation.zThe attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.z;`eos_token_id` should consist of positive integers, but is zq. Your generation will not stop until the maximum length is reached. Depending on other flags, it may even crash.r&)rrrr5rrrrrrrrirrris_floating_point_bos_token_tensorrr_decoder_start_token_tensor) rrr1rr5bos_token_tensoreos_token_tensorpad_token_tensordecoder_start_token_tensors ` rp_prepare_special_tokensz'GenerationMixin._prepare_special_tokens}s  H++<+I+IRXY*+<+I+IRXY*+<+I+IRXY%45F5]5]fl%m" ;; ) ).H.T*Zj '  ',<,A,AQ,F/99!<   #(8(D(4=Vq 02  NNFGWFXXqr s ;; ) ).H.Pn   'EJJ7GIY,Z,^,^,`(4=V##C  '  # #$4 5:JQ:N9S9S9U NNMN^M_`rr /?+.>+.>+8R5rocf|jry|jd|jd}|jjdvxs/t |j duxr|j j }|duxr|jxrt|tu}|xr|}t|dd||jjz}t|drGt|jj}d|vxrt|d kD}|| z}d |v} || z}|j |st j#d |rpd t$j&d <t)|j*rH|j <|j j,r&t j#dd|j _|S)zp Determines whether to trigger auto-compilation of the model's forward pass at generation time. FrVrW)cudaxpuneuronN hf_quantizerrcpurdiskzsYou have set `compile_config`, but we are unable to meet the criteria for compilation. Compilation will be skipped.0TOKENIZERS_PARALLELISMzWhen using Flash Attention and a static cache, you cannot use the option `CompileConfig(fullgraph=True)` as FA introduces graph breaks. We overrode the option with `fullgraph=False`.)disable_compilerrrboolcompile_config_compile_all_devicesrr rrBrrrrrrrrenvironrr fullgraph) rrrr valid_hardwareusing_compilable_cache can_compileall_model_deviceshas_cpu_offloadhas_disk_offloads rp_valid_auto_compile_criteriaz,GenerationMixin._valid_auto_compile_criterias  , ,  !2L4D4D^4TU))-FF $  , ,D 8 r=N=]=]=r=rK  "'d!2!ou7K7K!oPTUZP[coPo$?)?  4 . : 4,,;; ;K 4 ) #D$6$6$=$=$? @ #'88WSAR=SVW=WO . .K &)::  // /K  + + 7   #  36BJJ/ 0,DKK8$33?DUDdDdDnDn''eBG%44>roc#K|jj}|dk(r?|jjdk7r&tj d|j d d|dk(r,|jjdk7r|j |yyy#|dk(r,|jjdk7r|j |wwwxYww)N grouped_mmrCzWe will be switching to 'batched_mm' for the decoding stage as it is much more performant than 'grouped_mm' on smaller inputs. If you experience any issues with this, please open an issue on the Hugging Face Transformers GitHub repository. batched_mm)r_experts_implementationrrr info_onceset_experts_implementation)roriginal_experts_implementations rp_optimize_model_for_decodez*GenerationMixin._optimize_model_for_decodes*.++*M*M' +l :t{{?O?OSX?X   C   + +L 9 Q .,>4;;CSCSW\C\//0OPD]>.,>4;;CSCSW\C\//0OPD]>sAC B"2C 3CC rc(| dt|x}vrytj|jj ddj d|d|d|s9t |jj ddj d|d |S) zP Returns the Hub repo for a deprecated generation mode, if any. N/_ z6 was moved to a `custom_generate` repo: https://hf.co/zC. To prevent loss of backward compatibility, add `custom_generate='z*'` to your `generate` call before v4.62.0.zY requires `trust_remote_code=True` in your `generate` call, since it loads https://hf.co/r)GENERATION_MODES_MAPPINGrrnamereplacetitler)rrrrrepos rp_get_deprecated_gen_repoz(GenerationMixin._get_deprecated_gen_repos  &#>VWf>g6gd*h##++C5;;=>>tuytz{PPTvV6 6 !"''//S9??ABC004vQ8  roc|jdd|jdd||d}tjr(tjrtjnd}|t xs t |xr|dkDn||d<|jD cic] \}} |  ||  }}} t|trtjtjjj} tj|jj} | | z } | Dcic]}||vs||j|}}|Scc} }wcc}w)zn Extracts and returns the generation mode related keyword arguments from the provided kwargs. rNrX)rrXrVrr synced_gpus)rdist is_availableis_initializedget_world_sizerrrrrrrrr[rr) rrrrgrVrr world_sizekr[usual_mode_kwargscustom_generate_kwargsnew_custom_keyss rp_extract_generation_mode_kwargsz/GenerationMixin._extract_generation_mode_kwargs1sQ K6#)::.CT#J. "  /3.?.?.AdFYFYF[T((*ab "( ) I-CD-I ]z\]~ }- 4J3O3O3Q!c41aUVUb!Q$!c!c ox 0 ' 1 1/2I2I J U U Z Z \ %,%6%6%G%R%R%W%W%Y "47HHO@O%_1STX^S^aA&6%_ "%_%%"d&`s E)E2 E<ErgrrSc  ^| jdd} | tt| trdhd}tj Dcic] \}}||vs ||}}}|j | |j | fd| i| }|d5d|i|S| jddk(rtjd||n| jd}| td |jd k(r |jd j}n@|jd k(r|j}ntd |j|td||td||td||td|| td| | td| |tjd|| jdd }|d kDrtjd|d|jd5||j |fi| d d| }t#t%|Dcgc]'}|d|j&|d|j(z)}}t+j,|t*j.|j0}|jd }|S|j3| | |||}| jdduxr.|duxs|j4duxr|j6j4du}| jdduxr.|duxs|j8duxr|j6j8du}|j |fi| \}}|j;|}|j=|| | }t| t>r| }n|tAtC|tD|}|jG|jI|jK||||#tMjN|f||||||| | || d || S||n tQ}||n tS}dtUtWjX|jZj\j_v}|jdddu} |ja||jb|\}!}"}dtWjXj\j_vr|!|d<|!jdd }#|!j0}$|jg|| |$ |jhjjs|jl|#d kDrt%|!jdd k(r|jdd}%|%G|%jd|!jdk(r.t+jn|%ddd!fd k(jq}&n,t+jr|!ddd!f|jlk(d kD}&|&rtjd"|jhjjs |"d#k(rd$|_:| s/|jhjjs|r|jw|!|||d<n-| r+|"dk(r&t%|djdd kDr td%|jd&ddu}'d&tUtWjX|jZj\j_v}(|'s-|(r+|jhjjs|jy|!||d&<|jhjjrd'|vr|j{|!||"|}|jhjjr.|j}|#|"||j~|!j0(\})}n|"dk(r|!n|jd})|jd5|)t|j|j|jhjjd)|\})}|jr!|j|)|jd*})||j|)j|)jdd }*|j||||"|!|*+}|jr d,|vrd |d,<|j||*||j4d z }+|!jdd |*k7r-|"d#k(r(|jhjjs|+|!jdd z }+|j||||#|+|j0jB|)j0jBk7r`tjd-|)j0jBd.|j0jBd/|j0jBd0t|j||*|!|||!j0|| | 1 },|j|||jd*2}-|jt|d3<|||)f|,|-|d4||}.|.Scc}}wcc}w)6aV Generates sequences of token ids for models with a language modeling head. Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`. For an overview of generation strategies and code examples, check out the [following guide](../generation_strategies). Parameters: inputs (`torch.Tensor` of varying shape depending on the modality, *optional*): The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs` should be in the format of `input_ids`. For encoder-decoder models *inputs* can represent any of `input_ids`, `input_values`, `input_features`, or `pixel_values`. generation_config ([`~generation.GenerationConfig`], *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which has the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. stopping_criteria (`StoppingCriteriaList`, *optional*): Custom stopping criteria that complements the default stopping criteria built from arguments and a generation config. If a stopping criteria is passed that is already created with the arguments or a generation config an error is thrown. If your stopping criteria depends on the `scores` input, make sure you pass `return_dict_in_generate=True, output_scores=True` to `generate`. This feature is intended for advanced users. prefix_allowed_tokens_fn (`Callable[[int, torch.Tensor], list[int]]`, *optional*): If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments: the batch ID `batch_id` and `input_ids`. It has to return a list with the allowed tokens for the next generation step conditioned on the batch ID `batch_id` and the previously generated tokens `inputs_ids`. This argument is useful for constrained generation conditioned on the prefix, as described in [Autoregressive Entity Retrieval](https://huggingface.co/papers/2010.00904). synced_gpus (`bool`, *optional*): Whether to continue running the while loop until max_length. Unless overridden, this flag will be set to `True` if using `FullyShardedDataParallel` or DeepSpeed ZeRO Stage 3 with multiple GPUs to avoid deadlocking if one GPU finishes generating before other GPUs. Otherwise, defaults to `False`. assistant_model (`PreTrainedModel`, *optional*): An assistant model that can be used to accelerate generation. The assistant model must have the exact same tokenizer. The acceleration is achieved when forecasting candidate tokens with the assistant model is much faster than running generation with the model you're calling generate from. As such, the assistant model should be much smaller. streamer (`BaseStreamer`, *optional*): Streamer object that will be used to stream the generated sequences. Generated tokens are passed through `streamer.put(token_ids)` and the streamer is responsible for any further processing. negative_prompt_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The negative prompt needed for some processors such as CFG. The batch size must match the input batch size. This is an experimental feature, subject to breaking API changes in future versions. negative_prompt_attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Attention_mask for `negative_prompt_ids`. custom_generate (`str` or `Callable`, *optional*): One of the following: - `str` (Hugging Face Hub repository name): runs the custom `generate` function defined at `custom_generate/generate.py` in that repository instead of the standard `generate` method. The repository fully replaces the generation logic, and the return type may differ. - `str` (local repository path): same as above but from a local path, `trust_remote_code` not required. - `Callable`: `generate` will perform the usual input preparation steps, then call the provided callable to run the decoding loop. For more information, see [the docs](../../generation_strategies#custom-generation-methods). kwargs (`dict[str, Any]`, *optional*): Ad hoc parametrization of `generation_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*. Return: [`~utils.ModelOutput`] or `torch.LongTensor`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True` or when `config.return_dict_in_generate=True`) or a `torch.LongTensor`. If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible [`~utils.ModelOutput`] types are: - [`~generation.GenerateDecoderOnlyOutput`], - [`~generation.GenerateBeamDecoderOnlyOutput`] If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible [`~utils.ModelOutput`] types are: - [`~generation.GenerateEncoderDecoderOutput`], - [`~generation.GenerateBeamEncoderDecoderOutput`] rN>rrrrglobal_keys_to_excluderrpagedzDetected cache_implementation=paged: switching to continuous batching. You should consider using generate_batch directly instead.rz7inputs or input_ids must be provided for CB generation.rrr z5inputs must be a 1D or 2D tensor, got inputs.dim() = zTstopping_criteria is not supported for continuous batching. Got stopping_criteria = zbprefix_allowed_tokens_fn is not supported for continuous batching. Got prefix_allowed_tokens_fn = zPassistant_model is not supported for continuous batching. Got assistant_model = zCstreaming is not supported for continuous batching. Got streamer = zXnegative_prompt_ids is not supported for continuous batching. Got negative_prompt_ids = znnegative_prompt_attention_mask is not supported for continuous batching. Got negative_prompt_attention_mask = zFsynced_gpus is not ignored for continuous batching. Got synced_gpus = rzHnum_beams is not supported for continuous batching yet. Got num_beams = z. )rrreq_rr_r) rrrUrrnrVrorprrrrrrzA decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.rTz1`attention_mask` passed to `generate` must be 2D.rr)rrrr5r)rrCrr)rrrrrrr/z~You are calling .generate() with the `input_ids` being on a device type different than your model's device. `input_ids` is on z, whereas the model is on z. You may experience unexpected behaviors or slower generation. Please make sure that you have put `input_ids` to the correct device by calling for example input_ids = input_ids.to('z ') before running `.generate()`.) rrlrmrnrUrrrorp)rrrr$)rUrrrn)Qrrstrlocalsrrrrrrrr:rtolistNotImplementedErrorgenerate_batchrranger prompt_idsgenerated_tokensrir4rrrqr_rrget_generation_modererrrr`rrrrrr8rNrrrrrrrrrr=rrrrr?rr$rr r4rBr8rIrrr token_healing heal_tokensputrCrr0rr-rrrrr)/rrrrUrrnrgrVrrorprrrrsrrgenerate_argumentsrrrJir`sequences_as_tensorrrrrrdeprecated_mode_repodecoding_methodaccepts_attention_maskr1rrrrrhas_right_paddingkwargs_has_position_idsaccepts_position_idsrrrprepared_logits_processorprepared_stopping_criteriaresults/ rprzGenerationMixin.generateRs Z#JJ':DA  &:os+K& "@Fx~~?O!ueSV^tSt#u*!u !u  % %f -'@t'@'@(3D(HN( $,M$M:LM M ::, - 8 NN3  &1Vvzz+7NF~ !Z[[zz|q ))!,335" #Y&**,IZ![\\!,)kWhVlm(3)y^v]z{*)gUdThi#),pemdq*rss".)oYlXpq.9)FeCdGH &!hZeYijk ;2I1}!j^g]kkmno*d))"A$"A"ABS"^W]"^_`"aG `eehipeq_rZ[$qc #..4s1D1U1UUI #(,,y SWS^S^"_ "5"?"?"B & &"&!E!E      "  JJ| $ , :"d*R.?.J.Jd.R :&&11T9  JJ| $ , :"d*R.?.J.Jd.R :&&11T9  +J$*I*IJ[*f_e*f'<+??P#<<_N_apq ox 0-O ! )%d4j2J?2[\O ##L$5$5$78 &&8IKab +"++"3!1"3)A /$7/M 4"3) "0@/K+QdQf1B1N-ThTj!1S9J9J4<<9X9c9c9h9h9j5k!k$0$4$45Et$LTX$X!9=8R8R %22L9 5 ' g//@KKPPR R6C "? 3"((+ %% $$%68QZ`$a{{--!22>:PQ>VYZgZmZmVnrsVs".!1!12BD!I!-.2F2F-J]J]2](- .B2G12L(M(R(R(T%). -22FJ[JmJm2m(nqr(r%$NNp{{--2Bo2U*.  '(1O1OTj-1-X-X0,.L) *';.3|DT7U7[7[3\_`3` !TUU"."2"2>4"HPT"T-W5F5Ft||5T5_5_5d5d5f1gg&+? HfHf+/+T+TUbdp+qL ( ;; ) ).?|.SNN|-=?PL ;; ) )&*&T&T%!1)'8'T'T$++ 'U' #I|*:[)H lN^N^_jNkI#E$"D"D# -779J9_9_`#{{==#  #  <  * *((4J4N4N{4[\I   LL )%??1- ::/#9#9-'- ;   ( ( */?|/S-.L) * ''(9;KMcd -77!;    "&6 6 O3KK22  3 3A 6 6  ** |_jJZ  ;;  y//44 4 MM@@I@P@P@U@U?VW++**+,TTXT_T_TdTdSef* *  %)$>$>/!1+%=- ''% 3+I%? % !&*%@%@//,00=&A& "%6$?$? [!!   78/  %    S "v@s f$f$1,f*this_peer_finishedc|r_tj|rdnd|}tj|tjj |j dk(ryy|ryy)z Returns whether there are still unfinished sequences in the device. The existence of unfinished sequences is fed through `this_peer_finished`. ZeRO stage 3-friendly. r~rvr)opFT)rir4rh all_reduceReduceOpSUMr?)rrrgrthis_peer_finished_flags rp_has_unfinished_sequencesz)GenerationMixin._has_unfinished_sequences s^ ',ll:L3RU^d&e # OO3 8I8I J&++-4 roc tdjj}}tj }t d|}j |dDcgc]}|j}}|ddjj|j}tj||k(||}|jd k(r|S|ddd fj} jd 0j!jd d fd | D} n fd | D} t#t%| | D]\} \} } || }tj&||k(j)r5 |j+| Dcic]}j|fd}}t-|dk(ru|| fxxdz cc<|j/||dd } |jd k(rt-|||k7dk(r||d <|j1|j3d ||| <|Scc}wcc}w)a Generates sequences of token ids for models with a language modeling head. Parameters: input_ids (`torch.LongTensor`): The sequence used as a prompt for the generation. tokenizer (`PreTrainedTokenizerBase`, *optional*): The tokenizer used to decode the input ids. Return: `torch.LongTensor` where each sequence has its tail token replaced with its appropriate extension. Nzs When generating with token healing, you must pass the model's tokenizer to the `tokenizer` argument of `generate`.r)rrT)skip_special_tokenspt)return_tensorspaddingrrr_c3|K|]3}ttj|jd5yw)r_N)r rvdecoderb)r(t space_tokrs rpr+z.GenerationMixin.heal_tokens..4 s1bTUc9#3#3A#67??YObs9<c3\K|]#}ttj|%ywr&)r rvr)r(rrs rpr+z.GenerationMixin.heal_tokens..6 s#JAc9#3#3A#67Js),)prefixg$@rvrt)r)rrrr get_vocabr,rstriprrrriwherenumelrxconvert_tokens_to_idsconvert_ids_to_tokens enumeratezipr>r? extensionsrrrr)rrrrr vocab_trierr)promptstail_ids tail_toks batch_idxtail_idtail_tok batch_idsalt_tokseq_bias trimmed_idsrs ` @rprzGenerationMixin.heal_tokens s  *  &/%;%;Y=S=Sl #I$7$7$9: ,ALY'0&6&6yVZ&6&[\1779\\   )BBy''( KK \ 9<S  ??  ! QU#**,  * *3 / ;!77 8W8WX[8\]^_`IbYabIJJI.7Hi8P.Q" p *I*!),Iyyl2388:  R\QfQfnvQfQwFM009;TAH8}! gZ C '  $ $8 $ <#CR.K   "a'9Y,678A=". B#'==1F1Fq1I]n=#oIi E" pHy]Ds #IIc |j}|j} |j} |j} |j} |j } t d|D}|j}| r| rdnd}| r| rdnd}| r| rdnd}| r| rdnd}| r| rdnd}| rF|jjr0| rdjdnd}| rdjdnd}|jd}d}tj|tj|j }|j!|r|j#|j$n |j&}d}|j)|||j* }|j-|||j rR|rCd rd nd}|j.|fd|i}|j15|di|ddi}dddd}|j3|jj|r|r|j4dddddfj7dtj8|j}|||} | r| r|| fz }| r||fz }| rY||jjr |j:fn |j<fz }|jjr||j>fz }| r3||jjr |j@fn |jBfz }|rHtDjFjI| d}!tjJ|!d jMd }"ntjN| d}"|r|"|z|d |z zz}"tjP||"dddfgd}||jS|"jU||||z}|jWdk(}~|j-|||j rR||jY| rrd}#t fdtZDrt]fdtZD}$|$}#|jjrt_|||||||# Sta||||||#S|S#1swYxYw)a Generates sequences of token ids for models with a language modeling head using **multinomial sampling** and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. Parameters: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The sequence used as a prompt for the generation. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. stopping_criteria (`StoppingCriteriaList`): An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`] used to tell if the generation loop should stop. generation_config ([`~generation.GenerationConfig`]): The generation configuration to be used as parametrization of the decoding method. synced_gpus (`bool`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). streamer (`BaseStreamer`, *optional*): Streamer object that will be used to stream the generated sequences. Generated tokens are passed through `streamer.put(token_ids)` and the streamer is responsible for any further processing. model_kwargs: Additional model specific kwargs will be forwarded to the `forward` function of the model. If model is an encoder-decoder model the kwargs should include `encoder_outputs`. Return: [`~generation.GenerateDecoderOnlyOutput`], [`~generation.GenerateEncoderDecoderOutput`] or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a [`~generation.GenerateDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a [`~generation.GenerateEncoderDecoderOutput`] if `model.config.is_encoder_decoder=True`. c34K|]}t|dyw)rN)r)r(rs rpr+z*GenerationMixin._sample.. s'lh.(I'lsrnNrrcrdrFrrrr$rrr.Trrrrrr9 num_samplesc3&K|]}|v ywr&rnr( cache_keyrs rpr+z*GenerationMixin._sample.. N9 ,Nc3,K|] }|vs| ywr&rnrs rpr+z*GenerationMixin._sample..  iyy\hOh i  r`rarbrsrtrurvrwrVr`rarbrcrdrV)1rr,r- output_scores output_logitsreturn_dict_in_generaterrfrrrrrirrrrSget_compiled_callrI__call___prefillrrrr[rTrbrfloat32rurcrvrwrdr rsoftmax multinomialsqueezeargmaxr@rrCrendrMnextrrr_)%rrrUrrrgrrrr,r-rrrhas_eos_stopping_criteriarfra raw_logitsrurvrwrsrtrrunfinished_sequences model_forwardprefill_consumedrJrrnext_token_logitsnext_token_scoresprobs next_tokensr rs% ` rpr[zGenerationMixin._sample^ sV):: -??0EE)77 )77 "3"K"K$''lZk'l$l!%// 0M3 RD $;@QRX\"9>O2VZ'>CW^b #t{{'E'EVg.?!@!D!D\!Rmq H\ ./33ODbf " __Q' "$zz*EJJyO_O_`00?PQ  " "#4#C#C D  !--   #4#A#AA   ,,-?U^UeUe,f,8,Eq4$AtAA 4H LX  446N+MlMMGN# CC#';;#A#ADL 1!(q"ax 8 ; ;U]]clcscs ; t !1 [batch_size * num_beams, ...]rrr Nrrrir)r4rs rp_flatten_beam_dimz!GenerationMixin._flatten_beam_dim s>V\\"}}VeAhq&9%:U12Y%FGGrorcht|j}tj|||g|ddzS)z=[batch_size * num_beams, ...] -> [batch_size, num_beams, ...]rNr)r4rrrs rp_unflatten_beam_dimz#GenerationMixin._unflatten_beam_dim s3V\\"}}Vj)% Use a heuristic that assumes the best score comes from the current length minus the decoder prompt length. -> See detailed discussion: https://github.com/huggingface/transformers/pull/20901#issuecomment-1369845565 2. `early_stopping == "never"`: -> Estimate the best score using either `max_length` or `cur_len`, depending on the sign of `length_penalty`. -> A positive length penalty favors longer sequences, so we use `max_length` in that case. NOTE: the canonical beam search implementation can be replicated with `early_stopping="never"` and `length_penalty=0.0`, which are NOT the default flags. The default behavior was empirically found to produce better sequences (prior to 2022), and changing it is BC breaking. neverr~NrT)r:keepdimrer)rirrr) rrrrrr_rrrbest_hypothetical_lengthbest_possible_running_scoreworst_finished_scores rp_check_early_stop_heuristicz+GenerationMixin._check_early_stop_heuristic7 sJ W $#)='14F'F $'.1C'C $&9!RaR%&@D\^lDl&m#${{+;UYY{XYcg=hij=kmst2UYY '*> >BPT6   ro!next_token_hits_stopping_criteriactj|}tj||duz}tj|}||z|zS)zv Beam Search stopping condition -- halts the generation loop if any of these conditions becomes False T)rirr>)rrrrimprovement_possibleexists_open_beamvalid_continuationss rp%_beam_search_has_unfinished_sequencesz5GenerationMixin._beam_search_has_unfinished_sequencesf sZ %yy)LM#YY'78Nd!$($6$67KMf$g!!%!3!34EG`!a*,19q!W}-||JxGLLRQRSV__ !:\!IH^!!Q2D(D"DE57PPProrrrc||jtjdzz}tj||d}|j ||}|j ||} |j ||} || | fS)z Given the top-K continuations, their scores, and whether they hit a stopping criteria, select the best non-finished beams to continue beam search in the next iteration. rrr)rrirrr) rrrrrrtopk_running_log_probsnext_topk_indicesrrrs rp%_get_running_beams_for_next_iterationz5GenerationMixin._get_running_beams_for_next_iteration s"02S2V2VW\WdWd2ehn2n!n!JJ'=KAN ../EGXY"001GIZ[#112KM^_ "57KKKrotop_num_beam_maskc| | dddfz}|| dz| z |zz }tj|dd|duz}||jtjdzz }||jtjdzz }||dzz }tj||fd}tj||fd}tj||fd}tj||fd}tj || d}|j ||}|j ||}|j ||}|j ||}||||fS) z Updates the finished beams if (and only if) there are new completed sequences that have a higher score than the current finished sequences. NrrT)axiskeepdimsrr9r)rir>rrr@rr)rr`rrrr{rrrrrrrrrrdid_top_num_beams_just_finishedbeams_in_batch_are_fullmerged_sequences merged_scoresmerged_beam_indicesmerged_is_sent_finishedtopk_merged_indicess rp_update_finished_beamsz&GenerationMixin._update_finished_beams s2+LN_`dfg`gNh*h'(GaK:L,LQ_+_`"')),<2PT"UYgkoYo"p144U]]CfLL??CCEMMRU[[[ ;;vEE !99i1G%HaP ;"?QG #ii7P(QWXY"'))-=?^,_ef"g#jj)DQG&&'79LM ((8KL ))*=?RS --.EGZ[+|5EEEroc |j}|j}|j} |j} |j} |j } |j } |j}|j}|j}|j}|j}|j}|jdd\}}||z}|jjdk(r|j j"}n|jjdk(r|j%j&}nT|jjdk(r|j j(}n$|j j+j,}|}d}||jdnd}t/dd|z|z}t1j2t1j4|t0j6 t1j8||z t0j6 fd j;|j<}|j>}|r tAd | r| rd nd}| r| rd nd}| r| rd nd} | r| rd nd}!| r| rd nd}"| r| rd nd}#| rF|j jBr0| rd jEdnd}$| rd jEdnd}%| |xs|dnd}&t1jF|||f|&t0jH|j<}'|jK||||'ddddd|f<|'jMjO}(t1j8||ft0jP|j<})d|)ddddf<t1jF||fdt0jP|j<}*t1j8||ft0j6|j<}+t1j4|dft0j6|j<},t1j8||ft0j6|j<}-t1jF||||z fdt0jR|j<}.|.jMjO} |}/d}0|jU|||jV }1|jY|||j<r/|0rG|j[|'ddddd|f}/drdnd}2|j\|/fd|2i}3|d i|3ddi}1d}0|j_1|j jB|r|r|1j`dddddfj;dt0jb|j<}4tdjfji|4d }5||/|5}5| r| r||4jOfz }| r| r||5jOfz }| rY|!|j jBr |1jjfn |1jlfz }!|j jBr|"|1jnfz }"| r3|#|j jBr |1jpfn |1jrfz }#~1|jK|5||}5|5|)dddddfz}5t1jt|5|||zf}5|jw|5|'|.||||||| \}6}7}8||j[|7ddddd|dzf|}-|jK|-||}-|jy|6|7|8|-|\}'})}.|j{|(|7|*|6| |8|,|+|-||||||\}(}*} }+jEdR|j[|.d ||z f}9t}|d!r|jd|9d<ndj|9|dz}|j|,|)|*|+|||||" },|j|,|+|-| }|jY|||j<r/|j[|(ddd|ddf}(|j[|*ddd|f}*|j[| ddd|ddf} | dzj7jd j/}:||:z};|(ddd|;f}(| ddd|:f} | rz| sd}*d}.= rrc3,K|] }|vs| ywr&rnrs rpr+z/GenerationMixin._beam_search..> rr) r`rzrarbr{rsrtrurvrwrV)r`rzrarbr{rcrdrV)Irrr,r-rrrrfrrr_rrrrreraudio_vocab_sizeget_output_embeddings out_featuresoutput_vocab_sizerer`rrir@rrHzerosrr low_memoryrrrfullint64rdetachrfloatint32rrrrrrTrbrr rrrurcrvrwrdrr rrrr" reorder_cacherrrrrMrr}ry)>rrrUrrrgrrrr,r-rrrrfrrr_rrbatch_size_unflattenedrrr`rr n_eos_tokensrr sequential all_scoresrr{rurvrwrsrtoutput_fill_valuerr`rrrrrrflat_running_sequencesr model_outputsrrrb log_probsrrrbeam_idxmax_generated_length output_lengthr rs> ` rpr\zGenerationMixin._beam_search s Z):: (:: -??0EE)77 )77 "3"K"K%// *99*99&11 %// 0EE*3//"1*='+y8 >> " "&9 955J ^^ $ $(H H335BBJ ^^ $ $(; ;66J446AAJ$" 1=0H|))!,a Aq</09< !II ZZ5:: 6 ]U^E^glgqgq8r s  "Y    '11 t  4 RD 3 RD 5-rd $;@QRX\"9>O2VZ'>CW^b #t{{'E'EVg.?!@!D!D\!Rmq H\ ./33ODbf "@L?WL;LO]_!JJ J /(++##   -1,D,DYPZ\e,f!Q.)%,,.446 $kk:y*A]f]m]mn%)AqrE"jj*i!8TQVQ\Q\eneueuv !;; I'>ejjYbYiYij/4jj*aPUPZPZclcscs.t+-2KK  #5::i>N>N- ) %zz J$8 9bPUP[P[dmdtdt ,224::< !*    #4#A#AA & ,,-?U^UeUe,f)-)?)?@QRSUVX`Y`X`R`@a)b&,8,Eq4$AtAA* AU Ye  !% F| F F #  CC#';;#A#ADL 1#))!R(366D ^g^n^n6oF 11&b1AI()?KI' 6<<>"33J*}9??#4"66J$&;;99'99;+668& {{55(]-K-K,MM(');;99'<<>+99;)00J RI!$71d $CCI i*i*>T1UVIQUPmPm&/"3%9#5#+#%%Qn Q MN24M1B&&'=aMgPQkM>Q'RS1 -150H0H1:}1 - LPKuKu-'=*C2S# LvL H 24HFJE`E`#'='-)*C4W!12S"3##5--FaF BI{L2B. 12>112FsGVhLhGh2ij4!126:6I6I,WhJiks6tL!23 !23AA(KkG262R2R4W$7'!1%#5--3S 3 /&*%O%O3 1 &" O,,-?U^UeUe,f`**9Q8M9M8Mq5P+QR ,,[S?S>SUV;V.WX ".!1 7 7 9>>1>EIIK*-AA a-/0 #A'<(<'<$<= " " ENoNN iO ii $Y/{{--7'%0%%!-'9*?'9%5*?$)  5'%0%%!-1"7$)   roc f 34 ds td|jdvs t jdtur td|j |||||| | } |j } |j}|j}|j}|j}|j}|r|rdnd}|r|rdnd}|r|rdnd}|r|rdnd}|r|rdnd}|rF|jjr0|r d jd nd}|r d jd nd}|jdd \}}|d kDr tdtj |tj"|j$}d}d}|j'|||j$r|jd }| j)|\}} |j+|j$}| | j+|j$} |jd |jd z }!||d}"t-j, }#t/|#|jd |jj}#t1|#|jd }#|#jdx}$8|!dkDr3|!|$jdz}%t3|#|%|jj}#|s|!d znd}&|j4|f|&|d|#}'d|'vr|!d z|'d<|di|'}(|(j6dd|! d z dfj+tj8|j$33j;4t=|dkDr.8s#[AJq!Qw$7#[c36K|]}dd|ddfywr&rn)r(rrs rpr+z5GenerationMixin._assisted_decoding..:s'fq(9!Q'(B'frB)is_decoder_attention heuristicc3&K|]}|v ywr&rnrs rpr+z5GenerationMixin._assisted_decoding..lrrc3,K|] }|vs| ywr&rnrs rpr+z5GenerationMixin._assisted_decoding..mrrrr)Brrrrrrkrfr,r-rrrrrrrirrrrget_candidatesrrr&r(r'rrbrrrr{_speculative_samplingrrrrrrr@rrCrVcropupdate_candidate_strategyrTrl_split_model_outputsrvrurcrwrdrrrr rVrnum_assistant_tokens_schedulenum_assistant_tokensrrMrrrr_)5rrrUrrrgrrrVrXrrrjrfr,r-rrrrarrurvrwrsrtrrrrrcandidate_input_idscandidate_logitscandidate_lengthis_done_candidatecandidate_kwargsr new_lengthrrrJr valid_tokens n_matchesrselected_tokenscandidate_new_tokens new_cur_lennewly_added_lengthr rrArs5 ` @@rpr]z"GenerationMixin._assisted_decoding] stK(JK K  2 26\ \L$$%678KG\] ]";;/'+-& 3%<  &// -??0EE)77 )77 "3"K"K0M3 RD $;@QRX\"9>O2VZ'>CW^b #t{{'E'EVg.?!@!D!D\!Rmq H\ ./33ODbf " (oobq1 G >UV V$zz*EJJyO_O_`"!,,-?U^UeUe,fooa(G5H4V4VW`4a 1 !1"5"8"8"E +#3#6#6t{{#C 288;iooa>PP  12Et L  $yy6 6 "5";";A"> @^@^   77GI\IbIbcdIef  0 4 4^ DD QVfijVj- 0B0B20FF #89I:W[WbWbWuWu#v ?Q#3a#7W[ =4==#%9#5# L </1AA1E -.*\*G!,<+d}|jd}d}|jjs||rd}|jdx}|j} |r|djd| z }n^|jjrdnd} |j| } | 1|jd| jdk(r|jd| z }|j "|j |f||d |} |di| d diSd ttd j_ |j } tj|| d }d|vr td|j||r|j|jn |j}|j!dd} |j!dd}d} |D]d}| |jd z}| | ddd|f|d<||dd| |f|d<|j |fi|} |di| d di}|j"|d<|} f| |d<||d<S)a Perform the prefill stage of generation. Note that usually, the prefill stage is always the first iteration of a new input batch, and thus multimodal inputs etc should be treated as if it's the first iteration. However, for assisted decoding, assistants call `generate` several time in a row for a same batch of inputs, so we need to pass `is_first_iteration` here for such cases. NrFTrVrrrr?r.@_dynamorr9z+Cannot use prefill chunking without a cacherrrn)rrrrrprefill_chunk_sizerrricache_size_limitsplitrrSrrIrrrV)rrrrrrruse_inputs_embedsr r rrr chunk_size input_chunksrr input_chunkcurrent_lengthrJs rprzGenerationMixin._prefills$ $$((9 !{{---2KPb $ !%%&78 8E E..0K '3O'D'J'J1'MP['[$AEA_A_%=eu"!-!1!12D!E!-)//!2DH\H\]^H_2_+4??1+= +K(  / / 7=4==%9#5 L 9,9D9 9 ACGE9 % , , =*==J ;;y*"EL 4 !NOO44\CTU&&'8'G'GH]]  *--.>EN'++NDALK+ - !,{/@/@/D!D!-5CADV5WL!12+3?;~C]@]3^L0AtAA+^Q]^ 'I,IDI292I2I ./,  -.B])]+]"Dj]!++d2 ] #+C+>S +I"JT"Q ] .4 ]d ]38nt+]#\\D0]). t(;] ]F:> $)$+$0$6$56 $  $L#),@@#)+??# 3 3 #R-1!& z!)z!<<z!ell#z!llT) z!  z!  z!x.).`5%@5PTUXZ]U]P^5n,),\6!)6!pA/)A/+d2A/A/ % & A/F1)1AD1RU1fi1 1f T2M-N SW  $k)k+kk( k  k  k kZZ'BZtZ26,0 MS)MS+MS$($;MS s"T) MS^9)99=c3h9\l9 9vQQ*'+ ' t  t 0& c3h&BU]]_'+597;9=TX#'7;-137>B15b)b t#b,d2b.4 b 0$6 b #+C+>S +I"JT"Q bD[b""34b>*b#\\D0b). t(;bx$.b %** *bbH Dt]b]i]inr&]aS))S6>?X6YS   Sv"-1w)w##w.w0 w , w  w>*w !1!1 1wrH%,,H5<<HH JELLJcJcJV[VbVbJJ ell%,,5<<&, -2\\, "\\, \\,  ,, ,  ,  ,  , s , , , \M-2\\M,,M,1<<Ms MM,4Q$||4Q!<<4Q$ll 4Q  4Q  4Q4Q4Q4Q4Q4Q u||U\\5<<7 84QlL L!& L$)<< L ,1<< L  L u||U\\5<<7 8L,3F<<3F!& 3F\\ 3F  3F ll 3F$)<<3F.3\\3F ,,3F,1<<3F!<<3F3F3F 3F3F s !3F" u||U\\5<<E F#3Fz" [)[##[.[0 [ , [  [ e.. .[F "-1267;CG9=h)h##h.h0 h , h  h>*h((4/h""34h&&?@h56h !1!1 1h` $( U)U##U,U U ! Urorcd|dd| df}|jd}|ddtj||fjdd}|jd}|ddtj||fjdd} | |z } tj| } | | k} | j ddkj } |r| |k(r| dz} |ddd| dzf}|| fS|jd}|dd| ddf}| |krF|dd| ddf}tj||z d}|j|j n|}tj|djddddf}| dkDr&tj|ddd| f|fd}|| fS|}|| fS)a Applies sampling as in the speculative decoding paper (https://huggingface.co/papers/2211.17192, algorithm 1). Returns the selected tokens, as well as the number of candidate matches. NOTE: Unless otherwise stated, the variable names match those in the paper. Nrr9rr)rr) rrirr rand_likerrrclampdiv_rr@)rOrPrQrArRnew_candidate_input_idsqq_ir)p_iprobability_ratior_i is_acceptedrVrUgamma p_n_plus_1 q_n_plus_1p_primers rprIrIs2!6F5F5G2GH   R (A Au||,-/F F G O OPQST UCr"A Au||,-/F F G O OPQST UCc  //+ ,C**K,&&2&.2779IY*:: Q .q/IM//AB &  ""!!&&q)q)Q' u 1i?+Jkk: #:CG LL ' G   g1 5 = =a @q I q= 99&=a)m&La%PVXYL  ""L  ""roc*t|dk(r