Ti 2ddlmZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl mZddlmZddlmZddlZdd lmZdd lmZmZmZmZdd lmZmZmZmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3m4Z4ddl5m6Z6ddl7m8Z8m9Z9m:Z:m;Z;mm?Z?m@Z@mAZAddlBmCZCmDZDmEZEmFZFddlGmHZHeErddlImJZJmKZKmLZLmMZMmNZNejePZQ ddl#mRZRGddeeZTy#eS$rdZRYwxYw)) annotationsN)ABCabstractmethod) OrderedDict)Callable) nullcontext)partial)Any)nn) BatchSampler ConcatDataset DataLoader RandomSampler)EvalPredictionPreTrainedTokenizerBaseTrainerTrainerCallback)FeatureExtractionMixin)BaseImageProcessor) WandbCallback)ProcessorMixin)TRAINING_ARGS_NAME)EvalLoopOutput)BaseDataCollator) BaseEvaluatorSequentialEvaluator BaseModel)BaseModelCardCallbackBaseModelCardData)Router)DefaultBatchSamplerGroupByLabelBatchSamplerMultiDatasetDefaultBatchSamplerNoDuplicatesBatchSamplerProportionalBatchSamplerRoundRobinBatchSampler)BaseTrainingArguments BatchSamplersMultiDatasetBatchSamplers)disable_loggingfullnameis_datasets_availableis_training_available)deprecated_kwargs)Dataset DatasetDictIterableDatasetIterableDatasetDictValue)TrackioCallbackceZdZdZeZeZeZ e Z e Z ed d" d#fd Z d$ d%dZd&dZd$d'fd Zd(d Z d)d Zed*d Z d+ d,d Zd-d Zd$d.fd Z d/dZ d0 d1fd Z d0 d2fd Zd3dZd$d4dZ d5 d6dZ d7 d8dZ d9dZ!d:dZ"d$d;dZ#ddZ& d d?dZ' d@dZ( d$ dAdZ)e* d dBdZ+ dC dDd Z, d$ dEfd! Z-xZ.S)F BaseTrainerup BaseTrainer is a simple but feature-complete training and eval loop for PyTorch based on the 🤗 Transformers :class:`~transformers.Trainer`. This trainer integrates support for various :class:`transformers.TrainerCallback` subclasses, such as: - :class:`~transformers.integrations.WandbCallback` to automatically log training metrics to W&B if `wandb` is installed - :class:`~transformers.integrations.TensorBoardCallback` to log training metrics to TensorBoard if `tensorboard` is accessible. - :class:`~transformers.integrations.CodeCarbonCallback` to track the carbon emissions of your model during training if `codecarbon` is installed. - Note: These carbon emissions will be included in your automatically generated model card. See the Transformers `Callbacks `_ documentation for more information on the integrated callbacks and how to write your own callbacks. Args: model (:class:`~sentence_transformers.base.model.BaseModel`, *optional*): The model to train, evaluate or use for predictions. If not provided, a `model_init` must be passed. args (:class:`~sentence_transformers.base.training_args.BaseTrainingArguments`, *optional*): The arguments to tweak for training. Will default to a basic instance of :class:`~sentence_transformers.base.training_args.BaseTrainingArguments` with the `output_dir` set to a directory named *tmp_trainer* in the current directory if not provided. train_dataset (Union[:class:`datasets.Dataset`, :class:`datasets.DatasetDict`, :class:`datasets.IterableDataset`, Dict[str, :class:`datasets.Dataset`]], *optional*): The dataset to use for training. Must have a format accepted by your loss function. eval_dataset (Union[:class:`datasets.Dataset`, :class:`datasets.DatasetDict`, :class:`datasets.IterableDataset`, Dict[str, :class:`datasets.Dataset`]], *optional*): The dataset to use for evaluation. Must have a format accepted by your loss function. loss (Optional[Union[:class:`torch.nn.Module`, Dict[str, :class:`torch.nn.Module`], Callable[[:class:`~sentence_transformers.base.model.BaseModel`], :class:`torch.nn.Module`], Dict[str, Callable[[:class:`~sentence_transformers.base.model.BaseModel`]]]], *optional*): The loss function to use for training. Can either be a loss class instance, a dictionary mapping dataset names to loss class instances, a function that returns a loss class instance given a model, or a dictionary mapping dataset names to functions that return a loss class instance given a model. In practice, the latter two are primarily used for hyper-parameter optimization. evaluator (Union[:class:`~sentence_transformers.base.evaluation.BaseEvaluator`, List[:class:`~sentence_transformers.base.evaluation.BaseEvaluator`]], *optional*): The evaluator instance for useful evaluation metrics during training. You can use an ``evaluator`` with or without an ``eval_dataset``, and vice versa. Generally, the metrics that an ``evaluator`` returns are more useful than the loss value returned from the ``eval_dataset``. A list of evaluators will be wrapped in a :class:`~sentence_transformers.base.evaluation.SequentialEvaluator` to run them sequentially. callbacks (List of [:class:`transformers.TrainerCallback`], *optional*): A list of callbacks to customize the training loop. Will add those to the list of default callbacks detailed in [here](callback). If you want to remove one of the default callbacks used, use the [`Trainer.remove_callback`] method. optimizers (`Tuple[:class:`torch.optim.Optimizer`, :class:`torch.optim.lr_scheduler.LambdaLR`]`, *optional*, defaults to `(None, None)`): A tuple containing the optimizer and the scheduler to use. Will default to an instance of :class:`torch.optim.AdamW` on your model and a scheduler given by :func:`transformers.get_linear_schedule_with_warmup` controlled by `args`. Important attributes: - **model** -- Always points to the core model. If using a transformers model, it will be a [`PreTrainedModel`] subclass. - **model_wrapped** -- Always points to the most external model in case one or more other modules wrap the original model. This is the model that should be used for the forward pass. For example, under `DeepSpeed`, the inner model is wrapped in `DeepSpeed` and then again in `torch.nn.DistributedDataParallel`. If the inner model hasn't been wrapped, then `self.model_wrapped` is the same as `self.model`. - **is_model_parallel** -- Whether or not a model has been switched to a model parallel mode (different from data parallelism, this means some of the model layers are split on different GPUs). - **place_model_on_device** -- Whether or not to automatically place the model on the device - it will be set to `False` if model parallel or deepspeed is used, or if the default `TrainingArguments.place_model_on_device` is overridden to return `False` . - **is_in_train** -- Whether or not a model is currently running `train` (e.g. when `evaluate` is called while in `train`) processing_class) tokenizerNNc ts#td|jjd|Ed}tj d|j jd|d|j |}n8t||j s"tdt|j d |=| | |_ |j}nYtd |jjd | -tjd |jjd | |_ | -tjd |jjd|j dddij}|jr;|j j"s%|j j%|j|Bt'|dr6t|j(t*t,t.t0fr |j(}||j3|||}t5ddg||gD]\}}t|t6s|j8$t;t=|}t>dt@dtBdtDdi}|jGDcic]*\}}|tI|jKtM|d,}}}td|d|d|d|d t|tNrt|tPs tQ|}t|tNrt|tPs tQ|}|A|?|jRd!k7r0td"|jRd#|jjd$tT||jrdn|||||||nd%|| | | | | |& |jXd%k(rd|_,iid'|_-d|_.|||t_d(|j`jbDr tdjfjid)d*tjFt_d+|j`jbDr tdjfjid,d*||jm|jn}t|tNr|jGDcic]\}}||jq||c}}|_9t5ddg||gD]\}}| t|tNstd-|d.tu|jwtu|jwz x}s^td/|d0ty|d1t{|d2k(rd3nd4d5|d6 n|jq|||_9|t|t|s t|}||_@|j|j|d7|_A|jX|j|d7|_,|j|ycc}}wcc}}w)8Nz To train a z model, you need to install the `accelerate` and `datasets` modules. You can do so with the `train` extra: pip install -U "sentence-transformers[train]" tmp_trainerzNo `args` passed, using `z (output_dir=z)`.) output_dirzPlease pass an instance of `z` as the `args` argument.`z4` requires either a `model` or `model_init` argumentz` requires either a `model` or `model_init` argument, but not both. `model_init` will overwrite your model when calling the `train` method.z7`compute_metrics` is currently not compatible with the z. Please use the `evaluator` argument instead for detailed evaluation metrics, or the `eval_dataset` argument for the evaluation loss.unuseduse_configured_stateT)r=accelerator_config processor)modelargsr8trainevalstringint64float32boolnullzThe provided `z6_dataset` must have Features. Specify them with e.g.: z _dataset = z_dataset.cast(Features(z)) or by providing the Features to the IterableDataset initialization method. See the Datasets documentation for more information on dataset Features: https://huggingface.co/docs/datasets/en/about_dataset_featuresnoz%You have set `args.eval_strategy` to zu, but you didn't provide an `eval_dataset` or an `evaluator`. Either provide an `eval_dataset` or an `evaluator` to `z7`, or set `args.eval_strategy='no'` to skip evaluation.dummy) rCrD data_collator train_dataset eval_datasetr8 model_initcompute_metrics callbacks optimizersoptimizer_cls_and_kwargspreprocess_logits_for_metrics)rErFc3<K|]}t|tywN) isinstancer.0callbacks d/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py z'BaseTrainer.__init__..scxz(M2c WANDB_PROJECTzsentence-transformersc3<K|]}t|tywrX)rYr5rZs r]r^z'BaseTrainer.__init__..s/ 6>Jx 1/ r_TRACKIO_PROJECTz,If the provided `loss` is a dict, then the `z"_dataset` must be a `DatasetDict`.z:If the provided `loss` is a dict, then all keys from the `z;_dataset` dictionary must occur in `loss` also. Currently, z occursz in `z_dataset` but not in `loss`. dataset_name)Dr. RuntimeError model_class__name__loggerinfotraining_args_classrY ValueErrorr,rQcall_model_init __class__warningto_dict hub_model_idmodel_card_datamodel_id set_model_idhasattrrBrrrrget_data_collatorzipr2 column_namesnextiterstrintfloatrJitemsr4gettypedictr1 eval_strategysuper__init__rPaccum_loss_componentscan_return_lossanycallback_handlerrSosenviron setdefaultr5get_default_lossrC prepare_losslosssetkeyssortedlenrr evaluatorrOpreprocess_datasetadd_model_card_callback)selfrCrDrOrPrrrNr8rQrRrSrTrUrVr=default_args_dictrgdatasetsamplenaive_type_mappingkeyvalueexample_featuresloss_fnmissingrps r]rzBaseTrainer.__init__sJ4%&d..7789@@  <&J KK3D4L4L4U4U3VVbcmbnnqr s++z+BDD$":":;.x8P8P/Q.RRkl  =%",,,."Qt~~'>'>&??s#tuu%//01^^)DO  & NNI$..JaJaIbc'' !445KT4R5 ')    U%:%:%C%C  ! ! . .t/@/@ A  ${+"9;MOegu!v %    22T\l2mM%('6):]L" $    c4CXCXCbCbc c JJ ! !/3J K  &3/ BFBWBWBaBa/ ,  JJ ! !"35L M <((4D dD !fjfpfpfrsMb\[bt'8'8%'HHsDI),gv->P\@])^ % g?!'40$F|nTvw"',,.1C 4DDD7D$TUaTbc&&,Wo%6fCLTUDUS[]<^^cdpcqrNO ))$6DI  I})M+I6I"    )!%!8!8U\!8!]D     ( $ 7 7 SY 7 ZD  $$%67y$Fts /W9W?ct|jDcgc]}|jc}vr|js t d|j |j |j|jScc}w)as Load the data collator for the trainer. Args: model (:class:`~sentence_transformers.base.model.BaseModel`): The model to train, evaluate or use for predictions. args (:class:`~sentence_transformers.base.training_args.BaseTrainingArguments`): The arguments to tweak for training. processing_class (Union[:class:`transformers.PreTrainedTokenizerBase`, :class:`transformers.BaseImageProcessor`, :class:`transformers.FeatureExtractionMixin`, :class:`transformers.ProcessorMixin`], *optional*): The processing class to use for tokenization or image processing. Returns: :class:`BaseDataCollator`: The data collator to use for the trainer .. note:: This method can be overridden by subclassing the trainer to use a custom data collator. alYou are using a Router module in your model, but you did not provide a `router_mapping` in the training arguments. This means that the Router module will not be able to route the inputs to the correct submodules. Please provide a `router_mapping` that maps column names to routes, e.g. {'column_one': 'query', 'column_two': 'document', 'column_three': 'document'}.) preprocess_fnrouter_mappingprompts)r!childrenrprrndata_collator_class preprocessr)rrCrDr8modules r]rxzBaseTrainer.get_data_collator8sx6 U^^5EF6f&&F FtObObf ''**..LL(  GsA8c|j|}|j||j|j|j|j |j |y)a) Add a callback responsible for automatically tracking data required for the automatic model card generation This method is called in the ``__init__`` method of the trainer subclass. Args: default_args_dict (Dict[str, Any]): A dictionary of the default training arguments, so we can determine which arguments have been changed for the model card. .. note:: This method can be overridden by subclassing the trainer to remove/customize this callback in custom uses cases )rCtrainerN)model_card_callback_class add_callback on_init_endrDstatecontrolrC)rrmodel_card_callbacks r]rz#BaseTrainer.add_model_card_callbackasR#<<=NO -.'' 4::t||SWS]S]gk'lct||}t|ds|St|jt r|jj D]k\}}t|tjjs|||j|<@t|dsM|j|||j|<m|St|jtjjs|j ||_|St|jdr!|j|j||_|S)N)trialrrC) rrorwrYrrrtorchr Moduleoverride_model_in_loss)rrrCrrrps r]rozBaseTrainer.call_model_initts'e'4tV$L dii & $  1 Q W!'588??;%,U^DIIcNWg.%)%@%@%%PDIIcN  Q DIIuxx7 %(DI  TYY (33DIIuEDI rc ddlm}|jD]`\}}|dk(rt||r||_t|t j jsDt|||j||b|S)NrrrC) sentence_transformers.base.modelrnamed_childrenrYrCrr rsetattrr)rrrCrnamechilds r]rz"BaseTrainer.override_model_in_losssl>..0 OKD%w:eY#?" E588??3dD$?$?u$MN  O  rct|tjjr|j |j }n!||j |j }t |ddrbt|dtr.|djjDcgc]}|d }}n|dg}|D]}t|dsd|_ |Scc}w)Nrequires_media_countsFrtrack_media_countsT) rYrr rtodevicegetattrr! sub_modulesvaluesrwr)rrrCroute input_modulesrs r]rzBaseTrainer.prepare_losss dEHHOO ,775<<(D;>>%,,/D 40% 8%(F+7AEA[A[\iAjknAorwAw**=9#> x $44]C CAFaWZchcocoApD & &} 5g > ""=1':a?:rcd}d|vrd}nd|vrd}|rM|j}t|dr|j|j|}n1ddlm}||j||j j}d |vr|jd jj}|j|d xxdzcc<|jD]\}}|d k(r |dk(r|d |n|} t|j|z jd || <tjd |j|j |j||<|t" |I||St" |I|S)NrrE eval_lossrF_nested_gatherr) nested_gather) parallel_moder_rr)copyrwrrtransformers.trainer_pt_utilsrrDrrritemrroundrrrrrlog) rlogs start_timer accum_lossesrrrrlog_keyrps r]rzBaseTrainer.logs T>#M D "M 99;Dt-.#2243M3Mm3\] G,..}=TYYMdMd  ,&$((1557<<>**=9'BaGB"."4"4"6JCg~ :G6:Qq6WZG$)599;+>*D*D*F$JDMEJ\\5;;u||FD..}=cB   !7;tZ0 07;t$ $rc d}g}t}|D]}d}|D]'}|jd|zs|dt| }n|||vr8|j||j |j D cic]&\}} |j |s|t|d| (c} }|jdd} || fScc} }w)aTurn the inputs from the dataloader into the separate model inputs & the labels. Example:: >>> list(inputs.keys()) ['return_loss', 'label', 'sentence_0_input_ids', 'sentence_0_token_type_ids', 'sentence_0_attention_mask', 'sentence_1_input_ids', 'sentence_1_token_type_ids', 'sentence_1_attention_mask'] >>> features, labels = self.collect_features(inputs) >>> len(features) 2 >>> list(features[0].keys()) ['input_ids', 'token_type_ids', 'attention_mask'] >>> list(features[1].keys()) ['input_ids', 'token_type_ids', 'attention_mask'] >>> torch.equal(labels, inputs["label"]) True ) input_idssentence_embedding pixel_valuesinput_features input_valuespixel_values_videosNrlabel)rendswithraddappendr startswithr) rrfeature_suffixesr seen_prefixescolumnprefixsuffixrrrs r]rzBaseTrainer.collect_features#s(   tFF* ??3<0#Ns6{l3F ~=!8   f % OOr:3[^[i[ijp[qSV/6r s tGT*ss 8C C cj||j|d}n |j}t| |||S)NrFrf)rrPrevaluate)rrP ignore_keysmetric_key_prefixrps r]rzBaseTrainer.evaluateNs@  #22> !M    4+<+->-C-C-E(Fq(II$*! "88:[]PWP\P\@] ))..K& ggll;?  K$7 $ 4::;K;KSWS]S]SiSi!/!   +T2!,.? @ )..01 ]C>>%6$7q"9:BSBWBWX[B\!%6$7q">? ] /0 %  s ?BGG$c Jtjd|jjd|jjd |jjx}rC|j ddd}|j jjt| |j|jjy#t$rY2wxYw#t$rB}tjd|jjdt|Yd}~yd}~wwxYw) NzLoading best model from z (score: z).-rcz#Could not load the best model from z . Error: )rkrlrbest_model_checkpoint best_metricrsplitrCrtset_best_model_stepr~ Exception_load_from_checkpointerrorr})r checkpointstepexcs r]_load_best_modelzBaseTrainer._load_best_models .tzz/O/O.PPYZ^ZdZdZpZpYqqstu !ZZ===z=!((a04 **>>s4yI   & &tzz'G'G H       LL>tzz?_?_>``ijmnqjrist u  s+AC"%C CC D" 8DD"ct|tr,|jD]\}}|j||yt |j ddhzx}r"t d|r|dznddt|dy) Nrf return_lossrgz/The following column names are invalid in your  rez dataset: zH. Avoid using these column names, as they are reserved for internal use.)rYrrvalidate_column_namesrrzrnr)rrrgoverlaps r]r1z!BaseTrainer.validate_column_namess gt $)0 O% g**7*N O '../=.2QQ Q7 QAXd,QTBTjlAmmvw{}DxEwFFZZ  Rrc|||||d}tj|jjrAt |jjt r|jj|fi|St |jjr|jj|fi|St|tr=|jjtjk7rtjdy|jjtjk(r t|fi|S|jjtjk(rt|fddi|S|jjtj k(r t#|fi|S|jjtjk(rt t%||fi|Sy)aS Returns the appropriate batch sampler based on the ``batch_sampler`` argument in ``self.args``. This batch sampler class supports ``__len__`` and ``__iter__`` methods, and is used as the ``batch_sampler`` to create the :class:`torch.utils.data.DataLoader`. .. note:: Override this method to provide a custom batch sampler. Args: dataset (Dataset): The dataset to sample from. batch_size (int): Number of samples per batch. drop_last (bool): If True, drop the last incomplete batch if the dataset size is not divisible by the batch size. valid_label_columns (List[str]): List of column names to check for labels. The first column name from ``valid_label_columns`` found in the dataset will be used as the label column. generator (torch.Generator, optional): Optional random number generator for shuffling the indices. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. ) batch_size drop_lastvalid_label_columns generatorseedBWhen using an IterableDataset, you cannot specify a batch sampler.Nprecompute_hashesT)r7)inspectisclassrD batch_sampler issubclassr"callablerYr2r) BATCH_SAMPLERrkrq NO_DUPLICATESr%NO_DUPLICATES_HASHEDGROUP_BY_LABELr#r)rrr4r5r6r7r8batch_sampler_kwargss r]get_batch_samplerzBaseTrainer.get_batch_samplers>%"#6"   ??49922 3 499CZCZ\o8p*499**7K6JK K DII++ ,*499**7K6JK K g /yy&&-*E*EEcd 99 " "m&A&A A+GL7KL L 99 " "m&H&H H+GdtdOcd d 99 " "m&B&B B+GL7KL L 99 " "m&A&A A&}W 'RkVjk k Brc,|||d}tj|jjrAt |jjt r|jj|fi|St |jjr|jj|fi|S|jjtjk(r tdd|i|S|jjtjk(r tdd|i|Sy)a/ Returns the appropriate multi-dataset batch sampler based on the ``multi_dataset_batch_sampler`` argument in ``self.args``. This batch sampler class supports ``__len__`` and ``__iter__`` methods, and is used as the ``batch_sampler`` to create the :class:`torch.utils.data.DataLoader`. .. note:: Override this method to provide a custom multi-dataset batch sampler. Args: dataset (ConcatDataset): The concatenation of all datasets. batch_samplers (List[BatchSampler]): List of batch samplers for each dataset in the concatenated dataset. generator (torch.Generator, optional): Optional random number generator for shuffling the indices. seed (int, optional): Optional seed for the random number generator )batch_samplersr7r8rNr) r;r<rDmulti_dataset_batch_samplerr>r$r?r* ROUND_ROBINr' PROPORTIONALr&)rrrGr7r8multi_batch_sampler_kwargss r]get_multi_dataset_batch_samplerz+BaseTrainer.get_multi_dataset_batch_samplers.-"& " ??499@@ Aj II 1 13RG 949988_D^_ _ DII99 :849988_D^_ _ 99 0 04M4Y4Y Y)X'X=WX X 99 0 04M4Z4Z Z+ZGZ?YZ Z [rc |j}tj}|jj%|j |jj||jj |jj|jj|jjd}t|trg|j||jjd|jjtj k7ret"j%dnNt|t&r t)dt|t*r|j-D]}t|tst)d|j-Dcgc]6}|j/|||jj|j0|8}}t3|j-}|j5||||jj} | |d<n\t|t6r:|j/|||jj|j0|} | |d<nt)d |d |d t9|fi|Scc}w) a[Shared logic for building train/eval/test DataLoaders. Args: dataset: The dataset to build a DataLoader for. batch_size: The batch size to use. dataset_kind: A label for error messages, e.g. "train", "eval", or "test". Returns: A prepared DataLoader for the given dataset. ) collate_fn num_workers pin_memorypersistent_workersprefetch_factor)r4r5r9zcSentence Transformers is not compatible with IterableDatasetDict. Please use a DatasetDict instead.zYSentence Transformers is not compatible with a DatasetDict containing an IterableDataset.)r4r5r6r7)rrGr7r8r=z Unsupported `zC_dataset` type. Use a Dataset, DatasetDict, or IterableDataset for .)rNr GeneratorrDr8 manual_seeddataloader_num_workersdataloader_pin_memorydataloader_persistent_workersdataloader_prefetch_factorrYr2rdataloader_drop_lastr=r)r@rkrqr3rnr1rrEr6r rLr0r) rrr4 dataset_kindrNr7dataloader_params sub_datasetrGr=s r]_build_dataloaderzBaseTrainer._build_dataloadersX ** OO% 99>> %  ! !$)).. 1(99;;))99"&))"I"I#yyCC   g /  $ $",!%!?!?  yy&&-*E*EEcd !4 5u  -&~~/  k?;$s $+>>#3  &&)"ii<<(5(I(I' ' N $GNN$45G @@-#YY^^ AM 2? o .  ) 22%))88$1$E$E# 3M2? o . ~-pq}p~~A '7%677E s;Jc:|j#td|jjdd|j_|jj |j|j|jjd|_ |jS)a@ Returns the training [`~torch.utils.data.DataLoader`]. Will use no sampler if `train_dataset` does not implement `__len__`, a random sampler (adapted to distributed training if necessary) otherwise. Subclass and override this method if you want to inject some custom behavior. z4Training requires specifying a train_dataset to the rSFrEr[) rOrnrprj accelerator even_batchespreparer^rDtrain_batch_size_train_dataloader)rs r]get_train_dataloaderz BaseTrainer.get_train_dataloaderms    %STXTbTbTkTkSllmno o ).%!%!1!1!9!9  " "4#5#5tyy7Q7Q`g " h" %%%rcV|F|j:|j tgStd|jj d||n |j}d|j _|j j|j||jjdS)a Returns the evaluation [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior. Args: eval_dataset (`torch.utils.data.Dataset`, *optional*): If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. It must implement `__len__`. z6Evaluation requires specifying an eval_dataset to the rSTrFr`) rPrrrnrprjrarbrcr^rDeval_batch_size)rrPs r]get_eval_dataloaderzBaseTrainer.get_eval_dataloaders  D$5$5$=~~)!"~%UVZVdVdVmVmUnnopq q'3'?|TEVEV )-%''  " "<1J1JY_ " `  rcd|j_|jj|j||jj dS)a Returns the test [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior. Args: test_dataset (`torch.utils.data.Dataset`, *optional*): The test dataset to use. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. It must implement `__len__`. Ttestr`)rarbrcr^rDrh)r test_datasets r]get_test_dataloaderzBaseTrainer.get_test_dataloadersK)-%''  " "<1J1JY_ " `  rc.||n|jj}tj|dtj d|t |jdr2|jj||jjn|jj||j|jj|tj|jtjj|ty)NTrzSaving model checkpoint to save_safetensors)safe_serialization)rDr=rrrkrlrwrCsave_pretrainedror8rsaverrr)rr= state_dicts r]_savezBaseTrainer._saves#-#9Ztyy?S?S  J. 1*>? 4990 1 JJ & &zdiiF`F` & a JJ & &z 2  ,  ! ! 1 1* =  499bggll:7IJKrc|jj}|||jj}|jj|j y)N)trust_remote_code)rCrprvload_state_dictrs)rcheckpoint_pathri loaded_models r]r(z!BaseTrainer._load_from_checkpointsBjj** "?djjFbFbc  ""<#:#:#<=rct|ds||S|j|||j||j|jr|j |}d|_|S)aE Preprocess the dataset by optionally lazily adding a dataset name column, required for multi-dataset training with multiple losses or for dataset-specific router mappings. Args: dataset (DatasetDict | Dataset | None): The dataset to preprocess. If None, no preprocessing is done. dataset_name (str | None): The name of the dataset, used for multi-dataset training with multiple losses. Returns: DatasetDict | Dataset | None: The preprocessed dataset, perhaps with dataset names added as a lazy column. #_sentence_transformers_preprocessedrfT)rwr1#should_dataset_name_column_be_addedrDradd_dataset_name_columnr{)rrrgs r]rzBaseTrainer.preprocess_datasetsf 7A BgoN ""7"F  3 3GTYY R227;G7;3rcrt|ttfxrt|txs|jxrt|jtxsd|j xrVt|j txr:tt t|j jtS)a We should add a dataset name column to the dataset, if the dataset is a DatasetDict, *and* one of: a. The loss is a dictionary, or b. The prompts contain a mapping of dataset names, or c. The router_mapping contains a mapping of dataset names. ) rYr1r3rrrr{r|r)rrrDrs r]r|z/BaseTrainer.should_dataset_name_column_be_addeds'K1D#EF tT "  ?DLL$!? ##Ot22D9OtD)<)<)C)C)E$FGN  rct|ttfr0|jD]\}}|j ||||<|S||St|t r3|j t|jfd|i|j|St|trG|j}|rtd|d<|jt|j|d|}|Std)N)rrgrgrGrfT)batchedrz`Unsupported `dataset` type. Use a Dataset, DatasetDict, IterableDataset, or IterableDatasetDict.)rYr3r1rr}r0 set_transformr add_dataset_name_transform_format_kwargsr2rr4maprn)rrrg inner_datasetrs r]r}z#BaseTrainer.add_dataset_name_columns g 3[A B/6}} + m(,(D(D)!-)E) %  N  N gw '  ! !33!-,, 6% 1''H+0?(kk33!-! "Gr rc |r||}|rt|jdr||Stt|jd}|g|z|d<|S)aA transform/map function that adds the dataset name to the batch. Args: batch (dict[str, list[Any]]): The batch of data, where each key is a column name and each value is a list of values. dataset_name (str | None, optional): The name of this dataset, only if there are multiple datasets that use a different loss. Defaults to None. transform (Callable[[dict[str, list[Any]]], dict[str, list[Any]]], optional): An optional transform function to apply on the batch before adding the dataset name. Defaults to None. Returns: dict[str, list[Any]]: The "just-in-time" transformed batch with the dataset name added. rrg)rrr)batchrg transformkwargsr4s r]rz&BaseTrainer.add_dataset_name_transform.sd, e$ED03|7KLelln-a01 !- ;n rc  r|jsy|r%|jjj||r%|jjj ||r%|jjj ||jj |jj|y)N) model_name) is_world_process_zerorCrt set_language set_licenseadd_tags_create_model_cardrDr=) rlanguagelicensetagsrfinetuned_fromtasks dataset_tagsr dataset_argsrs r]create_model_cardzBaseTrainer.create_model_cardPs))+   JJ & & 3 3H =  JJ & & 2 27 ;  JJ & & / / 5 %%dii&:&:z%Rrc t|jtr)tjt |j}n |j}t |||\}}|j|}hdt|jzs|jDcgc]\}}||vs |js|c}}|jjd|jDcgc]\}}||vs |js|c}}ddg|d<|jj!D]\} } t|jhdz} | r| j#nd} |jDcic]\}}t%j&| |s||!} }}| rJ|| D]A}d|vs|dDcgc]'t)fd| j+Ds&)c}|d<Cnt-d| d| j!Dcic] \}}||vs ||}}}| j!Dcic] \}}||vs ||}}}|rC|| j/t1|j+| |jjd |sg|| j/t1|j+| dd ||fScc}}wcc}}wcc}}wcc}wcc}}wcc}}w) a5 We have to override the optimizer_grouped_parameters because the Trainer superclass bases it on the `model` itself, but the BaseModel losses can have weights that should be updated as well, e.g. SoftmaxLoss (see #2872). This method requires `transformers` >= 4.43.0. >rCparamsoptimizer_dict)r weight_decayrrrc3&K|]}|u ywrXr)r[paramps r]r^z;BaseTrainer.get_optimizer_cls_and_kwargs..s=sQVaun=ssz*No parameters found matching the pattern 'z^' in the model. Please check the pattern and ensure it matches some of the model's parameters.)rlrr)rYrrr Sequentialrrget_optimizer_cls_and_kwargsget_decay_parameter_namesrrnamed_parameters requires_gradrDrlearning_rate_mappingrrresearchallrrnrr)rrDrC loss_model optimizer_clsoptimizer_kwargsdecay_parametersnrparameter_pattern learning_rateoptimizer_param_keysoptimizer_param_keymatching_paramsgroupmatching_params_with_decaymatching_params_without_decayrps ` r]rz(BaseTrainer.get_optimizer_cls_and_kwargsis dii &{499'=>JJ*/'*NtU_*`' ' 99*E4s;K;P;P;R7SS'1&A&A&C"aM]H]bcbqbq%)II$:$: '1&A&A&C"aQaHafgfufu%(  2 - . 150J0J0P0P0R*  , }#&'7'<'<'>#?Bg#g @T"6":":"M>S>S>U,sdaYZbrYrQT,s ),s) !45<<"&'A'H'H'J"K+(, (>(>- !45<<"&'D'K'K'M"N+(+I* X...u r +*m,ssT' K4 KK2 K? K KK K?'K$'K$ K)#K)> K/ K/)NNNNNNNNNNNr:NN)rCBaseModel | NonerDzBaseTrainingArguments | NonerOCDataset | DatasetDict | IterableDataset | dict[str, Dataset] | NonerPrrznn.Module | dict[str, nn.Module] | Callable[[BaseModel], torch.nn.Module] | dict[str, Callable[[BaseModel], torch.nn.Module]] | Nonerz*BaseEvaluator | list[BaseEvaluator] | NonerNzBaseDataCollator | Noner8]PreTrainedTokenizerBase | BaseImageProcessor | FeatureExtractionMixin | ProcessorMixin | NonerQzCallable[[], BaseModel] | NonerRz'Callable[[EvalPrediction], dict] | NonerSzlist[TrainerCallback] | NonerTz?tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]rUz9tuple[type[torch.optim.Optimizer], dict[str, Any]] | NonerVz;Callable[[torch.Tensor, torch.Tensor], torch.Tensor] | NonereturnNonerX)rCrrDr(r8rrr)rzdict[str, Any]rr)rr)rtorch.nn.ModulerCrrr)rz8Callable[[BaseModel], torch.nn.Module] | torch.nn.ModulerCrrr)rCrrr)FN)rCrrdict[str, torch.Tensor | Any]rrJrz2torch.Tensor | tuple[torch.Tensor, dict[str, Any]])rzdict[str, torch.Tensor]rr)rdict[str, float]rz float | Nonerr)rrrz9tuple[list[dict[str, torch.Tensor]], torch.Tensor | None])NNrF)rPz#Dataset | dict[str, Dataset] | Nonerlist[str] | Noner r}rr) r rr r}r z bool | Nonerrr r}rr)rr)rr0rg str | Nonerr)NNr)rr0r4r~r5rJr6rr7torch.Generator | Noner8r~rzBatchSampler | None)Nr) rr rGzlist[BatchSampler]r7rr8z int | Nonerr )r'Dataset | DatasetDict | IterableDatasetr4r~r[r}rr)rr)rPz.Dataset | DatasetDict | IterableDataset | Nonerr)rlrrr)r=rrr)rxr}rr)rDatasetDict | Dataset | Nonergrrr)rrrDr(rz nn.Module | dict[str, nn.Module]rrJ)rz=DatasetDict | IterableDatasetDict | Dataset | IterableDatasetrgrrr)rdict[str, list[Any]]rgrrz=Callable[[dict[str, list[Any]]], dict[str, list[Any]]] | Nonerr) NNNNNNNNN)rrrrrstr | list[str] | Nonerrrrrrrrrrrrrr)rDr(rCrrztuple[Any, Any])/rj __module__ __qualname____doc__rrir model_card_data_classrrrrr(rmr/rrxrrorrrrrrrrrrr-r1rErLr^rfrirmrtr(rr|r} staticmethodrrr __classcell__)rps@r]r7r77s@DK- 5*/!34#'-1]a\` @D15 59CG26Vb^bei/u8u8+u8[ u8 Z u8  u8>u8/u8u8$3%u8&A'u8(0)u8*T+u8,#\-u8.(c/u80 1u85u8~' ' $'  '  ' Rm&2F  ,   % 22.2 2 < 2h@*'%R) 3) B) Z=A(,!' N9 N& N N  N -1(,!' ///* / & /  / /b" "15,0?l?l?l ?l . ?l * ?l?l ?lJ-1 *[*[+*[* *[  *[  *[XV88V8V8 V8  V8p&* 6 &L(> X\3JT %< - $ /    4$(.N.!. & .`$(SW# Q  F $"'+!%%)(,/3*./3SSS% S  S # S&S-S(S-S S4FJQ/)Q/2BQ/ Q/Q/rr7)U __future__rr;rrrabcrr collectionsrcollections.abcr contextlibr functoolsr typingr rr torch.utils.datar r rr transformersrrrr%transformers.feature_extraction_utilsr#transformers.image_processing_utilsrtransformers.integrationsrtransformers.processing_utilsrtransformers.trainerrtransformers.trainer_utilsr(sentence_transformers.base.data_collatorr%sentence_transformers.base.evaluationrrrr%sentence_transformers.base.model_cardrr "sentence_transformers.base.modulesr!"sentence_transformers.base.samplerr"r#r$r%r&r'(sentence_transformers.base.training_argsr(r)r*sentence_transformers.utilr+r,r-r.%sentence_transformers.util.decoratorsr/datasetsr0r1r2r3r4 getLoggerrjrkr5 ImportErrorr7rrr]rs" ##$" SSZZHB3835ET6Z5utnnCZZ   8 $9 C/'3C/ Os9D DD