TiwjzdZddlmZddlZddlZddlmZddlmZm Z ddl m Z ddl Z ddlZddlmZmZmZmZ ddlmZ dd lmZmZej8eZd d d d d Zdd Z dd Z!ddZ"ddZ#ddZ$ddZ%GddZ& d ddZ' d ddZ(ddZ)y#e$rdZYhwxYw#eef$rdZdZYpwxYw)zAModality detection, input parsing, and message format conversion.) annotationsN) defaultdict)AnyLiteral)urlparse) MessageFormatModality PairInput SingleInput)Image) AudioDecoder VideoDecoderflat)apertus deepseek_v3gpt_ossseed_ossc0|jdxrd|vS)zQCheck if a string looks like a valid URL (starts with http(s) and has no spaces).)zhttp://zhttps:// ) startswithtexts e/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/sentence_transformers/base/modality.py_looks_like_urlr,s ??2 3 G4Gct|r4t|jj}|j |S|jj |xrt jj |S)zOCheck if a string is a URL or local file path with one of the given extensions.)rrpathlowerendswithosisfile)r extensionsrs r_is_media_url_or_pathr#1sZt~""((*}}Z(( ::<  , E1EErc>|jdryt|dS)z:Check if a string is an image URL, file path, or data URI.z data:image/T)z.jpgz.jpegz.pngz.gifz.bmpz.tiffz.webp)rr#rs ris_image_url_or_pathr%9s }% 'b ccrcdt|dryt|xrt|jdvS)z.Check if a string is a video URL or file path.)z.mp4z.aviz.movz.wmvz.flvz.mkvT)zwww.youtube.comz youtube.comzyoutu.bez m.youtube.com)r#rrnetlocrs ris_video_url_or_pathr(@s8T#ST 4  Xd^%:%:?&rct|dS)z/Check if a string is an audio URL or file path.)z.mp3z.wavz.oggz.flacz.aac)r#rs ris_audio_url_or_pathr*Ls 'P QQrc0t|ttfrt|dk7ryt|dtrt|dtry|D]E}t|t r d|vrd|vryt|ts.|s1t|dt sEyy)aCheck if a sample is a non-text pair (2-element tuple/list with at least one non-string element). Text pairs ``(str, str)`` are handled natively by tokenizers and detected as ``"text"`` modality by :func:`infer_modality`. This helper detects pairs that contain at least one non-string element (e.g. an image, audio array, or dict), which require conversion to message format. FrrolecontentT) isinstancetuplelistlenstrdict)sampleelems r_is_non_text_pairr8Qs fudm ,F q0@&)S!jC&@ dD !fnd9J dD !dz$q'4/H  rceZdZdZ d ddZddZ ddZddZdddZ ddZ e dd Z dd Z dd Z dd Zy)InputFormatteraHandles input parsing, modality detection, and message format conversion. This class manages the complete input preprocessing pipeline: 1. Parsing raw inputs to detect their modality (text, image, audio, video, message) 2. Converting inputs to different chat template formats 3. Normalizing mixed-modality inputs Different models require different message/chat template formats: - **Structured format**: Content is a list of dicts with type annotations [{"role": "user", "content": [{"type": "text", "text": "hello"}]}] - **Flat format**: Content is the direct value [{"role": "user", "content": "hello"}] Args: model_type: The model type string (e.g. from ``config.model_type``). message_format: Message format to use. Options: - ``"structured"``: Content is a list of dicts with type/modality keys - ``"flat"``: Content is the direct value - ``"auto"``: Automatically infer from processor (default) processor: Optional processor to infer format from when ``message_format="auto"``. supported_modalities: Optional list of modalities supported by the model. When provided, string inputs that look like media URLs/paths are only classified as non-text if the model actually supports that modality. This prevents text-only models from misclassifying text containing media URLs. Nc||_||_||_|dk(r!|r|j||_yd|_y||_y)Nauto structured) model_type processorsupported_modalities _infer_formatmessage_format)selfr>rBr?r@s r__init__zInputFormatter.__init__sG%"$8! V #CL$"4"4Y"?D R^D "0D rc|jtvrt|jSt|ddttrsygd}t fd|Dryy)aInfer the message format expected by the processor. Checks known model types first, then inspects the processor's chat template for patterns indicating structured format. Defaults to ``"structured"`` if neither approach is conclusive. Args: processor: The processor/tokenizer to inspect. Returns: ``"structured"`` or ``"flat"`` message format. chat_templateNr=)z content[0]z.typez'type'z"type"z item.typezmessage.content[c3&K|]}|v ywN).0patterntemplates r z/InputFormatter._infer_format..sFww("Fsr)r>!KNOWN_MODEL_TYPES_MESSAGE_FORMATSgetattrr0r4any)rCr?structured_patternsrLs @rrAzInputFormatter._infer_formats] ??? ?4T__E E9ot<(C(  F2EF Frc |sddgittfSg}tt}d}|D]}t|r|jd|fd}%t ||j }|dk(r"t |tr|d}|d|dd<n|dk(rdt^t |trN|j}|jjd j}|j|dd<n|d k(r=t |tr-|d}|d jd gj|d n|d k(rtt |trt|} |j!d | } | j}|d jd gj|j"j$|j"j&t)t+| jj,d d n|dk(rt |tr|g}n|}|j||f|rg} |D]\} }| dk(r!| j|j/|,| dk(rEt |t0t(fr/t|dk(r!| j|j/|v| dk(r| j|t | t0r|n| |i} | j|j3| dd| i|fSt5|\}}t)|}t7|}t|dk(r\|j9}t |t:r||i}n|d j=}|D cic]} | |Dcgc]}||  c}}} }nUt>jAd|dd|Dcgc]*\}}|j3t |t0r|n||i,c}}i}d}|||fScc}wcc}} wcc}}w)a-Parse inputs and group by modality. Analyzes a list of inputs to detect their modality (text, image, audio, video, message) and groups them appropriately for the processor. Handles mixed modalities by converting to message format when necessary. Non-text pairs (e.g. ``(image, text)`` or ``(image, image)``) are detected and converted to message format with ``"query"``/``"document"`` roles via :meth:`pair_to_messages`. Args: inputs: List of inputs to parse. Can be: - str: Text inputs - tuple/list of str: Text pairs (for cross-encoders) - tuple/list of mixed types: Non-text pairs (e.g. image + text) - dict: Chat messages, audio data, or multimodal inputs - PIL.Image.Image: Image inputs - np.ndarray/torch.Tensor: Audio (1-2D) or video (3-5D) inputs Returns: Tuple of (modality, processor_inputs_dict, extra_modality_kwargs) where: - modality: Detected modality string (``"text"``, ``"image"``, etc.) or tuple of modalities - processor_inputs_dict: Dictionary mapping modality names to input lists - extra_modality_kwargs: Extra kwargs per modality (e.g. ``sampling_rate`` for audio) rFpairTr@audioarray sampling_rater)dimvideovideo_metadata)fpstotal_num_framesframes_indicesmessager,r-zMixed modalities detected: z!. Converting to 'message' format.)!rr5r8appendinfer_modalityr@r0r get_all_samplesdatameannumpy sample_rate setdefaultrr3get_frames_in_rangemetadata average_fps num_framesr2rangeshapepair_to_messagesr1 to_messagezipsetpopr4keysloggerdebug)rCinputs typed_inputsextra_modality_kwargs has_pairsitemmodalityvaluesamplesrj frame_batchmessagesmodtyped modalitiesprocessed_inputsunique_modalities ordered_keysentrys r parse_inputszInputFormatter.parse_inputss8FB<T):: :EG +D 1 ( 3D!&##VTN3 %dAZAZ[H7"z$'=W BFBW%g.?W$)AjQUWcFd..0 ))a)0668BIBUBU%g.?W$D$)?W %g.99:JBOVVW[\lWmnW$)AjQUWcFd Y "66q*E #((%g.99:JBOVV#}}88,0MM,D,D*.u[5E5E5K5KA5N/O*PY&:dD+A   5 1 2Q( 3Z H* < U&=OOD$9$9%$@AF]z%%'GCPUJZ[OOOD$9$9%$@AI%OOE*%/U%;E#uEOODOOE$:; <y(35JJ J'*L'9$ $ 01 O  !Q &(,,.H(C($,.>#?  02779 _k#lX[CBR)S%*)S$S#l #l LL67H6IIjk l+7'%OOZ%-HExY^N_`   !H)+@@@*T#ls' O%1 O =O%%/O+ O%c|\}}t|}t|}|jdk(r d|dd|dgSd}d|||dd|||dgS)aConvert a pair of inputs to query/document message format. Each element of the pair is wrapped in a message with role ``"query"`` (first element) or ``"document"`` (second element). The modality of each element is inferred individually via :func:`infer_modality`. Args: pair: A 2-element tuple or list of inputs (e.g. ``(image, text)``). Returns: List of two message dictionaries with ``"query"`` and ``"document"`` roles. rqueryr.r/documentct|tr,t|tr|Dcgc]}||vsd||||ic}Sd|||igScc}w)Ntype)r0r1r5)rzryrs r _to_contentz4InputFormatter.pair_to_messages.._to_content@sT(E*z$/EAIY#STX[c495YYXx67 7Zs A A)r`rB)rCrS query_itemdoc_itemquery_modality doc_modalityrs rrmzInputFormatter.pair_to_messages)sy $ H' 3%h/   & ( Z8#9   8^Z)P QK h,O P  rc |jdk(rNt|dk(r+tt|j \}}||dgSt j d||j Dcgc] \}}d|||i c}}dgScc}}w)aConvert a typed input dictionary to message format. Produces a single message with the given ``role``. For pair/multi-value inputs, use :meth:`pair_to_messages` instead (which is called automatically by :meth:`parse_inputs`). Args: typed_input: Dictionary mapping modality to input value (single value per modality). role: Role for the message (default: ``"user"``). Returns: List of message dictionaries (single message). rr-rzbFlat message format requested but multiple modalities detected. Falling back to structured format.r)rBr3nextiteritemsrswarning)rC typed_inputr._r{rzs rrnzInputFormatter.to_messageLs   & (;1$[%6%6%8 9:5!%%8999WbWhWhWjkOHeVXx?k   ls2B cP|sddgifSt|tr|fn|}ttt |j }g}t |D]}|Dcic]}||vs||||}}t|dk(rytt |j } t| ttfrAt| dk(r3td| Dr!|j|j| |j|j|dd|ifScc}w)acConvert a batch of modality-specific inputs into the unified message format. Args: modality: The modality key (string) or tuple of modality keys. processor_inputs: Dictionary mapping modality names to lists of inputs. Returns: Tuple of ``("message", {"message": [messages_per_sample, ...]})`` r^r-r,c3<K|]}t|tywrH)r0r4)rJvs rrMz2InputFormatter.batch_to_message..sOrghPZ[\^aPbOrs) r0r4r3rrvaluesrkr1r2allr_rmrn) rCrzprocessor_inputsr batch_sizer~irrr{s rbatch_to_messagezInputFormatter.batch_to_messagejs y"o- -$.x$=h[8 d#3#:#:#<=>? z" :ADTjSX[_iXi3 0 5a 88jKj;1$T+"4"4"678eeT]3E aCOrlqOrLrOOD$9$9%$@A OODOOK8 9 :9h///ks  D#' D#c|D]S}|D]L}|jd}t|tr%t|trt d|DsHyyUy)aCheck whether all messages in a batch contain only text content. Works with both flat format (``{"content": "hello"}``) and structured format (``{"content": [{"type": "text", "text": "hello"}]}``). Args: messages_batch: List of message lists, one per sample. Returns: True if every message contains only text, False if any contain non-text content. r/c3FK|]}|jdddk7yw)rrN)get)rJrys rrMz7InputFormatter.is_text_only_messages..s R$488FF3v=Rs!FT)rr0r4r2rP)messages_batchr~r^r/s ris_text_only_messagesz$InputFormatter.is_text_only_messagess`' !H# !!++i0gs+gt,R'RR$  ! !rcpg}|D]-}d|vsd|vrtjd|d&|d}t|txr|xrt|dt}|j dk(ri|rgt |dk(r%d|dvr|ji|d|dditjd t |d |j||j d k(rB|s@t|tr|ji|dd|d gi |j||j|0|S) a;Normalize messages to the target format (``self.message_format``). Extra keys beyond ``"role"`` and ``"content"`` are preserved during conversion. Args: messages: List of message dictionaries to normalize. Returns: Normalized list of message dictionaries. r.r/zInvalid message format: z . Skipping.rrr-rz;Cannot convert structured message to flat format: contains z# content items. Keeping structured.r=rr) rsrr0r2r5rBr3r_r4)rCr~ normalizedr^r/is_currently_structureds rnormalize_messagesz!InputFormatter.normalize_messagessS  +GW$ (@!9'+NOi(G&0$&?&lG&lPZ[bcd[egkPl #""f,1Hw<1$71:)=%%&P&P)WQZ=O&PQNN$$'L>1TV%%g.$$ 4=Tgs+%%&a&a)vW^>_=`&ab%%g.!!'*/ +2rc|jdk(r|Dcgc] }d|dg|z c}S|Dcgc]}dd|dgdg|zc}Scc}wcc}w)a'Prepend a system prompt to message format inputs. Args: messages: List of message lists (each message list represents one input). prompt: System prompt to prepend. Returns: Messages with system prompt prepended to each message list. rsystemrrr)rB)rCr~prompt message_lists rprepend_prompt_to_messagesz)InputFormatter.prepend_prompt_to_messagessn   & (_gh|h6:;lJh h!) VV,L+M N OR^ ^  i s AAc g}|D]N}t|tr|j||z(|j||dzgt|ddzP|S)abPrepend a prompt to text format inputs. For single texts, prepends the prompt directly. For text pairs (cross-encoder inputs), prepends only to the first text. Args: texts: List of text inputs (strings or pairs) prompt: Prompt to prepend Returns: Texts with prompt prepended rr-N)r0r4r_r2)rCtextsrresultrs rprepend_prompt_to_textsz&InputFormatter.prepend_prompt_to_textssb CD$$ ftm, vQ/04QR>AB  C  r)r<NN)r>r4rBrr@list[Modality] | NonereturnNone)rzLiteral['structured', 'flat'])rulist[SingleInput | PairInput]rzBtuple[Modality, dict[str, list], defaultdict[str, dict[str, Any]]])rSz tuple | listrlist[dict[str, Any]])user)rzdict[Modality, Any]r.r4rr)rzr rr5rz*tuple[Literal['message'], dict[str, list]])rlist[list[dict[str, Any]]]rbool)r~rrr)r~rrr4rr)rz'list[str | tuple[str, str] | list[str]]rr4rzlist[str | list[str]])__name__ __module__ __qualname____doc__rDrArrmrnr staticmethodrrrrrIrrr:r:fs<)/6: 1 1& 1 4 1  1 DtA-tA LtAl! F <0 0480 30:0%N 2 rrUrrYz;Multimodal dict input contains unrecognized modality keys: z. Expected keys from: z-Empty dict input is not a valid input sample.)r-r,)z#Unsupported tensor dimensionality: z ./ /TVLM M RZZ\NELLN *{{f$!& 9&++GPQ*4<+@+@*ABXY rc|sy|Dchc]}t||}}t|dk(r|jSdScc}w)aInfer the modality of a batch of input samples. If all samples share the same modality, that modality is returned. If the batch contains mixed modalities, ``"message"`` is returned, consistent with how :class:`InputFormatter` handles mixed-modality batches in :meth:`~InputFormatter.parse_inputs`. Args: samples: List of input samples to inspect. supported_modalities: Optional list of modalities the model supports. Passed through to :func:`infer_modality` to prevent misclassification of text as media modalities. Returns: The detected modality, or ``"message"`` for mixed-modality batches. rrTr-r^)r`r3rq)r|r@r6rs rinfer_batch_modalityrMsI$ bijX^.>RSjJj":!3:>> BBks>cHt|trdj|S|S)zSFormat a modality for display, e.g. ``("text", "image")`` becomes ``"text+image"``.+)r0r1join)rzs rformat_modalityres (E"xx!! Or)rr4rr)rr4r"ztuple[str, ...]rr)r6rrrrH)r6zSingleInput | PairInput | Anyr@rrr )r|rr@rrr )rzr rr4)*r __future__rloggingr collectionsrtypingrr urllib.parserrdrr)sentence_transformers.base.modality_typesrr r r PIL.Imager r ImportErrortorchcodec.decodersr rOSError getLoggerrrsrNrr#r%r(r*r8r:r`rrrIrrrsG" #! +>   8 $  %!H Fd R *II\ 37X )X/XXz37C *C/CC0]H WLLs#BB,B)(B), B:9B: