TiC ddlmZddlZddlZddlZddlmZddlmZddl Z ddl Z ddl m Z ddl mZddlmZdd lmZej&eZerdd lmZd d d dddedddf ddZd dddef ddZddZdddef ddZ d ddZy)) annotationsN)Callable) TYPE_CHECKING)Tensor)tqdm)cos_sim)normalize_embeddings)SentenceTransformerF iii dc T|j|||d| | | } t| |||||S)a@ Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: model (SentenceTransformer): SentenceTransformer model for embedding computation sentences (List[str]): A list of strings (texts or sentences) show_progress_bar (bool, optional): Plotting of a progress bar. Defaults to False. batch_size (int, optional): Number of texts that are encoded simultaneously by the model. Defaults to 32. query_chunk_size (int, optional): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). Defaults to 5000. corpus_chunk_size (int, optional): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). Defaults to 100000. max_pairs (int, optional): Maximal number of text pairs returned. Defaults to 500000. top_k (int, optional): For each sentence, we retrieve up to top_k other sentences. Defaults to 100. score_function (Callable[[Tensor, Tensor], Tensor], optional): Function for computing scores. By default, cosine similarity. Defaults to cos_sim. truncate_dim (int, optional): The dimension to truncate sentence embeddings to. If None, uses the model's ones. Defaults to None. prompt_name (Optional[str], optional): The name of a predefined prompt to use when encoding the sentence. It must match a key in the model `prompts` dictionary, which can be set during model initialization or loaded from the model configuration. Ignored if `prompt` is provided. Defaults to None. prompt (Optional[str], optional): A raw prompt string to prepend directly to the input sentence during encoding. For instance, `prompt="query: "` transforms the sentence "What is the capital of France?" into: "query: What is the capital of France?". Use this to override the prompt logic entirely and supply your own prefix. This takes precedence over `prompt_name`. Defaults to None. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] T)show_progress_bar batch_sizeconvert_to_tensor truncate_dim prompt_nameprompt)query_chunk_sizecorpus_chunk_size max_pairstop_kscore_function)encodeparaphrase_mining_embeddings) model sentencesrrrrrrrrrr embeddingss f/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/sentence_transformers/util/retrieval.pyparaphrase_miningr sN\+!J ()+%  c |dz }tj}d}d}tdt||D])} tdt||D] } ||| | |z|| | |z} t j | t |t| dddd\} } | jj} | jj} tt| D]n}t| |D][\}}| |z}| |z}||k7s| |||kDs"|j| ||||f|dz }||k\sG|j}|d}]p,t}g}|jsg|j\}}}t||g\}}||k7r-||f|vr'|j||f|j!|||g|jsgt|dd}|S) a Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: embeddings (Tensor): A tensor with the embeddings query_chunk_size (int): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). corpus_chunk_size (int): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). max_pairs (int): Maximal number of text pairs returned. top_k (int): For each sentence, we retrieve up to top_k other sentences score_function (Callable[[Tensor, Tensor], Tensor]): Function for computing scores. By default, cosine similarity. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] rrTFdimlargestsortedc |dS)Nrxs rz.paraphrase_mining_embeddings..s !A$r!keyreverse)queue PriorityQueuerangelentorchtopkmincputolist enumerateputgetsetemptyr'addappend)rrrrrrpairs min_score num_addedcorpus_start_idxquery_start_idxscoresscores_top_k_valuesscores_top_k_idx query_itr top_k_idx corpus_itrijentry added_pairs pairs_listscoresorted_isorted_js rrrYs/0 QJE    !EII!!S_6GH1$QJ9IJ 1O#?_?O-OP+.>AR.RSF 5:JJE3vay>24PU5 1 !1#6"9"9";"B"B"D /335<<> "3v;/ 1 -67G 7R-S 1)Iz')3A(:5AAv"5i"@"Ki"W #6y#A)#LaQR"ST!Q $ 1$)IIKE(-aI 1 1 116%KJkkmiik q!#QF^( x Xx$8 $K OOXx0 1   uh9 : kkm EJ r!ct|i|S)z8This function is deprecated. Use semantic_search instead)semantic_search)argskwargss rinformation_retrievalrWs D +F ++r! c Ft|tjtjfrt j |}n%t|t rt j|}t|jdk(r|jd}t|tjtjfrt j |}n%t|t rt j|}|j|jk7r|j|j}tt|Dcgc]}g}}tdt||D]}t||zt|} |jr5t j || |j} |j#d| } n||| } tdt||D]^} t| |zt|} |jr5t j | | |j} |j#d| }n|| | }|| |}t j$|t|t|dddd\}}|j'j)}|j'j)}tt|D]n}t+||||D]W\}}| |z}||z}t|||krt-j.||||f=t-j0||||fYpatt|D]I}tt||D]}|||\}}||d|||<t3||dd ||<K|Scc}w) a3 This function performs by default a cosine similarity search between a list of query embeddings and a list of corpus embeddings. It can be used for Information Retrieval / Semantic Search for corpora up to about 1 Million entries. Args: query_embeddings (:class:`~torch.Tensor`): A 2 dimensional tensor with the query embeddings. Can be a sparse tensor. corpus_embeddings (:class:`~torch.Tensor`): A 2 dimensional tensor with the corpus embeddings. Can be a sparse tensor. query_chunk_size (int, optional): Process 100 queries simultaneously. Increasing that value increases the speed, but requires more memory. Defaults to 100. corpus_chunk_size (int, optional): Scans the corpus 100k entries at a time. Increasing that value increases the speed, but requires more memory. Defaults to 500000. top_k (int, optional): Retrieve top k matching entries. Defaults to 10. score_function (Callable[[:class:`~torch.Tensor`, :class:`~torch.Tensor`], :class:`~torch.Tensor`], optional): Function for computing scores. By default, cosine similarity. Returns: List[List[Dict[str, Union[int, float]]]]: A list with one entry for each query. Each entry is a list of dictionaries with the keys 'corpus_id' and 'score', sorted by decreasing cosine similarity scores. rrdeviceTFr$) corpus_idrPc |dS)NrPr)r*s rr,z!semantic_search..s \]^e\fr!r-) isinstancenpndarraygenericr4 from_numpyliststackr3shape unsqueezer[tor2r6 is_sparsearange index_selectr5r7r8zipheapqheappush heappushpopr')query_embeddingscorpus_embeddingsrrrr_queries_result_listrD query_end_idxindices query_chunkrCcorpus_end_idx corpus_chunk cos_scorescos_scores_top_k_valuescos_scores_top_k_idxrH sub_corpus_idrPr\query_iddoc_itrs rrTrTs0"RZZ$<= ++,<= $d + ;;'78  ! !"a'+55a8#bjj"**%=>!,,->? %t ,!KK(9:#3#:#::+../@/G/GH',S1A-B'CD!2DD C(8$9;KL$]O.>>DT@UV  % %ll?MJZJaJabG*777CK*?=IK!&a->)?AR S ]  !14E!EsK\G]^N **,,'7PaPhPhi0==aI 01A.Q ( \BJ=BJJCs:a='9:4X]= 9 #%9'>&A&A&C&J&J&L ##7#;#;#=#D#D#F "3z?3 ] ,/0DY0OQhirQs,t](M5 0= @I.:H.x89EA/9E9;M))*=h*G%QZI[\] ]% ]$]N#123vS!4X!>?@ ^G28 Nct|tjstj|}tj||j}t |}g}t |t|}t td|zdt|}ttdt||d| D]}||||z|jz}|jjdvr||k\} | jd} | |k\} | js]| | } || }| j} |j| d \} }t!| |D]'\}}|j#|d |j%)|j|d \}} tt|D]}||d |k\s||j|d \}}|d |kDrV|t|krHt d|zt|}||j|d \}}|d |kDr|t|krH|j#|||k\j%t'|d d }g}t)}t+|D]U\}}g}|D]}||vs|j#|t||k\s4|j#||j-|Wt'|dd }|S)a Function for Fast Community Detection. Finds in the embeddings all communities, i.e. embeddings that are close (closer than threshold). Returns only communities that are larger than min_community_size. The communities are returned in decreasing order. The first element in each list is the central point in the community. Args: embeddings (torch.Tensor or numpy.ndarray): The input embeddings. threshold (float): The threshold for determining if two embeddings are close. Defaults to 0.75. min_community_size (int): The minimum size of a community to be considered. Defaults to 10. batch_size (int): The batch size for computing cosine similarity scores. Defaults to 1024. show_progress_bar (bool): Whether to show a progress bar during computation. Defaults to False. Returns: List[List[int]]: A list of communities, where each community is represented as a list of indices. rZ2rzFinding clusters)descdisable)cudanpurT)kr&Nr#ct|SNr3r*s rr,z%community_detection..Ss Ar!r-ct|Srrr*s rr,z%community_detection..cs #a&r!)r^r4rtensorr[r r6r3maxrr2Ttypesumanyr5rkr?r8r'r<r9update)r thresholdmin_community_sizerrextracted_communities sort_max_size start_idxrxthreshold_maskrow_wise_countlarge_enough_maskrrq top_k_indicescountrt top_k_valuesrK top_val_large top_idx_largeunique_communities extracted_ids cluster_id communitynon_overlapped_communityidxs rcommunity_detectionrs.0 j%,, /\\*-  Yz/@/@AI%j1J/ZAA 22B7ZIM aZ*-4FTePe*e  I ,BCjllR     ! !_ 4'94N+//2N!/2D D $((*+,=>N#$56J""$A)DA A}#&nm"D Gw%,,WVe_-C-C-EF G)oo0BDoQOL!3|,- e?2&)33=a=3E3E _c3E3d0M=(+i7MCPZO<[(+A ,=s:(O 7A!}7I7IMcg7I7h4 }(+i7MCPZO<[*00}PY?Y1Z1a1a1cd eA*eZ##8>NX\]EM!*+@!A; I#%  5C-'(//4 5 ' (,> >  % %&> ?  !9 :; 28HRVW r!)rr rz list[str]rboolrintrrrrrrrrr"Callable[[Tensor, Tensor], Tensor]rz int | Noner str | Nonerrreturnlist[list[float | int]])rrrrrrrrrrrrrr)r"list[list[dict[str, int | float]]])rorrprrrrrrrrrrr)g?rXiF) rztorch.Tensor | np.ndarrayrfloatrrrrrrrzlist[list[int]]) __future__rrlloggingr0collections.abcrtypingrnumpyr_r4rtqdm.autonotebookr similarityr rr getLogger__name__logger0sentence_transformers.sentence_transformer.modelr r rrWrTrr)r!rrs"  $  "(   8 $T $ #9@#"? ??? ?  ?  ?? ?7??? ??H!#9@ FFFF F  F 7 FFR, #9@ XXXX X  X 7 X(Xz # c)ccc c  c  cr!