FiddlmZddlZddlmcmZddlmZddlmZddl m Z ddl m Z ddl mZmZdd lmZmZdd lmZmZdd lmZmZd ZGd dej4ZGddejj4ZGddeZy)) annotationsN)nn) trunc_normal_)MLP)LOGGER)SAM2TwoWayTransformerTwoWayTransformer) MaskDecoderSAM2MaskDecoder)ImageEncoderViT PromptEncoder)get_1d_sine_peselect_closest_cond_framesgcVeZdZUdZdZded< d dfd ZdZxZS) SAMModelaSegment Anything Model (SAM) for object segmentation tasks. This class combines image encoders, prompt encoders, and mask decoders to predict object masks from images and input prompts. Attributes: mask_threshold (float): Threshold value for mask prediction. image_encoder (ImageEncoderViT): Backbone for encoding images into embeddings. prompt_encoder (PromptEncoder): Encoder for various types of input prompts. mask_decoder (MaskDecoder): Predicts object masks from image and prompt embeddings. pixel_mean (torch.Tensor): Mean values for normalizing pixels in the input image. pixel_std (torch.Tensor): Standard deviation values for normalizing pixels in the input image. Methods: set_imgsz: Set image size to make model compatible with different image sizes. Examples: >>> image_encoder = ImageEncoderViT(...) >>> prompt_encoder = PromptEncoder(...) >>> mask_decoder = MaskDecoder(...) >>> sam_model = SAMModel(image_encoder, prompt_encoder, mask_decoder) >>> # Further usage depends on SAMPredictor class Notes: All forward() operations are implemented in the SAMPredictor class. floatmask_thresholdc(t|||_||_||_|j dt j|jdddd|j dt j|jddddy)aInitialize the SAMModel class to predict object masks from an image and input prompts. Args: image_encoder (ImageEncoderViT): The backbone used to encode the image into image embeddings. prompt_encoder (PromptEncoder): Encodes various types of input prompts. mask_decoder (MaskDecoder): Predicts masks from the image embeddings and encoded prompts. pixel_mean (list[float]): Mean values for normalizing pixels in the input image. pixel_std (list[float]): Standard deviation values for normalizing pixels in the input image. Notes: All forward() operations moved to SAMPredictor. pixel_meanrF pixel_stdN) super__init__ image_encoderprompt_encoder mask_decoderregister_buffertorchTensorview)selfrrrrr __class__s d/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/ultralytics/models/sam/modules/sam.pyrzSAMModel.__init__7s( *,( \5<< +C+H+HQPQ+RTYZ [%,,y*A*F*Fr1a*PRWXct|jdr|jj|||j_|Dcgc]}|dz c}|j_|d|j_ycc}w)CSet image size to make model compatible with different image sizes. set_imgszrN)hasattrrr)rinput_image_sizeimage_embedding_sizeimg_sizer#imgszxs r%r)zSAMModel.set_imgszRsi 4%%{ 3    ( ( //4,EJ3KAG3K0&+Ah#4Ls A:))g33333^@gR]@gRY@)g(\2M@g(\L@gL@) rr rrrr r list[float]rr2returnNone) __name__ __module__ __qualname____doc__r__annotations__rr) __classcell__r$s@r%rrsg6 NE#>> model = SAM2Model(image_encoder, memory_attention, memory_encoder) >>> image_batch = torch.rand(1, 3, 512, 512) >>> features = model.forward_image(image_batch) >>> track_results = model.track_step(0, True, features, None, None, None, {}) rrrc$dt$|||_| |_| rdnd|_||_||_|r(tjjdddd|_ ||_ |r|sJ||_ ||_ ||_||_|j |_||_|j"|_t)|j$drRt)|j$j*dr2|j$j*j,j.d|_||_tjj3tj4|dd|j&|_t9|j6d tjj3tj4dd|j"|_tjj3tj4dd|j"|_t9|j:d t9|j<d | |_||_ ||_!| |_"||_#||_$| |_%||_&||_'||_(||_)||_*||_+||_,||_-|"|_.||_/||_0||_1||_2|jbr|j^sJ|j sJ|j^re|j rYtjj3tj4d|j"|_3t9|jfd | |_4d |_5|!rYtjj3tj4d|j&|_5t9|jjd |jm| |_7d |_8|#rRtsjtd tjv|jjxd d d|j_<y y )aInitialize the SAM2Model for video object segmentation with memory-based tracking. Args: image_encoder (nn.Module): Visual encoder for extracting image features. memory_attention (nn.Module): Module for attending to memory features. memory_encoder (nn.Module): Encoder for generating memory representations. num_maskmem (int): Number of accessible memory frames. image_size (int): Size of input images. backbone_stride (int): Stride of the image backbone output. sigmoid_scale_for_mem_enc (float): Scale factor for mask sigmoid probability. sigmoid_bias_for_mem_enc (float): Bias factor for mask sigmoid probability. binarize_mask_from_pts_for_mem_enc (bool): Whether to binarize sigmoid mask logits on interacted frames with clicks during evaluation. use_mask_input_as_output_without_sam (bool): Whether to directly output the input mask without using SAM prompt encoder and mask decoder on frames with mask input. max_cond_frames_in_attn (int): Maximum number of conditioning frames to participate in memory attention. directly_add_no_mem_embed (bool): Whether to directly add no-memory embedding to image feature on the first frame. use_high_res_features_in_sam (bool): Whether to use high-resolution feature maps in the SAM mask decoder. multimask_output_in_sam (bool): Whether to output multiple masks for the first click on initial conditioning frames. multimask_min_pt_num (int): Minimum number of clicks to use multimask output in SAM. multimask_max_pt_num (int): Maximum number of clicks to use multimask output in SAM. multimask_output_for_tracking (bool): Whether to use multimask output for tracking. use_multimask_token_for_obj_ptr (bool): Whether to use multimask tokens for object pointers. iou_prediction_use_sigmoid (bool): Whether to use sigmoid to restrict IoU prediction to [0-1]. memory_temporal_stride_for_eval (int): Memory bank's temporal stride during evaluation. non_overlap_masks_for_mem_enc (bool): Whether to apply non-overlapping constraints on object masks in memory encoder during evaluation. use_obj_ptrs_in_encoder (bool): Whether to cross-attend to object pointers from other frames in the encoder. max_obj_ptrs_in_encoder (int): Maximum number of object pointers from other frames in encoder cross-attention. add_tpos_enc_to_obj_ptrs (bool): Whether to add temporal positional encoding to object pointers in the encoder. proj_tpos_enc_in_obj_ptrs (bool): Whether to add an extra linear projection layer for temporal positional encoding in object pointers. use_signed_tpos_enc_to_obj_ptrs (bool): Whether to use signed distance in the temporal positional encoding in the object pointers. only_obj_ptrs_in_the_past_for_eval (bool): Whether to only attend to object pointers in the past during evaluation. pred_obj_scores (bool): Whether to predict if there is an object in the frame. pred_obj_scores_mlp (bool): Whether to use an MLP to predict object scores. fixed_no_obj_ptr (bool): Whether to have a fixed no-object pointer when there is no object present. soft_no_obj_ptr (bool): Whether to mix in no-object pointer softly for easier recovery and error mitigation. use_mlp_for_obj_ptr_proj (bool): Whether to use MLP for object pointer projection. no_obj_embed_spatial (bool): Whether to add no-object embedding to spatial frames. sam_mask_decoder_extra_args (dict | None): Extra arguments for constructing the SAM mask decoder. compile_image_encoder (bool): Whether to compile the image encoder for faster inference. r) kernel_sizestrideout_projweightrg{Gz?)stdNTzFImage encoder compilation is enabled. First forward pass will be slow.z max-autotuneF)mode fullgraphdynamic)=rrruse_high_res_features_in_samnum_feature_levelsuse_obj_ptrs_in_encodermax_obj_ptrs_in_encoderr rConv2dmask_downsampleadd_tpos_enc_to_obj_ptrsproj_tpos_enc_in_obj_ptrsuse_signed_tpos_enc_to_obj_ptrs"only_obj_ptrs_in_the_past_for_evalmemory_attentiond_model hidden_dimmemory_encodermem_dimr+rCrDshape num_maskmem Parameterzerosmaskmem_tpos_encr no_mem_embedno_mem_pos_encdirectly_add_no_mem_embedsigmoid_scale_for_mem_encsigmoid_bias_for_mem_enc"binarize_mask_from_pts_for_mem_encnon_overlap_masks_for_mem_encmemory_temporal_stride_for_eval$use_mask_input_as_output_without_sammultimask_output_in_sammultimask_min_pt_nummultimask_max_pt_nummultimask_output_for_trackinguse_multimask_token_for_obj_ptriou_prediction_use_sigmoid image_sizebackbone_stridesam_mask_decoder_extra_argspred_obj_scorespred_obj_scores_mlpfixed_no_obj_ptrsoft_no_obj_ptr no_obj_ptruse_mlp_for_obj_ptr_projno_obj_embed_spatial_build_sam_headsmax_cond_frames_in_attn!add_all_frames_to_correct_as_condrinfocompileforward%r#rrSrVrYrlrmr`rarbrerwr_rIrfrgrhrirjrkrdrcrKrLrOrPrQrRrorprqrrrtrurncompile_image_encoderr$s% r%rzSAM2Model.__init__sun +,H)'C!'>$'>$ "$)88??1aQq?#QD (@% $+ ++)B&/N,2T/!1*22- 4&& 3@S@S@\@\^f8g..77>>DDQGDL& % 2 25;;{AqRVR^R^3_ `d++6!HH..u{{1a/QR#hh00Q4??1STd''T2d))t4)B&*C&(@%2T/-J*/N,5Y1'>$$8!$8!-J*/N,*D'%.+F(.#6 0.  '' ''// //   D$@$@#hh00Q1PQDO $//t 4(@%$(! (-(:(:5;;q$,,;W(XD % $33 > '>$15. ! KK` a).""**# *D   & !r&cHt|jjS)z=Return the device on which the model's parameters are stored.)next parametersdevicer#s r%rzSAM2Model.deviceYsDOO%&---r&ctd)zWProcess image and prompt inputs to generate object masks and scores in video sequences.zPlease use the corresponding methods in SAM2VideoPredictor for inference.See notebooks/video_predictor_example.ipynb for an example.)NotImplementedError)r#argskwargss r%r{zSAM2Model.forward^s! J  r&c |j|_|j|jz|_t |j|j|jf|j|jfd|_td dtd|jdd|jdd|j|j|j|j|jd |jxsi|_|j rwt"j$j'|j|j|_|j*rUt-|j|j|jd|_n#t"j$j/|_|j0r:t"j$j'|j|j2|_y t"j$j/|_y ) zMBuild SAM-style prompt encoder and mask decoder for image segmentation tasks.r*) embed_dimr-r, mask_in_chansr?depth embedding_dimmlp_dim num_heads num_multimask_outputs transformertransformer_dimiou_head_depthiou_head_hidden_dimuse_high_res_featuresrkrorprjN)rUsam_prompt_embed_dimrlrmsam_image_embedding_sizersam_prompt_encoderr r rIrkrorprjrnsam_mask_decoderrKr rLinear obj_ptr_projrtrIdentityrPrWobj_ptr_tpos_projrs r%rvzSAM2Model._build_sam_headses$(OO!(,4;O;O(O%#0//----"#oot?# !0! "#-"77  !55 #"&"C"C'+'F'F 00 $ 8 8,0,P,P! //52!! $  ' ' % QD ,,$'$//[\$]! % 1 1 3D   ) )&+XX__T__dll%SD "%*XX%6%6%8D "r&c |jd}|j}|jd|jk(sJ|jd|jk(sJ|jd|jk(sJ|0|d}|d} |jd|k(r| jd|k(sNJt j |dd||j}t j|dt j|  } |t|jd k(r|jdd|dfk(sJ|jd d|jjk7rHtj|j|j|jjd d d} n|} nd} |j|| fd| \} } |j!||jj#| | |d |\} }}}|j$r(|dkD}t j&|ddddf| t(} tj| |j*|j*fd d }|dddf}|rvt j,|d}t j.||}| ||fj1d}|||fj1d}|jddkDr |||f}n| |}}|j3|}|j$r^|j4r|j7}nj|j}|j8r||z}|d|z |j:zz}| ||||||fS)a[ Forward pass through SAM prompt encoders and mask heads. This method processes image features and optional point/mask inputs to generate object masks and scores. Args: backbone_features (torch.Tensor): Image features with shape (B, C, H, W). point_inputs (dict[str, torch.Tensor] | None): Dictionary containing point prompts with keys 'point_coords' (Tensor of shape (B, P, 2) with float32 dtype, containing absolute pixel-unit coordinates in (x, y) format for P input points) and 'point_labels' (Tensor of shape (B, P) with int32 dtype, where 1 means positive clicks, 0 means negative clicks, and -1 means padding). mask_inputs (torch.Tensor | None): Mask of shape (B, 1, H*16, W*16), float or bool, with the same spatial size as the image. high_res_features (list[torch.Tensor] | None): List of two feature maps with shapes (B, C, 4*H, 4*W) and (B, C, 2*H, 2*W) respectively, used as high-resolution feature maps for SAM decoder. multimask_output (bool): If True, output 3 candidate masks and their IoU estimates; if False, output only 1 mask and its IoU estimate. Returns: low_res_multimasks (torch.Tensor): Tensor of shape (B, M, H*4, W*4) with SAM output mask logits. high_res_multimasks (torch.Tensor): Tensor of shape (B, M, H*16, W*16) with upsampled mask logits. ious (torch.Tensor): Tensor of shape (B, M) with estimated IoU for each output mask. low_res_masks (torch.Tensor): Tensor of shape (B, 1, H*4, W*4) with the best low-resolution mask. high_res_masks (torch.Tensor): Tensor of shape (B, 1, H*16, W*16) with the best high-resolution mask. obj_ptr (torch.Tensor): Tensor of shape (B, C) with object pointer vector for the output mask. object_score_logits (torch.Tensor): Tensor of shape (B, 1) with object score logits. Examples: >>> backbone_features = torch.rand(1, 256, 32, 32) >>> point_inputs = {"point_coords": torch.rand(1, 2, 2), "point_labels": torch.tensor([[1, 0]])} >>> mask_inputs = torch.rand(1, 1, 512, 512) >>> results = model._forward_sam_heads(backbone_features, point_inputs, mask_inputs) >>> ( ... low_res_multimasks, ... high_res_multimasks, ... ious, ... low_res_masks, ... high_res_masks, ... obj_ptr, ... object_score_logits, ... ) = results rrrr?N point_coords point_labelsrdtype)rrr@FbilinearTsize align_cornersrF antialias)pointsboxesmasks)image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsmultimask_output repeat_imagehigh_res_features)rrFrrdimr)rXrrrrr r[ronesint32lenrmask_input_sizeF interpolatetor get_dense_perowhere NO_OBJ_SCORErlargmaxarange unsqueezerrrsigmoidrqrs)r#backbone_features point_inputs mask_inputsrrBrsam_point_coordssam_point_labelssam_mask_promptsparse_embeddingsdense_embeddingslow_res_multimasksioussam_output_tokensobject_score_logitsis_obj_appearinghigh_res_multimaskssam_output_token best_iou_inds batch_inds low_res_maskshigh_res_masksobj_ptrlambda_is_obj_appearings r%_forward_sam_headszSAM2Model._forward_sam_headssb  # #A &")) %%a(D,E,EEEE %%a(D,I,IIII %%a(D,I,IIII  #+N; +N; #))!,16F6L6LQ6OST6T TT %{{1a6IZI`I`a  % 1au{{6 RR   "{(()Q.;3D3DRa3HQPQF3R RR  %)@)@)P)PP"#--NN#4#:#:;00@@"'#" ##.#O.2.E.E$&67!/F/ ++ LPK`K`.,,99;%6$4-/LaL HD"35H   2Q6 "'-=atm-LN`bn!o  mm //4??3  -QT2 !LL26Ma7J.z=/HISSTUVM0]1JKUUVWXN %%a(1,#4Z5N#O ,>@S>M##$45   ##*=*E*E*G'*:*=*=gmm*L'$$1G;%}'t7j@|#| |djB}(tE|(|&z |'}(|jG|(}(|(jIdjKd| |j>}(n&|%jMt |#| |j>}(|j>| kro|%jOd| | |j>z|j>}%|%j ddddj#dd}%|(jQ| |j>zd}(|j|%|j|(|%jRd}nd}n|jTr9|d|jVz})|)j dddj | | | | })|)S|jVjKd| |j>g}|jXjKd| |j>g}t7jZ|d}*t7jZ|d}+|j]|||*|+|})|)j dddj | | | | })|)Scc}wcc}}wcc}}w)zePrepare memory-conditioned features by fusing current frame's visual features with previous memories.rrrrcond_frame_outputsnon_cond_frame_outputsNmaskmem_featurescuda)r non_blockingmaskmem_pos_encrrrrr?)currcurr_posmemory memory_posnum_obj_ptr_tokens)/rrUrrYrr"rrrwvaluestrainingrdrangegetappendrtyperr\rKminrLrRitemsrQabszipr stackrOrPrWtensorrrrrr new_zerosreshaperepeat_interleaverXr_r]r^catrS),r# frame_idxis_init_cond_framecurrent_vision_featscurrent_vision_pos_embedsr output_dict num_framestrack_in_reverserCHWrr  tpos_sign_mul to_cat_memoryto_cat_memory_pos_embed cond_outputsselected_cond_outputsunselected_cond_outputsoutt_pos_and_prevsrt_post_relprev_frame_idxprevfeats maskmem_encrLtptr_cond_outputs pos_and_ptrst_diffpos_list ptrs_listobj_ptrs t_diff_maxtpos_dimobj_pospix_feat_with_memrmemory_pos_embeds, r%$_prepare_memory_conditioned_featuresz.SAM2Model._prepare_memory_conditioned_featuresus ! $ ) )! , OO"~1%b)00   q '+33Aq!<AA!Q1M M.A !572M{#789A= ==&'; : !#:4I3O3O3QRC3xROR ]](L(LAq$"2"23 5((50A::JY%6PY\aPaN)(11}&:a%?N%3uqyAo%EN*3Q'71'<%=%AN%3uqyAo%EN!":;??PTU;255ndKC&&s|4- 50 / < t</0336PVP[P[_ePe3f$$U]]1%5%=%=aA%FG"#45b9<ANAN3($( (=$#3"8"8": 3 $CC']m;!$Y]!3I   $A'>?FF.> F*IPVDVA1u!7AO%&>?CCAG^GbGbcdfjGklC$++VS^,DE F *-|*<'Hi${{9!CCAq!QO  ASd( s X.X!1X%c|djd}|j}|d\}} |djdddj|||| } |jr|j s|j |}|jxr|} | r+|j s|dkDj| j} ntj|} |jdk7r| |jz} |jdk7r| |jz} |j| | d} | d }|jE|dkDj!}|d|d z |jd j"|j$zz }|| d fS) zXEncode frame features and masks into a new memory representation for video segmentation.rrrr?rT)skip_mask_sigmoidvision_features.NNr)rrUrr"rcr "_apply_non_overlapping_constraintsrbrrr rr`rarVrurrrX)r#rrpred_masks_high_resris_mask_from_ptsrr"r#r$pix_featbinarize mask_for_mem maskmem_outrrs r%_encode_new_memoryzSAM2Model._encode_new_memorys ! $ ) )! , OO"~1'+33Aq!<AA!Q1M  - -dmm#'"I"IJ]"^ ::O?O DMM/!377GL!==)<=L  ) )S 0'$*H*HHL  ( (C /'$*G*GGL))(LTX)Y &'89  $ $ 0 3a 7>>@  %5o%F!FK$JcJcKf&,,K.!. .  -=!>>>r&c ^t|dkDrft|dd|ddD cgc]H\} } | jdddj| j d| j dg| J}} } nd}|W|j rK|djddd}|jd|j g|d}|j|||}nT|j|||dd|dd|dd|| | }| ||J| }|j||}|j|||||}|||fScc} } w)hPerform a single tracking step, updating object masks and memory features based on current frame inputs.rNrrr)rrrrrrr r!)rrrrr) rrrr"rrerUrr@_use_multimaskr)r#rrrrrrrrr r!prev_sam_mask_logitsr1srrI sam_outputsrs r% _track_stepzSAM2Model._track_stepDs # $q ( 4Sb 9:cr?K!Aq( !Q"''q 166!9AqA! ! !%   "t'P'P,B/771a@H$x}}RJ:b>JH22;J[\K@@##5%9"#%>*CBC*H%bc?'%!1A H$/#/K4GGG2 #223E|T 11"*)'"3!1 2K-x77O!sA D)c|r5|jdkDr&|j|||||du\}} ||d<| |d<yd|d<d|d<y)z^Run memory encoder on predicted mask to encode it into a new memory feature for future frames.rN)rrrGrrHrr)rYrM) r#rrrrun_mem_encoderrr current_outrrs r%_encode_memory_in_outputz"SAM2Model._encode_memory_in_output~sr t//!3040G0G%9%$2$7".d": 1H1 - o/?K* +-?t@AtB#'CtD#EtH $Itl.. -9d R h- ^I:b!H)?V88t2H !':x "; Pr&r=ceZdZdZ d dfd Zd dZd fd Zed dZdZ xZ S) SAM3ModelfSAM3Model class for Segment Anything Model 3 with memory-based video object segmentation capabilities.c$t$|g||||||||| | | | | ||||||||||||||||||| |!|"|#td dtd|jdd|jdd|j |j |j|j|jd |jxsi|_ y) rrr?rrrrrrNr) rrr r rrIrkrorprjrnrr|s% r%rzSAM3Model.__init__sN $ $ $  $   $  $  $  &$  %$  /$  1$  $$  &$  )$  $$  !$ !!$ " *#$ $ ,%$ & ''$ ( ,)$ * *+$ , $-$ . $/$ 0 %1$ 2 &3$ 4 ,5$ 6 /7$ 8 9$ : ;$ < =$ > ?$ @ %A$ B !C$ D (E$ F "G$ J!0! "#)"77  !55 #"&"C"C'+'F'F 00 $ 8 8,0,P,P! //52!! r&c|jj|}|jrN|jj |dd|dd<|jj |dd|dd<|Sr)rforward_image_sam2rIrrrrs r%rzSAM3Model.forward_imagecs))<`ar&cL|j|}|j||}|S)zThis function suppresses masks that shrink in area after applying pixelwise non-overlapping constraints. Note that the final output can still be overlapping. )rFr)r#rZ!pixel_level_non_overlapping_maskss r%"_suppress_object_pw_area_shrinkagez,SAM3Model._suppress_object_pw_area_shrinkage~s1 -1,S,ST^,_)22:?`a r&) rjirwrrFFrFFFrrFFFrFFr*TFFFFFFFFFNFrkrm)r0ztuple[int, int])g333333?) r5r6r7r8rrr)rorrr:r;s@r%rqrqsp"#!"+0-2 ""'%* %&+05#(()&+ % "!%"'(-+0 %$)!& %).%*$(&+I] &*.'] :;] <"=] >?] @A] B#'C] D#E] H $I] ~a      r&rq) __future__rr torch.nn.functionalr functionalr torch.nn.initrultralytics.nn.modulesrultralytics.utilsrblocksr r decodersr r encodersr rutilsrrrModulerr=rqrr&r%rsj # '&$<24= ?/ryy?/DcPcPLF Fr&