Fi6V:dZddlmZddlZddlmZddlmZddlZddl m Z ddl m cm Z ddlmcmZddlmZddlmZddlmZmZmZmZmZmZdd lmZd d lmZGd d e j>Z Gdde j>Z!Gdde j>Z"y)a@ ViTDet backbone adapted from Detectron2. This module implements Vision Transformer (ViT) backbone for object detection. Rope embedding code adopted from: 1. https://github.com/meta-llama/codellama/blob/main/llama/model.py 2. https://github.com/naver-ai/rope-vit 3. https://github.com/lucidrains/rotary-embedding-torch ) annotationsN)partial)Callable)Tensor) PatchEmbed)apply_rotary_enccompute_axial_cisconcat_rel_pos get_abs_poswindow_partitionwindow_unpartition)check_requirements) LayerScaleceZdZdZ d dfd Zd d dZd d dZd dZddZxZ S) AttentionzIMulti-head Attention block with relative position embeddings and 2d-rope.c t |||_||z|_|jdz|_||_t j||dz||_t j|||_ ||_ ||_ ||_ | |_ | |_| |_|j!|||j#|y)a  Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. qkv_bias (bool): If True, add a learnable bias to query, key, value. use_rel_pos (bool): If True, add relative positional embeddings to the attention map. rel_pos_zero_init (bool): If True, zero initialize relative positional parameters. input_size (tuple[int, int] or None): Input resolution for calculating the relative positional parameter size or rope size. cls_token (bool): Whether a cls_token is present. use_rope (bool): Whether to use rope 2d (independent of use_rel_pos, as it can be used together). rope_theta (float): Control frequencies of rope. rope_pt_size (tuple[int, int] or None): Size of rope in previous stage of training, needed for interpolation or tiling. rope_interp (bool): Whether to interpolate (or extrapolate) rope to match input size. )biasN)super__init__ num_headshead_dimscale cls_tokennnLinearqkvproj use_rel_pos input_sizeuse_rope rope_theta rope_pt_size rope_interp_setup_rel_pos_setup_rope_freqs) selfdimrqkv_biasr!rel_pos_zero_initr"rr#r$r%r& __class__s d/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/ultralytics/models/sam/sam3/vitdet.pyrzAttention.__init__,s< "y( ]]D( "99S#'9IIc3' '$  $(& -z: z*c|jsd|_d|_y|J|jdusJdt j t jd|dzdz |j|_t j t jd|dzdz |j|_|sVtjj|jdtjj|jd|\}}t j|dddf}t j|dddf}||z |dz z}|j|_ y) z%Setup relative positional embeddings.NFz not supportedrr{Gz?std)r! rel_pos_h rel_pos_wrr Parametertorchzerosrinit trunc_normal_arangelongrelative_coords)r)r,r"HWq_coordsk_coordsr>s r.r'zAttention._setup_rel_pos`s-!DN!DN %%%~~&77&ekk!jm2Ca2G&WXekk!jm2Ca2G&WX GG ! !$..d ! ; GG ! !$..d ! ;1<<?1d7+<<?47+#h.1q59.335r/c\|jsd|_y|J|j||_tt|j |j |_d}|jr|jd|dz }|j|d|d|}|jrtj|j dztj|j}tjtj||dddf}tj ||gd }||_y) zSetup 2d-rope frequencies.N)r*theta?rr)end_xend_y scale_posr1)dtypedevicer*)r# freqs_cisr%rr rr$ compute_cisr&rr8r9float32rJpolar ones_likecat)r)r"rHrLt cls_freqs_ciss r.r(zAttention._setup_rope_freqsxs}}!DN %%%    $ *D #  //    ))!,z!}>  "mm ''A "KK(:A>tQwGM =)")rndimshapemathsqrtrreshaperpermuteunbindrXr!r flattenr5r6r>Fscaled_dot_product_attentionviewr ) r)xsBr?r@_Lr]rrVrWvs r.forwardzAttention.forwards&AQ 66Q;JAq!Q6M6AAD66Q; ;ggGAq!DIIa!e$ $Ahhqk!!!Q4>>2>++aAq!,33A61a1%1   ! !Q !QA!  $ 4 4 DAq !T^^QUB7A !T^^QUB7A * *1a 3 19q$..!Q3;;Aq!QJRRSTVWYZ\^_Aq$..!R088Aq!DLLQPQSUVA IIaLr/) TFTNFFg@NF)r*intrrpr+boolr!rqr,rqr"tuple[int, int] | Nonerrqr#rqr$floatr%rrr&rq)TN)r,rqr"rrreturnNone)N)r"rrrtru)rtztuple[Tensor, Tensor]rhrrtr) __name__ __module__ __qualname____doc__rr'r(rXrn __classcell__r-s@r.rr)sS !"&-1#/3!2+ 2+2+ 2+  2+  2++2+2+2+2+-2+2+h60%#NM.r/rceZdZdZdddej ej ddddddddddf d fd Zd d ZxZ S) Blockz4Transformer blocks with support of window attention.@TFrNc .t|tdddlm}m}|||_t||||| | dk(r| n| | f| | || |_|r t||ntj|_ |dkDr||ntj|_ |||_||t||z||df|_|r t||ntj|_tj$||_| |_y) a Args: dim (int): Number of input channels. num_heads (int): Number of attention heads in each ViT block. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool): If True, add a learnable bias to query, key, value. drop_path (float): Stochastic depth rate. norm_layer (Callable): Normalization layer constructor. act_layer (Callable): Activation layer constructor. use_rel_pos (bool): If True, add relative positional embeddings to the attention map. rel_pos_zero_init (bool): If True, zero initialize relative positional parameters. window_size (int): Window size for window attention blocks. If it equals 0, then not use window attention. input_size (tuple[int, int] | None): Input resolution for calculating the relative positional parameter size. use_rope (bool): Whether to use rope 2d (independent of use_rel_pos, as it can be used together). rope_pt_size (tuple[int, int] | None): Size of rope in previous stage of training, needed for interpolation or tiling. rope_interp (bool): Whether to interpolate (or extrapolate) rope to match target input size, expected to specify source size as rope_pt_size. cls_token (bool): Whether a cls_token is present. dropout (float): Dropout rate. init_values (float | None): Layer scale init, None for no layer scale. timmr)DropPathMlp) rr+r!r,r"r#r%r&r) init_valuesr) in_featureshidden_features act_layerdropN)rrr timm.layersrrnorm1rattnrrIdentityls1 drop_pathnorm2rpmlpls2Dropoutdropout window_size)r)r*r mlp_ratior+r norm_layerrr!r,rr"r#r%r&rrrrrr-s r.rzBlock.__init__sV 6"-_  #/%0A%5zK;U%#   @K:c{;PRP[P[P]09C),R[[]_ i03   @K:c{;PRP[P[P]zz'* &r/c 0|}|j|}|jdkDr7|jd|jd}}t||j\}}|j |j |}|jdkDrt ||jf}||j|j|z}||j|j|j|j|j|z}|S)z&Forward pass of the transformer block.rrr1) rrr^r rrr rrrrr)r)rhshortcutr?r@pad_hws r.rnz Block.forward&s JJqM   a 771:qwwqzqA(D,<,<=IAv HHTYYq\ "   a "1d&6&6AGA t||DNN1$56 6  T^^DHHTXXdjjm5L,MNO Or/)"r*rprrprrsr+rqrrsrCallable[..., nn.Module]rrr!rqr,rqrrpr"rrr#rqr%rrr&rqrrqrrsr float | Nonerv) rwrxryrzr LayerNormGELUrrnr{r|s@r.r~r~s> /1||.0gg!"&-1/3!$(%I' I'I' I'  I'  I'-I',I'I' I'I'+I'I'-I'I' !I'"#I'$"%I'Vr/r~ceZdZdZddddddddd d ej ddd dd d d dd dddd d dd d dddf dfd ZeddZddZ ddgfddZ xZ S)ViTzThis module implements Vision Transformer (ViT) backbone in :paper:`vitdet`. "Exploring Plain Vision Transformer Backbones for Object Detection", https://arxiv.org/abs/2203.16527. iri rTrr)r1ro FNc t)|||_t|D cgc] } | |vs|  }!} t ||_dg|z|_t|tr|r dg|z|_n|D]} d|j | <||_ |jro|sJt|!dk(sJdt|j dk(sJd|dz}"tj|"tjdd|z|_t| t"rt%t't| d } t)||f||f||| |_| |_| |_|j,r|j.sJ|j.rD||z||zz}#|r|#dzn|#}$tjtj0d|$||_nd |_tj4d| |D%cgc]}%|%j7}&}%||_||_tj<|_d}'t|D]} tAd id |d |d|d|d|&| d| d| d|j | d|d| |!vr|ndd||z||zfd|d|||fn||fd|d|jd|d|}(| |!vr|'dz }'||_!|j>jE|(||_#|r|gt|j zn|g|_$|j2+tjJjM|j2d|r| |ntjN|_(|r| |ntjN|_)|jU|jV|atjX|jZ|d|_-|jBr-|j\r dtj^j`_1y y y y cc} wcc}%w)!ak Args: img_size (int): Input image size. Only relevant for rel pos or rope. patch_size (int): Patch size. in_chans (int): Number of input image channels. embed_dim (int): Patch embedding dimension. depth (int): Depth of ViT. num_heads (int): Number of attention heads in each ViT block. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool): If True, add a learnable bias to query, key, value. drop_path_rate (float): Stochastic depth rate. norm_layer (Callable or str): Normalization layer constructor or name. act_layer (Callable): Activation layer constructor. use_abs_pos (bool): If True, use absolute positional embeddings. tile_abs_pos (bool): If True, tile absolute positional embeddings instead of interpolation. rel_pos_blocks (tuple[int, ...] | bool): Blocks which have rel pos embeddings. rel_pos_zero_init (bool): If True, zero initialize relative positional parameters. window_size (int): Window size for window attention blocks. global_att_blocks (tuple[int, ...]): Indexes for blocks using global attention (other blocks use window attention). use_rope (bool): Whether to use rope 2d (independent of rel_pos_blocks, as it can be used together). rope_pt_size (int | None): Size of rope in previous stage of training, needed for interpolation or tiling. use_interp_rope (bool): Whether to interpolate (or extrapolate) rope to match target input size, expected to specify source size as rope_pt_size. pretrain_img_size (int): Input image size for pretraining models. pretrain_use_cls_token (bool): If True, pretraining models use class token. retain_cls_token (bool): Whether cls_token should be retained. dropout (float): Dropout rate. Applied in residual blocks of attn, mlp and inside the mlp. return_interm_layers (bool): Whether to return intermediate layers (all global attention blocks). init_values (float | None): Layer scale init, None for no layer scale. ln_pre (bool): If True, apply layer norm before transformer blocks. ln_post (bool): If True, apply layer norm after transformer blocks. bias_patch_embed (bool): If True, use bias in conv for patch embed. compile_mode (str | None): Mode to compile the forward, or None to disable. use_act_checkpoint (bool): If True, use activation checkpointing. FTrz&windowing not supported with cls tokenz$rel pos not supported with cls tokenrrgh㈵>)eps) kernel_sizestridein_chans embed_dimrNr*rrr+rrrr!r,rr"r#r%r&rrrr2r3)mode fullgraph)2rrpretrain_use_cls_tokenrangelist full_attn_idsrel_pos_blocks isinstancerqretain_cls_tokenlensumrr7r8randnclass_embeddingstrrgetattrr patch_embed tile_abs_pos use_abs_posr9 pos_embedlinspaceitem patch_sizer ModuleListblocksr~use_act_checkpointappendreturn_interm_layers channel_listr:r;rln_preln_postapply _init_weightscompilerntraining_dynamoconfig optimize_ddp)*r)img_sizerrrdepthrrr+drop_path_raterrrrrr,rglobal_att_blocksr#r%use_interp_ropepretrain_img_sizerrrrrrrbias_patch_embed compile_moderiwindow_block_indexesr num_patches num_positionsrhdpr cur_stageblockr-s* r.rz ViT.__init__?s4L &<#+0<Va1DU;UVV!"34$go nd +#'&5.D # .)-##A& .!1  ) ))+,1 [3[ [1t**+q0 X2X X0tOE#%<< Aq)8T0T#UD j# & Z!8dCJ%#Z0 +!  )&   ## ##   , :?PT^?^_K1G[1_[M\\%++a *RSDN!DN"'>5!IJAqvvxJJ$&mmo  u &A#$"  a&  & $!//2#4,-0D+DK!% 2H 4JK"=I> % GG ! !$..d ! ;/5j+2;;= 07z),R[[]  4%%&  # ==LTXYDL&&4==49 $$1,9& $u WZKs O0O0!O5ct|tjr~tjj |j dt|tjr8|j +tjj|j dyyyt|tjrUtjj|j dtjj|j dyy)zInitialize the weights.r2r3NrrE) rrrr:r;weightr constant_r)ms r.rzViT._init_weightss a # GG ! !!(( ! 5!RYY'AFF,>!!!&&!,-?' 2<< ( GG  affa ( GG  ahh ,)r/c^|j|}|jd|jd}}d}|jr5tj|j |j ddgd}d}|j=|t|j|j||f|j|jz}|j|}g}t|jD]E\}}|jr%|jrt!j ||d}n||}||j"d k(s|j$s_||j"vsn||j"d k(r|j'|}|dd|df}|j(d k(r|j+dd dd}nu|j(d k(sJt-j.|jdx}}|j1|jd|||jd j+dd dd}|j3|H|S) z&Vit forward path and get feature maps.rr1rrKN)tilingF) use_reentrantr[rZr)rr^rr8rQrrdrr rrr enumeraterrr checkpointrrrr]rbr_r`rar) r)rhhwrioutputsrblkfeatss r.rnz ViT.forwards   Q wwqz1771:1    4//1aAqIAA >> %K++A%%(( A KKN , &FAs&&4==))#qFFT''++1J1JqTXTfTfOf**2.. QA!QR%::?!MM!Q15E ::?*? IIekk!n55A!MM%++a.!Q BPXXYZ\]_`bcdEu%# &&r/ic>|jD]}|jdk7r|jj|d|jz|d|jzf|jj |d|jz|d|jzfy)zCSetup rel pos embeddings and rope freqs for a new input image size.rr)r"N)rrrr'rr()r)imgszrs r. set_imgszz ViT.set_imgszs[[ pE  A% JJ % %%(doo2MuUVx[_[j[jOj1k % l JJ ( (U1X5PRWXYRZ^b^m^mRm4n ( o  pr/)>rrprrprrprrprrprrprrsr+rqrrsrzCallable[..., nn.Module] | strrrrrqrrqrztuple[int, ...] | boolr,rqrrprztuple[int, ...]r#rqr%z int | Nonerrqrrprrqrrqrrsrrqrrrrqrrqrrqrz str | Nonerrq)rz nn.Modulertru)rhz torch.Tensorrtzlist[torch.Tensor])rz list[int]) rwrxryrzrrr staticmethodrrnrr{r|s@r.rr:s  #5@.0gg !1>"&-:#' %!$'+!%%*$(!%#'#'Af:f:f: f:  f:  f:f:f:f:f:3f:,f:f:f:/f:  !f:"#f:$+%f:&'f:(!)f:*+f:,-f:.!%/f:01f:23f:4#5f:6"7f:89f::;f:<=f:>!?f:@!Af:P--+Z-1$<ppr/r)#rz __future__rr_ functoolsrtypingrr8torch.nnrtorch.nn.functional functionalretorch.utils.checkpointutilsrr%ultralytics.models.sam.modules.blocksr$ultralytics.models.sam.modules.utilsrr r r r r ultralytics.utils.checksr model_miscrModulerr~rrr/r.rs| #  ++<8"l l^_BII_Dkp"))kpr/