FiddlmZddlZddlZddlmZddlZddlZddl m cm Z ddlm Z m Z ddlmZmZmZddlmZmZmZddlmZmZmZmZmZGd d e j8ZGd d e j8ZGd de j8ZGdde j8Z GddeZ!GddeZ"GddeZ#d&d'dZ$Gdde j8Z%Gdde j8Z&Gdde j8Z'Gdde j8Z(Gd d!e j8Z)Gd"d#e j8Z*Gd$d%e j8Z+y)() annotationsN)partial)Tensornn)MLP LayerNorm2dMLPBlock) AttentionTwoWayAttentionBlockTwoWayTransformer)add_decomposed_rel_posapply_rotary_enccompute_axial_ciswindow_partitionwindow_unpartitionc.eZdZdZddfd ZddZxZS)DropPathaImplements stochastic depth regularization for neural networks during training. Attributes: drop_prob (float): Probability of dropping a path during training. scale_by_keep (bool): Whether to scale the output by the keep probability. Methods: forward: Applies stochastic depth to input tensor during training, with optional scaling. Examples: >>> drop_path = DropPath(drop_prob=0.2, scale_by_keep=True) >>> x = torch.randn(32, 64, 224, 224) >>> output = drop_path(x) c>t|||_||_y)zOInitialize DropPath module for stochastic depth regularization during training.N)super__init__ drop_prob scale_by_keep)selfrr __class__s g/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/ultralytics/models/sam/modules/blocks.pyrzDropPath.__init__#s "*c.|jdk(s |js|Sd|jz }|jdfd|jdz zz}|j |j |}|dkDr|j r|j|||zS)zNApply stochastic depth to input tensor during training, with optional scaling.r r)r )rtrainingshapendim new_empty bernoulli_rdiv_)rx keep_probr! random_tensors rforwardzDropPath.forward)s >>S  H&   33 E*55i@ s?t11   y )=  r)rT)rfloatrboolr&rreturnr__name__ __module__ __qualname____doc__rr) __classcell__rs@rrrs + !rrcjeZdZdZdddddej df d fd Zd dZxZS) MaskDownSampleraA mask downsampling and embedding module for efficient processing of input masks. This class implements a mask downsampler that progressively reduces the spatial dimensions of input masks while expanding their channel dimensions using convolutional layers, layer normalization, and activation functions. Attributes: encoder (nn.Sequential): A sequential container of convolutional layers, layer normalization, and activation functions for downsampling and embedding masks. Methods: forward: Downsamples and encodes input mask to embed_dim channels. Examples: >>> mask_downsampler = MaskDownSampler(embed_dim=256, kernel_size=4, stride=4, padding=0, total_stride=16) >>> input_mask = torch.randn(1, 1, 256, 256) >>> output = mask_downsampler(input_mask) >>> print(output.shape) torch.Size([1, 256, 16, 16]) rNc Vt |ttj|tj|z}||z|k(sJt j |_d\} } t|D]} | |dzz} |jjt j| | ||||jjt| |jj|| } |jjt j| |d||_ |jit|jttfsJdt!|jdt||_ t#|jdk(sJyy) zXInitialize a mask downsampler module for progressive downsampling and channel expansion.)r r ) kernel_sizestridepaddingr r<NzUnsupported type z. Should be a list or tuple.)rrintmathlog2r SequentialencoderrangeappendConv2dr interpol_size isinstancelisttupletypelen) r embed_dimr<r=r> total_stride activationrH num_layers mask_in_chansmask_out_chans_rs rrzMaskDownSampler.__init__Jsx <0DIIf4EEF z!\111}} (,% ~z" +A*fai8N LL   !" +!#   LL   N ; < LL   -*M + BIIniQOP*    )d004-@ #D););$<#==YZ @"&m!4D t))*a/ // *rc|jp|jt|jddk7rKtj|j |jdddj |j}|j|S)zbDownsample and encode input mask to embed_dim channels using convolutional layers and LayerNorm2d.NFbilinearT)size align_cornersmode antialias) rHrJr!F interpolater*todtyperDrr&s rr)zMaskDownSampler.forwardrsw    )d.@.@DQSQTDV.V  ''#  bk ||Ar)rNr@r<r@r=r@r>r@rOr@rPtype[nn.Module]rHtuple[int, int] | Noner, r/r0r1r2rGELUrr)r3r4s@rr6r65sr,&(gg04&0&0&0 &0  &0  &0$&0.&0P rr6cNeZdZdZ d dfd ZddZxZS)CXBlockaYConvNeXt Block for efficient feature extraction in convolutional neural networks. This block implements a modified version of the ConvNeXt architecture, offering improved performance and flexibility in feature extraction. Attributes: dwconv (nn.Conv2d): Depthwise or standard 2D convolution layer. norm (LayerNorm2d): Layer normalization applied to channels. pwconv1 (nn.Linear): First pointwise convolution implemented as a linear layer. act (nn.GELU): GELU activation function. pwconv2 (nn.Linear): Second pointwise convolution implemented as a linear layer. gamma (nn.Parameter | None): Learnable scale parameter for layer scaling. drop_path (nn.Module): DropPath layer for stochastic depth regularization. Methods: forward: Processes the input tensor through the ConvNeXt block. Examples: >>> import torch >>> x = torch.randn(1, 64, 56, 56) >>> block = CXBlock(dim=64, kernel_size=7, padding=3) >>> output = block(x) >>> print(output.shape) torch.Size([1, 64, 56, 56]) ct|tj|||||r|nd|_t |d|_tj|d|z|_tj|_ tjd|z||_ |dkDr-tj|tj|zdnd |_|d kDrt!||_y tj"|_y ) aInitialize a ConvNeXt Block for efficient feature extraction in convolutional neural networks. This block implements a modified version of the ConvNeXt architecture, offering improved performance and flexibility in feature extraction. Args: dim (int): Number of input channels. kernel_size (int): Size of the convolutional kernel. padding (int): Padding size for the convolution. drop_path (float): Stochastic depth rate. layer_scale_init_value (float): Initial value for Layer Scale. use_dwconv (bool): Whether to use depthwise convolution. r )r<r>groupsư>epsr8rT) requires_gradNr)rrrrGdwconvrnormLinearpwconv1rdactpwconv2 ParametertorchonesgammarIdentity drop_path)rdimr<r>rxlayer_scale_init_value use_dwconvrs rrzCXBlock.__init__s, ii  #$3!    . yya#g. 779yyS#. &) LL/%**S/AQU V 1:C),R[[]rcb|}|j|}|j|}|jdddd}|j|}|j |}|j |}|j |j |z}|jdddd}||j|z}|S)z`Apply ConvNeXt block operations to input tensor, including convolutions and residual connection.rr;r )rmrnpermuterprqrrrvrx)rr&inputs rr)zCXBlock.forwards KKN IIaL IIaAq ! LLO HHQK LLO :: ! QA IIaAq ! DNN1% %r)r}rriT) ryr@r<r@r>r@rxr*rzr*r{r+r,r.r4s@rrfrfse:(,'S 'S'S 'S  'S !& 'S'SRrrfc.eZdZdZddfd ZddZxZS)FuseraA module for fusing features through multiple layers of a neural network. This class applies a series of identical layers to an input tensor, optionally projecting the input first. Attributes: proj (nn.Module): An optional input projection layer. Identity if no projection is needed. layers (nn.ModuleList): A list of identical layers to be applied sequentially. Methods: forward: Applies the fuser to an input tensor. Examples: >>> layer = CXBlock(dim=256) >>> fuser = Fuser(layer, num_layers=3, dim=256, input_projection=True) >>> x = torch.randn(1, 256, 32, 32) >>> output = fuser(x) >>> print(output.shape) torch.Size([1, 256, 32, 32]) c.t|tj|_tj t |Dcgc]}tj|c}|_ |r"|Jtj||d|_yycc}w)aInitialize the Fuser module for feature fusion through multiple layers. This module creates a sequence of identical layers and optionally applies an input projection. Args: layer (nn.Module): The layer to be replicated in the fuser. num_layers (int): The number of times to replicate the layer. dim (int | None): The dimension for input projection, if used. input_projection (bool): Whether to use input projection. Nr r?) rrrrwproj ModuleListrEcopydeepcopylayersrG)rlayerrQryinput_projectionrTrs rrzFuser.__init__sr KKM mm5CT$UaT]]5%9$UV ? "? #s:DI %VsBcZ|j|}|jD] }||} |S)zMApply a series of layers to the input tensor, optionally projecting it first.)rr)rr&rs rr)z Fuser.forwards0 IIaL[[ EaA r)NF)r nn.ModulerQr@ryz int | Nonerr+r,r.r4s@rrrs(;&rrc\eZdZdZdej ddf dfd ZxZS)SAM2TwoWayAttentionBlockaA two-way attention block for performing self-attention and cross-attention in both directions. This block extends the TwoWayAttentionBlock and consists of four main components: self-attention on sparse inputs, cross-attention from sparse to dense inputs, an MLP block on sparse inputs, and cross-attention from dense to sparse inputs. Attributes: self_attn (Attention): Self-attention layer for queries. norm1 (nn.LayerNorm): Layer normalization after the first attention block. cross_attn_token_to_image (Attention): Cross-attention layer from queries to keys. norm2 (nn.LayerNorm): Layer normalization after the second attention block. mlp (MLP): MLP block for transforming query embeddings. norm3 (nn.LayerNorm): Layer normalization after the MLP block. norm4 (nn.LayerNorm): Layer normalization after the third attention block. cross_attn_image_to_token (Attention): Cross-attention layer from keys to queries. skip_first_layer_pe (bool): Flag to skip positional encoding in the first layer. Methods: forward: Processes input through the attention blocks and MLP. Examples: >>> block = SAM2TwoWayAttentionBlock(embedding_dim=256, num_heads=8) >>> sparse_input = torch.randn(1, 100, 256) >>> dense_input = torch.randn(1, 256, 16, 16) >>> sparse_output, dense_output = block(sparse_input, dense_input) ir;FcXt|||||||t|||d||_y)avInitialize a SAM2TwoWayAttentionBlock for performing self-attention and cross-attention in two directions. This block extends the TwoWayAttentionBlock and consists of four main components: self-attention on sparse inputs, cross-attention from sparse to dense inputs, an MLP block on sparse inputs, and cross-attention from dense to sparse inputs. Args: embedding_dim (int): The channel dimension of the embeddings. num_heads (int): The number of heads in the attention layers. mlp_dim (int): The hidden dimension of the MLP block. activation (type[nn.Module]): The activation function of the MLP block. attention_downsample_rate (int): The downsample rate for attention computations. skip_first_layer_pe (bool): Whether to skip the positional encoding in the first layer. r;rQrqN)rrrmlp)r embedding_dim num_headsmlp_dimrPattention_downsample_rateskip_first_layer_pers rrz!SAM2TwoWayAttentionBlock.__init__ s3.  7JHacvw}g}PZ[r)rr@rr@rr@rPrarr@rr+r-Noner/r0r1r2rReLUrr3r4s@rrrsq>&(gg)*$)\\\ \ $ \ $' \"\ \\rrcXeZdZdZej df dfd ZxZS)SAM2TwoWayTransformeraA Two-Way Transformer module for simultaneous attention to image and query points. This class extends the TwoWayTransformer, implementing a specialized transformer decoder that attends to an input image using queries with supplied positional embeddings. It is particularly useful for tasks like object detection, image segmentation, and point cloud processing. Attributes: depth (int): Number of layers in the transformer. embedding_dim (int): Channel dimension for input embeddings. num_heads (int): Number of heads for multihead attention. mlp_dim (int): Internal channel dimension for the MLP block. layers (nn.ModuleList): List of SAM2TwoWayAttentionBlock layers comprising the transformer. final_attn_token_to_image (Attention): Final attention layer from queries to image. norm_final_attn (nn.LayerNorm): Layer normalization applied to final queries. Methods: forward: Processes input image embeddings and query embeddings through the transformer. Examples: >>> transformer = SAM2TwoWayTransformer(depth=5, embedding_dim=256, num_heads=8, mlp_dim=2048) >>> image_embedding = torch.randn(1, 256, 64, 64) >>> query_embedding = torch.randn(1, 100, 256) >>> output = transformer(image_embedding, query_embedding) >>> print(output[0].shape, output[1].shape) torch.Size([1, 100, 256]) torch.Size([1, 256, 64, 64]) r;c t|||||||tj|_t |D]/}|jj t||||||dk(1y)aInitialize a SAM2TwoWayTransformer instance. This transformer decoder attends to an input image using queries with supplied positional embeddings. It is designed for tasks like object detection, image segmentation, and point cloud processing. Args: depth (int): Number of layers in the transformer. embedding_dim (int): Channel dimension for the input embeddings. num_heads (int): Number of heads for multihead attention. Must divide embedding_dim. mlp_dim (int): Channel dimension internal to the MLP block. activation (type[nn.Module]): Activation function to use in the MLP block. attention_downsample_rate (int): Downsampling rate for attention computations. r)rrrrPrrN)rrrrrrErFr) rdepthrrrrPrirs rrzSAM2TwoWayTransformer.__init__Wsm,  y':Ohimmo u A KK  ("/'#).G)*a   r)rr@rr@rr@rr@rPrarr@r-rrr4s@rrr;scB')gg)*""" "  " $ "$'" ""rrcBeZdZdZdddd dfdZd d dZxZS) RoPEAttentionaImplements rotary position encoding for attention mechanisms in transformer architectures. This class extends the base Attention class by incorporating Rotary Position Encoding (RoPE) to enhance the positional awareness of the attention mechanism. Attributes: compute_cis (Callable): Function to compute axial complex numbers for rotary encoding. freqs_cis (torch.Tensor): Precomputed frequency tensor for rotary encoding. rope_k_repeat (bool): Flag to repeat query RoPE to match key length for cross-attention to memories. Methods: forward: Applies rotary position encoding and computes attention between query, key, and value tensors. Examples: >>> rope_attn = RoPEAttention(embedding_dim=256, num_heads=8, rope_theta=10000.0, feat_sizes=(32, 32)) >>> q = torch.randn(1, 1024, 256) >>> k = torch.randn(1, 1024, 256) >>> v = torch.randn(1, 1024, 256) >>> output = rope_attn(q, k, v) >>> print(output.shape) torch.Size([1, 1024, 256]) g@F) r) rope_theta rope_k_repeat feat_sizesct||i|tt|j|j z||_|j |d|d}||_||_y)zYInitialize RoPEAttention with rotary position encoding for enhanced positional awareness.)rythetarr end_xend_yN) rrrr internal_dimr compute_cis freqs_cisr)rrrrargskwargsrrs rrzRoPEAttention.__init__se $)&)"#4$:K:Kt~~:]eop$$:a= 1 $N "*rc|j|}|j|}|j|}|j||j}|j||j}|j||j}t j |jdx}}|jj|j|_|jjd|jdk7r1|j||j|j|_|jd|jdk7r|jsJ|jd|z }t||ddddd|f|j|j\}|ddddd|f<tj |||}|j#|}|j%|}|S)z[Apply rotary position encoding and compute attention between query, key, and value tensors.rVrrN)rrepeat_freqs_k)q_projk_projv_proj_separate_headsrrAsqrtr!rr^devicerrrXrr\scaled_dot_product_attention_recombine_headsout_proj) rqkvnum_k_exclude_ropewh num_k_ropeouts rr)zRoPEAttention.forwards KKN KKN KKN  DNN 3  DNN 3  DNN 3 !''"+&&A**1884 >>   "aggbk 1!--AQ-?BB188LDN 772;!''"+ %%% %%VVBZ"44 "2 aKZK nn-- # 1Q;J;  ,,Q15##C(mmC  r)rr*rr+rtuple[int, int])r) r torch.Tensorrrrrrr@r-rr.r4s@rrr|s@4$#&. ++ + $ + !!rrc||S|jdddd}||}|jdddd}|r||}|S)z^Apply pooling and optional normalization to a tensor, handling spatial dimension permutations.rr}r r;)r~)r&poolrns rdo_poolrsN | !Q1A QA !Q1A G Hrc>eZdZdZ d dfd ZddZxZS)MultiScaleAttentionaImplements multiscale self-attention with optional query pooling for efficient feature extraction. This class provides a flexible implementation of multiscale attention, allowing for optional downsampling of query features through pooling. It's designed to enhance the model's ability to capture multiscale information in visual tasks. Attributes: dim (int): Input dimension of the feature map. dim_out (int): Output dimension of the attention module. num_heads (int): Number of attention heads. scale (float): Scaling factor for dot-product attention. q_pool (nn.Module | None): Optional pooling module for query features. qkv (nn.Linear): Linear projection for query, key, and value. proj (nn.Linear): Output projection. Methods: forward: Applies multiscale attention to the input tensor. Examples: >>> import torch >>> from torch import nn >>> x = torch.randn(1, 64, 64, 256) >>> msa = MultiScaleAttention(dim=256, dim_out=256, num_heads=8) >>> output = msa(x) >>> print(output.shape) torch.Size([1, 64, 64, 256]) ct|||_||_||_||z}|dz|_||_tj||dz|_ tj|||_ y)z]Initialize multiscale attention with optional query pooling for efficient feature extraction.r}N) rrrydim_outrscaleq_poolrroqkvr)rryrrrhead_dimrs rrzMultiScaleAttention.__init__si  "i't^  99S'A+.IIgw/ rc|j\}}}}|j|j|||zd|jd}t j |d\}}} |j r[t|j|||d|j }|jdd\}}|j|||z|jd}tj|jdd|jdd| jdd}|jdd}|j|||d}|j|}|S)zVApply multiscale attention with optional query pooling to extract multiscale features.r}r;r ) r!rreshaperrtunbindrrr\r transposer) rr&BHWrTrrrrs rr)zMultiScaleAttention.forward sWW 1ahhqk!!!QUAt~~rB,,sA&1a ;; !Q2. B! 0 00 0  0*rrcpeZdZdZddddej df d fd Zd dZxZS) MultiScaleBlockaA multiscale attention block with window partitioning and query pooling for efficient vision transformers. This class implements a multiscale attention mechanism with optional window partitioning and downsampling, designed for use in vision transformer architectures. Attributes: dim (int): Input dimension of the block. dim_out (int): Output dimension of the block. norm1 (nn.Module): First normalization layer. window_size (int): Size of the window for partitioning. pool (nn.Module | None): Pooling layer for query downsampling. q_stride (tuple[int, int] | None): Stride for query pooling. attn (MultiScaleAttention): Multi-scale attention module. drop_path (nn.Module): Drop path layer for regularization. norm2 (nn.Module): Second normalization layer. mlp (MLP): Multi-layer perceptron module. proj (nn.Linear | None): Projection layer for dimension mismatch. Methods: forward: Processes input tensor through the multiscale block. Examples: >>> block = MultiScaleBlock(dim=256, dim_out=512, num_heads=8, window_size=7) >>> x = torch.randn(1, 56, 56, 256) >>> output = block(x) >>> print(output.shape) torch.Size([1, 28, 28, 512]) @r LayerNormNrc Xt |t|trt t t |d}||_||_|||_ | |_ d|c|_ |_ |jrt j||d|_ t||||j|_|dkDr t!|nt j"|_|||_t)|t+||z|d| |_||k7rt j.|||_yy) z\Initialize a multiscale attention block with window partitioning and optional query pooling.rirjNF)r<r= ceil_mode)rrrr;r)rrrIstrrgetattrrryrnorm1 window_sizerq_stride MaxPool2drattnrrwrxnorm2rr@rror) rryrr mlp_ratiorx norm_layerr act_layerrrs rrzMultiScaleBlock.__init__Ds  j# & Z!8dCJ _ &#'  4= == (V[\DI'  99   1:C),R[[](   )# $    '> #w/DI rc|}|j|}|j|jk7r%t|j ||j }|j }|dkDr-|jd|jd}}t||\}}|j|}|jrN|j |jdz}|jdd\}}|||zz |z}|||zz |z}||z||zf}|j dkDrt||f}||j|z}||j|j|j|z}|S)z]Process input through multiscale attention and MLP, with optional windowing and downsampling.rr r;r})rryrrrrrr!rrrrrxrr) rr&shortcutrrrpad_hwpad_hpad_ws rr)zMultiScaleBlock.forwardtsV JJqM 88t|| #tyy|TYY7H&& ?771:qwwqzqA(K8IAv IIaL ==**dmmA.>>K>>!A&DAq 1{?2kAE 1{?2kAE%iU+F   a "1k6Aq6BA t~~a( ( txx 1 67 7r)ryr@rr@rr@rr*rxr*rznn.Module | strrrbrrarr@rrcr4s@rrr&sD&1+/%'WW.0 .0.0 .0  .0  .0$.0).0#.0.0`!rrceZdZdZ d dfd Zd dZejd dZeZ ejd dZ ejd dZ xZ S) PositionEmbeddingSineaA module for generating sinusoidal positional embeddings for 2D inputs like images. This class implements sinusoidal position encoding for 2D spatial positions, which can be used in transformer-based models for computer vision tasks. Attributes: num_pos_feats (int): Number of positional features (half of the embedding dimension). temperature (int): Temperature parameter for the sinusoidal functions. normalize (bool): Whether to normalize the positional embeddings. scale (float): Scaling factor for the embeddings when normalize is True. cache (dict): Cache for storing precomputed embeddings. Methods: _encode_xy: Encodes 2D positions using sine and cosine functions. encode_boxes: Encodes box coordinates and dimensions into positional embeddings. encode_points: Encodes 2D point coordinates with sinusoidal positional embeddings. forward: Generates sinusoidal position embeddings for 2D inputs. Examples: >>> pos_emb = PositionEmbeddingSine(num_pos_feats=128) >>> x = torch.randn(1, 3, 224, 224) >>> embeddings = pos_emb(x) >>> print(embeddings.shape) torch.Size([1, 256, 224, 224]) ct||dzdk(sJd|dz|_||_||_| |s t d|dt jz}||_i|_ y)z>Initialize sinusoidal position embeddings for 2D image inputs.r;rzExpecting even model widthNz+normalize should be True if scale is passed) rr num_pos_feats temperature normalize ValueErrorrApircache)rrrrrrs rrzPositionEmbeddingSine.__init__sy q A%C'CC%*a/&"  YJK K =KE  rct|t|k(r"|j|jcxk(rdk(sJJ||jz}||jz}tj|j tj |j}|jd|dzz|j z z}|dddf|z }|dddf|z }tj|dddddfj|dddddfjfdjd}tj|dddddfj|dddddfjfdjd}||fS)zVEncode 2D positions using sine/cosine functions for transformer positional embeddings.r r_rr;Nrry) rMr"rrtarangerfloat32rrstacksincosflatten)rr&yx_embedy_embeddim_tpos_xpos_ys r _encode_xyz PositionEmbeddingSine._encode_xysa1vQAFFaff$9$999$999djj.djj. T//u}}QXXV  Q%1*%58J8J%JK4 5(4 5( U1add7^//15ADqD>3E3E3GHaPXXYZ[ U1add7^//15ADqD>3E3E3GHaPXXYZ[e|rc~|j||\}}tj|||dddf|dddffdS)zOEncode box coordinates and dimensions into positional embeddings for detection.Nr r)rrtcat)rr&rrrrrs r encode_boxesz"PositionEmbeddingSine.encode_boxessBq!, uyy%!T' AagJ?QGGrc|j|j|jc\}}\}}\}} ||k(r||k(r ||k(r|| k(sJ|j|j|j\} } | j||d| j||d} } t j | | |dddddffdS)z>Encode 2D points with sinusoidal embeddings and append labels.rNr;r)r!rrrrtr ) rr&rlabelsbxnxbynyblnlrrs r encode_pointsz#PositionEmbeddingSine.encode_pointss()ww$R(2rHRRxB"HrbBh>>qyy{AIIK@ u}}RR0%--B2Kuyy%q!Tz(:;CCrc ^|jd|jdf}||jvr1|j|dj|jddddStjd|jddztj |j jdddj|jdd|jd}tjd|jddztj |j jdddj|jd|jdd}|jrDd}||ddddddf|zz |jz}||ddddddf|zz |jz}tj|jtj |j }|jd|dzz|jz z}|dddddddf|z }|dddddddf|z }tj|dddddddddfj|dddddddddfjfd jd }tj|dddddddddfj|dddddddddfjfd jd }tj ||fd j#dd dd} | d|j|<| S) zBGenerate sinusoidal position embeddings for 2D inputs like images.rVrNrr rrir;r8rr})r!rrepeatrtrrrviewrrrrrrrrr r~) rr& cache_keyrrrkrrrposs rr)zPositionEmbeddingSine.forwardsWWR[!''"+.  "::i(.55aggaj!QJ J LLAGGBK!O5== R T!R^ VAGGAJ1772; /  LLAGGBK!O5== R T!Q^ VAGGAJ Q /  >>CBC!3c!9:TZZGGArs!3c!9:TZZGG T//u}}QXXV  Q%1*%58J8J%JK1a &.1a &. U1aADqD=1557q!Q1}9M9Q9Q9STZ[\ddefg U1aADqD=1557q!Q1}9M9Q9Q9STZ[\ddefgiiA.66q!QB #A 9 r)i'TN)rr@rr@rr+r float | None)r&rrrr-z!tuple[torch.Tensor, torch.Tensor]) r&rrrrrrrr-r)r&rrrr rr-r)r&rr-r) r/r0r1r2rrrtno_gradr encoderr)r3r4s@rrrs:!"    * U]]_HH FU]]_DDU]]_rrc>eZdZdZddfd ZddZd dZd dZxZS) PositionEmbeddingRandomaQPositional encoding using random spatial frequencies. This class generates positional embeddings for input coordinates using random spatial frequencies. It is particularly useful for transformer-based models that require position information. Attributes: positional_encoding_gaussian_matrix (torch.Tensor): A buffer containing random values for encoding. Methods: _pe_encoding: Positionally encodes points that are normalized to [0,1]. forward: Generates positional encoding for a grid of the specified size. forward_with_coords: Positionally encodes points that are not normalized to [0,1]. Examples: >>> pe = PositionEmbeddingRandom(num_pos_feats=64) >>> size = (32, 32) >>> encoding = pe(size) >>> print(encoding.shape) torch.Size([128, 32, 32]) ct|||dkrd}|jd|tjd|fztj ddtj j_y)zHInitialize random spatial frequency position embedding for transformers.Nrg?#positional_encoding_gaussian_matrixr;F) rrregister_bufferrtrandnuse_deterministic_algorithmsbackendscudnn deterministic)rrrrs rrz PositionEmbeddingRandom.__init__ sf  =ESLE BEEKKYZ\iXjLkDkl **51-2*rcd|zdz }||jz}dtjz|z}tjtj |tj |gdS)zEEncode normalized [0,1] coordinates using random spatial frequencies.r;r rr)r nprrtr rr)rcoordss r _pe_encodingz$PositionEmbeddingRandom._pe_encoding+s[Va$BBBRUUV#yy%))F+UYYv->?RHHrcn|\}}tj||f|jj|jj}|j ddz }|j ddz }||z }||z }|j tj||gd}|jdddS)zIGenerate positional encoding for a grid using random spatial frequencies.)rr_rrg?r rr;) rtrur rr_cumsumr*rr~)rrXrrgridrrpes rr)zPositionEmbeddingRandom.forward4s1zz F;;BB::@@  ++!+$s*++!+$s*A+A+   u{{GW+=2F Gzz!Q""rc|j}|dddddf|dz |dddddf<|dddddf|dz |dddddf<|j|S)z_Positionally encode input coordinates, normalizing them to [0,1] based on the given image size.Nrr )cloner*)r coords_input image_sizer)s rforward_with_coordsz+PositionEmbeddingRandom.forward_with_coordsDse##% Aq/JqM9q!Qw Aq/JqM9q!Qw  ((r)@N)rr@rrr-r)r)rr-r)rXrr-r)r1rr2rr-r) r/r0r1r2rr*r)r3r3r4s@rrr s* 3I# )rrceZdZdZddej ej ddddf d fd Zd dZxZ S) BlockaTransformer block with support for window attention and residual propagation. This class implements a transformer block that can use either global or windowed self-attention, followed by a feed-forward network. It supports relative positional embeddings and is designed for use in vision transformer architectures. Attributes: norm1 (nn.Module): First normalization layer. attn (REAttention): Self-attention layer with optional relative positional encoding. norm2 (nn.Module): Second normalization layer. mlp (MLPBlock): Multi-layer perceptron block. window_size (int): Size of attention window. If 0, global attention is used. Methods: forward: Processes input through the transformer block. Examples: >>> import torch >>> block = Block(dim=256, num_heads=8, window_size=7) >>> x = torch.randn(1, 56, 56, 256) >>> output = block(x) >>> print(output.shape) torch.Size([1, 56, 56, 256]) rTFrNc t ||||_t|||||| dk(r| n| | f|_|||_t |t||z||_| |_ y)aInitialize a transformer block with optional window attention and relative positional embeddings. This constructor sets up a transformer block that can use either global or windowed self-attention, followed by a feed-forward network. It supports relative positional embeddings and is designed for use in vision transformer architectures. Args: dim (int): Number of input channels. num_heads (int): Number of attention heads in the self-attention layer. mlp_ratio (float): Ratio of mlp hidden dimension to embedding dimension. qkv_bias (bool): If True, adds a learnable bias to query, key, value projections. norm_layer (type[nn.Module]): Type of normalization layer to use. act_layer (type[nn.Module]): Type of activation function to use in the MLP block. use_rel_pos (bool): If True, uses relative positional embeddings in attention. rel_pos_zero_init (bool): If True, initializes relative positional parameters to zero. window_size (int): Size of attention window. If 0, uses global attention. input_size (tuple[int, int] | None): Input resolution for calculating relative positional parameter size. r)rqkv_bias use_rel_posrel_pos_zero_init input_size)rrrqN) rrr REAttentionrrr r@rr) rryrrr8rrr9r:rr;rs rrzBlock.__init__fsv> _  #/%0A%5zK;U    _ #s3?7KQZ[&rcx|}|j|}|jdkDr7|jd|jd}}t||j\}}|j |}|jdkDrt ||jf}||z}||j |j|zS)zfProcess input through transformer block with optional windowed self-attention and residual connection.rr r;)rrr!rrrrr)rr&rrrrs rr)z Block.forwards JJqM   a 771:qwwqzqA(D,<,<=IAv IIaL   a "1d&6&6AGA qL488DJJqM***r)ryr@rr@rr*r8r+rrarrar9r+r:r+rr@r;rbr-rr) r/r0r1r2rrrdrr)r3r4s@rr6r6Ls:&(ll%'WW!"&-1-' -'-' -'  -' $ -'#-'-' -'-'+-' -'^+rr6cReZdZdZ d dfd ZddZxZS)r<aVRelative Position Attention module for efficient self-attention in transformer architectures. This class implements a multi-head attention mechanism with relative positional embeddings, designed for use in vision transformer models. Attributes: num_heads (int): Number of attention heads. scale (float): Scaling factor for attention computation. qkv (nn.Linear): Linear projection for query, key, and value. proj (nn.Linear): Output projection layer. use_rel_pos (bool): Whether to use relative positional embeddings. rel_pos_h (nn.Parameter): Relative positional embeddings for height dimension. rel_pos_w (nn.Parameter): Relative positional embeddings for width dimension. Methods: forward: Applies multi-head attention with optional relative positional encoding to input tensor. Examples: >>> attention = REAttention(dim=256, num_heads=8, input_size=(32, 32)) >>> x = torch.randn(1, 32, 32, 256) >>> output = attention(x) >>> print(output.shape) torch.Size([1, 32, 32, 256]) ct|||_||z}|dz|_t j ||dz||_t j |||_||_|jrx|Jdt jtjd|dzdz ||_ t jtjd|dzdz ||_ yy) a2Initialize a Relative Position Attention module for transformer-based architectures. This module implements multi-head attention with optional relative positional encodings, designed specifically for vision tasks in transformer models. Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. qkv_bias (bool): If True, adds a learnable bias to query, key, value projections. use_rel_pos (bool): If True, uses relative positional encodings. rel_pos_zero_init (bool): If True, initializes relative positional parameters to zero. input_size (tuple[int, int] | None): Input resolution for calculating relative positional parameter size. Required if use_rel_pos is True. rr})biasNzBInput size must be provided if using relative positional encoding.r;rr )rrrrrrorrr9rsrtzeros rel_pos_h rel_pos_w) rryrr8r9r:r;rrs rrzREAttention.__init__s. ")#t^ 99S#'9IIc3' &   ) o+o o)\\%++a*Q-6G!6KX*VWDN\\%++a*Q-6G!6KX*VWDN rc |j\}}}}|j|j|||zd|jdj ddddd}|jd||jz||zdj d\}}} ||j z|jddz} |jr(t| ||j|j||f||f} | jd} | | zj||j||dj dddddj|||d}|j|S) zVApply multi-head attention with optional relative positional encoding to input tensor.r}rr;rr r8rVr)r!rrrr~rrrr9rrBrCsoftmaxrr) rr&rrrrTrrrrrs rr)zREAttention.forwards7WW 1ahhqk!!!QUAt~~rBJJ1aQRTUWXY++aT^^!3QUB?FFqI1aDJJ!++b""55   )$4>>4>>TUWXSY\]_`[abD|||# AXOOAt~~q!R 8 @ @Aq!Q O W WXY[\^_ac dyy|r)TFTN)ryr@rr@r8r+r9r+r:r+r;rbr-rrr.r4s@rr<r<sp8!"&-1$X $X$X $X  $X  $X+$X $XLrr<cTeZdZdZ d dfd ZddZxZS) PatchEmbedaImage to Patch Embedding module for vision transformer architectures. This module converts an input image into a sequence of patch embeddings using a convolutional layer. It is commonly used as the first layer in vision transformer architectures to transform image data into a suitable format for subsequent transformer blocks. Attributes: proj (nn.Conv2d): Convolutional layer for projecting image patches to embeddings. Methods: forward: Applies patch embedding to the input tensor. Examples: >>> patch_embed = PatchEmbed(kernel_size=(16, 16), stride=(16, 16), in_chans=3, embed_dim=768) >>> x = torch.randn(1, 3, 224, 224) >>> output = patch_embed(x) >>> print(output.shape) torch.Size([1, 768, 14, 14]) cbt|tj|||||||_y)aInitialize the PatchEmbed module for converting image patches to embeddings. This module is typically used as the first layer in vision transformer architectures to transform image data into a suitable format for subsequent transformer blocks. Args: kernel_size (tuple[int, int]): Size of the convolutional kernel for patch extraction. stride (tuple[int, int]): Stride of the convolutional operation. padding (tuple[int, int]): Padding applied to the input before convolution. in_chans (int): Number of input image channels. embed_dim (int): Dimensionality of the output patch embeddings. bias (bool): Whether to include a bias term in the convolutional layer. )r<r=r>r@N)rrrrGr)rr<r=r>in_chansrNr@rs rrzPatchEmbed.__init__s-, IIh {SYcjquv rcH|j|jddddS)zQCompute patch embedding by applying convolution and transposing resulting tensor.rr;r}r )rr~r`s rr)zPatchEmbed.forward(s!yy|##Aq!Q//r)r9r9rL)rrr}iT)r<rr=rr>rrJr@rNr@r@r+r-rrr.r4s@rrHrHsr,(0"*#)w$w w! w  w  ww w40rrHr)r&rrrrnrr-r), __future__rrrA functoolsrnumpyr(rttorch.nn.functionalr functionalr\rultralytics.nn.modulesrrr transformerr r r utilsrrrrrModulerr6rfrrrrrrrrrr6r<rHrrrWs,"  ==KKtt!ryy!DGbiiGTRbiiRj-BII-`4\34\n>->BIIIX  L"))L^obiiodoBIIod?)bii?)DX+BIIX+vO"))Od1010r