Bi& dZddlZddlZddlZddlmZddlmZddlZddlm Z ddl m Z m Z m Z mZmZmZmZddlmZmZmZmZmZdd lmZmZmZmZmZmZdd lm Z m!Z!e jDe#Z$ed Z%d e&de'fdZ(de&de)e&e&ffdZ*dde+e,zde'de&fdZ-ddedzde'de&fdZ. dde/e e zde'de'de/e&fdZ0 dde1de/e&de&de+de+de'ddfd Z2e%jgd!gd"#ddde jhfd$ee&ejjd% fd&ee'ejld'd( fd)ee'ejld*d+ fd,ed-eddf d.Z7de&de'fd/Z8e%jgd0gd1#dddddde jhfdee&dzejjd2 fdee'ejld3d4d5 fd6ee'ejld7d8 fd9ee'ejld:d;d< fd=ed,ed-eddfd>Z9d?e&dzd=e&dzde'd6e'd9e'd,e&dzddfd@Z:de&de'd6e'd9e'd,e&dzddf dAZ;e%jgdBdCdDg#de jhfd$ee&ejjdE fd,ed-eddfdFZe%jgd]gd^#de jhfd_ee&ejjd` fdaee&ejjdb fd,ed-eddf dcZ?e%jgddgde# ddfee&dzejjdg fdhee&dzejjdi fdGee'ejldj fdkee'ejldldm fdnee'ejldodp fdqee&dzejldr fdsee&dzejldt fdUee'ejldVdu fdXee/e&dzejldv fdZee/e&dzejldw fdxee&dzejldy fdzee'ejld{d| fd}ee'ejld~d fdee'ejlddd fdee'ejlddd fd,eddf"dZ@e%jgdgd#dde jhfdee&ejjd fdee&dzejjd fd,ed-eddf dZAy)z7Contains commands to interact with buckets via the CLI.N)datetime) Annotated)logging) BUCKET_PREFIX BucketFile BucketFolder FilterMatcher_is_bucket_path_parse_bucket_path_split_bucket_id_and_prefix)SoftTemporaryDirectory StatusLineare_progress_bars_disableddisable_progress_barsenable_progress_bars)FormatWithAutoOpt SearchOptTokenOptapi_object_to_dict get_hf_api typer_factory)OutputFormatWithAutooutz"Commands to interact with buckets.)helppathreturnc$|jdS)Nzhf://) startswith)rs ]/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/huggingface_hub/cli/buckets.py _is_hf_handler!;s ??7 ##argumentc|jtr t|S t|S#t$rt d|dtdwxYw)zParse a bucket argument accepting both 'namespace/name(/prefix)' and 'hf://buckets/namespace/name(/prefix)'. Returns: tuple: (bucket_id, prefix) where bucket_id is "namespace/bucket_name" and prefix may be empty string. zInvalid bucket argument: -. Must be in format namespace/bucket_name or znamespace/bucket_name)rrr r ValueError)r#s r _parse_bucket_argumentr'?sb =)!(++ *844  'z2 /!6 8   s - A Fsizehuman_readablecz|s t|SdD]$}|dkr|dk(r |d|cS|dd|cS|dz}&|ddS)zFormat a size in bytes.)BKBMBGBTBir+ z.1fz PB)str)r(r)units r _format_sizer3Psj 4y- $;s{q''3Zq' '    3Zs r"mtimecP|y|r|jdS|jdS)z0Format mtime datetime to a readable date string.z %b %d %H:%Mz%Y-%m-%d %H:%M:%S)strftime)r4r)s r _format_mtimer8^s, }~~m,, >>- ..r"itemsquietc i}|D]e}|jjd}|}|ddD]}||vrdii||<||d}|d}t|tr ||vsWdii||<_d|i||<gd} d} d} |s{|D]i}t|tst |j |} t| t| } t|j|} t| t| } k| dkDr| dz| z} g}t||d| | | |S) aBuild a tree representation of files and directories. Produces ASCII tree with size and date columns before the tree connector. When quiet=True, only the tree structure is shown (no size/date). Args: items: List of BucketFile/BucketFolder items human_readable: Whether to show human-readable sizes and short dates quiet: If True, show only the tree structure without sizes/dates Returns: List of formatted tree lines /N __children____item__rr6 prefix_widthmax_size_widthr)) rsplit isinstancerrr3r(maxlenr8r4 _render_tree)r9r)r:treeitempartscurrentpart final_partrBrCmax_date_widthsize_strdate_strliness r _build_treerSgsS&D 5 $#2J 4D7"!/ 4 dmN3G 4 2Y dL )('5r&: ##-t"4GJ  5 LNN  DD$ +' >B!$^S]!C(^D!$^S]!C  D A )A->LE   !%%  Lr"noderRindentrBrCc .t|j}t|D]\}\}} |t|dz k(} | rdnd} d| v} | j di} |dkDrz| rd|z}nQ| j d}|9t |j |}t|j|}|d |d d |}nd|z}|j|d || || rd nd n|j|| || rd nd | s|| rd ndz}t| |||||y)z:Recursively render a tree structure with size+date prefix.ru └── u ├── r>rr0r?N>r6 r<z u│ rA) sortedr9 enumeraterGgetr3r(r8r4appendrH)rTrRrUrBrCr)r9inamevalueis_last connectoris_dirchildrenprefixrJrPrQ child_indents r rHrHsL 4::< E%e,=D%s5zA~%$+L 5(99^R0 ! |+yy,#+DII~FH,TZZHH (>*: (:;2hZHF </F LLF82fXi[fsRT>UV W LLF8I;tfFS4KL M !wVHEL )--  1r"create)zhf buckets create my-bucketz hf buckets create user/my-bucketz-hf buckets create hf://buckets/user/my-bucketz*hf buckets create user/my-bucket --privatez+hf buckets create user/my-bucket --exist-ok)r^examples bucket_idzTBucket ID: bucket_name, namespace/bucket_name, or hf://buckets/namespace/bucket_nameprivatez --privatezCreate a private bucket.exist_okz --exist-okz3Do not raise an error if the bucket already exists.tokenformatct|}|jtr3 t|\}}|r t j d|dtd|}|j||r|nd|} tjd| j| jy#t$r#}t j t|d}~wwxYw) zCreate a new bucket.rkNz-Cannot specify a prefix for bucket creation: . Use namespace/bucket_name or namespace/bucket_name.)rirjzBucket created)handleurl) rrrr'r&typer BadParameterr1 create_bucketrresultrqrr) rhrirjrkrlapi parsed_idrde bucket_urls r rfrfsF 5 !CM* - 6y A Iv $$? {K11>?UW  """#J JJ (9(9z~~N -$$SV, , -sB C#CCcd|jtr|ttd}d|vS|}d|vS)zFCheck if argument is a bucket ID (namespace/name) vs just a namespace.Nr<)rrrG)r#rs r _is_bucket_idr| s?=)M*,- $; $;r"z list | ls) zhf buckets listzhf buckets list huggingfacez$hf buckets list --search "my-prefix"zhf buckets list user/my-bucketz!hf buckets list user/my-bucket -Rz!hf buckets list user/my-bucket -hz%hf buckets list user/my-bucket --treez(hf buckets list user/my-bucket --tree -hz+hf buckets list hf://buckets/user/my-bucketz%hf buckets list user/my-bucket/sub -RzyNamespace (user or org) to list buckets, or bucket ID (namespace/bucket_name(/prefix) or hf://buckets/...) to list files.z--human-readablez-hz$Show sizes in human readable format.as_treez--treez3List files in tree format (only for listing files). recursivez --recursivez-Rz0List files recursively (only for listing files).searchc|duxr t|}|r(|tjdt|||||yt ||||||y)zList buckets or files in a bucket. When called with no argument or a namespace, lists buckets. When called with a bucket ID (namespace/bucket_name), lists files in the bucket. Nz'Cannot use --search when listing files.)r#r)r}r~rk) namespacerr)r}r~rk)r|rsrt _list_files _list_buckets)r#r)r}r~rrkrl is_file_modes r list_cmdrsgt4'CM(,CL  $$%NO O)   )  r"rc  |rtjd|rtjd|8|jtr#|t td}|j d}t |}|j||Dcgc]T}|j|j|rt|j|n |j|j|jdV}}tj|ddd  ycc}w) zList buckets in a namespace.z'Cannot use --tree when listing buckets.z,Cannot use --recursive when listing buckets.Nr<rn)rr)idrir( total_files created_atright)r(r) alignments)rsrtrrrGrstripr list_bucketsrrir3r(rrrtable) rrr)r}r~rkrwbucketr9s r rres  !JKK  !OPP!5!5m!Dc-023 $$S) 5 !C&&6&J   ))~~AOLn=U[U`U`!-- ++   E IIe IJ s ADc |r6tjtjk(rt j dt |} t|\}}t|j||xsd|} | stjdytd| D} |rAtjtjk(} t| || D] } t!| nZtjtjk(r9t!tj"| D cgc] } t%| c} d ntjtjk(rF| D]@} t'| t(rt!| j*d ,t!| j*Bn| D]} t'| t(r7t-| j.|}t!d d d |dd | j*d Jt1| j2|}t-| j4|}t!|d d |dd | j*|s| rtj6dyyy#t$r#}t j t|d}~wwxYwcc} w)zList files in a bucket.z%Cannot use --tree with --format json.rnNrdr~z(empty)c3<K|]}t|tywN)rEr).0rJs r z_list_files..sKT*T<8Ks)r)r:r@)rUr<r6z>12rXz>19z!Use -R to list files recursively.)rmoderjsonrsrtrr'r&r1listlist_bucket_treetextanyr:rSprintdumpsrrErrr8 uploaded_atr3r(r4hint)r#r)r}r~rkrwrhrdryr9has_directoriesr:linerJ mtime_strrPs r rrs,3883888  !HII 5 !C)28< 6   >T   E  KUKKO0666nER D $K  ).. . djjuEt,T2EaPQ )// / !D$ - 1o&dii  ! GD$ -)$*:*:NK C9S/DII;a@A' >B)$**nE #b3r$))EF G  45)9U )  Q(()0FsIJ  J!I??Jinfozhf buckets info user/my-bucketz+hf buckets info hf://buckets/user/my-bucketzFBucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_namect|} t|\}}|j |}tj|dy#t$r#}tjt |d}~wwxYw)zGet info about a bucket.rnNr)id_key) rr'r&rsrtr1 bucket_infordict)rhrkrlrwrx_ryrs r rrsf$ 5 !C)-i8 1__Y 'FHHVD! )  Q(()sA A1A,,A1delete)z hf buckets delete user/my-bucketz-hf buckets delete hf://buckets/user/my-bucketz&hf buckets delete user/my-bucket --yesz-hf buckets delete user/my-bucket --missing-okyesz--yesz-yzSkip confirmation prompt. missing_okz --missing-okz3Do not raise an error if the bucket does not exist.c|jtr4 t|\}}|r t j d|dtd|}n$d|vr t j d|dtdtjd|d | t| }|j|| tjd |y#t$r#}t j t |d}~wwxYw)zDelete a bucket. This deletes the entire bucket and all its contents. Use `hf buckets rm` to remove individual files. Nz-Cannot specify a prefix for bucket deletion: rorpr<zInvalid bucket ID: r%z(Are you sure you want to delete bucket ''?rrn)rzBucket deleted)rh) rrr'r&rsrtr1rconfirmr delete_bucketrv) rhrrrkrlrxrdryrws r rrsLM* - 6y A Iv $$? {K11>?UW  I   !)-;;H/I_ a  KK:9+RHcR 5 !CiJ7JJ95% -$$SV, , -sC C, C''C,z remove | rm)z)hf buckets remove user/my-bucket/file.txtz2hf buckets rm hf://buckets/user/my-bucket/file.txtz.hf buckets rm user/my-bucket/logs/ --recursivez:hf buckets rm user/my-bucket --recursive --include "*.tmp"z8hf buckets rm user/my-bucket/data/ --recursive --dry-runzBucket path: namespace/bucket_name/path or hf://buckets/namespace/bucket_name/path. With --recursive, namespace/bucket_name is also accepted to target all files.z0Remove files recursively under the given prefix.dry_runz --dry-runz8Preview what would be deleted without actually deleting.includezQInclude only files matching pattern (can specify multiple). Requires --recursive.excludezLExclude files matching pattern (can specify multiple). Requires --recursive.c t|\}|dk(r|stjddd|s|r|stjdt |} |r:t tjtjk(} | jd g} | j|jd xsdd D]B} t| ts| j!| | jd t#| dD| j%d t#| d|s|r9t'||}| Dcgc] }|j)|j*s|"}}n| }|Dcgc]}|j*}}t-d|D}t/|d }|stj0dyt#|d|}|sK|sItj0dj3d|Dtj4d|ddd|rGtj0dj3fd|Dtj0d|dy| j7|tj8d|dd t#||!y|jd }|stjd"|r7tj0d#t:d |tj0d$ytj4d%|d&d|| j7|gtj8d'|(y#t$r#} tjt | d} ~ wwxYwcc}wcc}w))zbRemove files from a bucket. To delete an entire bucket, use `hf buckets delete` instead. Nr6z?No file path specified. To remove files, provide a path (e.g. 'zd/FILE') or use --recursive to remove all files. To delete the entire bucket, use `hf buckets delete z`.z,--include and --exclude require --recursive.rn)enabledzListing files from remoter<TrzListing files from remote (z files))include_patternsexclude_patternsc34K|]}|jywr)r()rfs r rzremove..s7A7s)r)zNo files to remove.z file(s) totaling  c3&K|] }d| yw)rXN)rrs r rzremove..sBtD6{BszRemove z from 'rFrc3:K|]}dtd|yw)delete: r<N)r)rrrhs r rzremove..s#cPT- 1TFKcsz (dry run) z would be removed.)rzRemoved ')rh files_deletedr(zFile path cannot be empty.rz"(dry run) 1 file would be removed.zRemove 'z' from 'z File removed)rrh)r'r&rsrtr1rrrrrhumanupdaterrrErr\rGdoner matchesrsumr3rjoinrbatch_bucket_filesrvr)r#r~rrrrrkrlrdryrwstatus all_filesrJmatcherr matched_files file_paths total_sizerP count_label file_pathrhs @r remover!sB|)28< 6|I   k"DDM;b R  7I  !OPP 5 !CCHH0D0J0J$JK 12&( (( ==%-)  UD $ +  & ;C N;K7ST U  1#i.1AIJ g#WwWG(1M1W__QVV5LQMMM%M&34aff4 4777  4@ HH* + Z));H:F 7 HHTYYBzBB C KK'+gi[C O  HHTYYcXbcc d HHz+.@A B  y< {m79+Q 7j/  MM#& $$%AB B  HHx yk9+F G HH9 :  hyk)B?SI y)= > YGY )  Q(()@N5s)M; M3M3)M8 M0 M++M0move)z/hf buckets move user/old-bucket user/new-bucketz/hf buckets move user/my-bucket my-org/my-bucketzIhf buckets move hf://buckets/user/old-bucket hf://buckets/user/new-bucketfrom_idzMSource bucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_nameto_idzRDestination bucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_namec2t|\}}|r tjd|dtdt|\}}|r tjd|dtdt |}|j ||t jd||y)z2Move (rename) a bucket to a new name or namespace.z)Cannot specify a prefix for bucket move: rorprn)rrz Bucket movedN)r'rsrtrr move_bucketrrv) rrrkrlparsed_from_id from_prefix parsed_to_id to_prefixrws r rrs4#9"ANK  7yA--:O;Q S  5U;L)  7w?--:O;Q S  5 !COON,O?JJ~~\Jr"sync)z2hf buckets sync ./data hf://buckets/user/my-bucketz2hf buckets sync hf://buckets/user/my-bucket ./dataz;hf buckets sync ./data hf://buckets/user/my-bucket --deletez^hf buckets sync hf://buckets/user/my-bucket ./data --include "*.safetensors" --exclude "*.tmp"zIhf buckets sync ./data hf://buckets/user/my-bucket --plan sync-plan.jsonlz'hf buckets sync --apply sync-plan.jsonlzhf buckets cp my-config.json hf://buckets/user/my-bucket/logs/zKhf buckets cp my-config.json hf://buckets/user/my-bucket/remote-config.jsonz7hf buckets cp - hf://buckets/user/my-bucket/config.jsonzVhf buckets cp hf://buckets/user/my-bucket/logs/ hf://buckets/user/archive-bucket/logs/zehf buckets cp hf://datasets/user/my-dataset/processed/ hf://buckets/user/my-bucket/dataset/processed/srczMSource: local file, any hf:// handle (model, dataset, bucket), or - for stdindstz@Destination: local path, bucket hf://... handle, or - for stdoutc t|}t|}|duxr t|}t|}|duxr t|}|dk(} |dk(} |r.|r, |j||tjd||y|s0|s.| s,|t j dt j d| r|st j d| r;|r9t|\} } | d k(s| jd rt j d | r|st j d |s6| s4tjj|rt j d |rt|\}} | d k(s| jd rt j d| jd dd}| rt!}|s t# t%5}tjj'||}|j)|| |fgt+|d5}|j-dx}r=t.j0j2j5||j-dx}r=dddddd|st7y||}nrtjj|s0|jtj8s|jd r!tjj'||}n|}tjj;|}|rtj<|d|j)|| |fgtjd||yy| ret|\}}t.j>j2j-}|jA|||fgtjdd|ytjjC|st j d|t|\}} | d k(r tjjE|}n6| jd r#| tjjE|z}n| }|jA|||fgtjd|tF|d |y#t$r#} t j t| d} ~ wwxYw#1swYmxYw#1swYrxYw#|s t7wwxYw)zCopy files to or from buckets.rnN-Copied)rrz2Missing destination. Provide a bucket path as DST.z;One of SRC or DST must be a bucket path (hf://buckets/...).z+Stdin upload requires a bucket destination.r6r<zAStdin upload requires a full destination path including filename.z"Cannot pipe to stdout for uploads.zNSource must be a file, not a directory. Use `hf buckets sync` for directories.zJSource path must include a file name, not just a bucket or directory path.rr=rbiHT)rj Downloaded)addUploadedstdinzSource file not found: )$rr!r copy_filesr&rsrtr1rrvr endswithosrisdirrsplitrrr rdownload_bucket_filesopenreadsysstdoutbufferwritersepdirnamemakedirsrrisfilebasenamer)rrrkrlrw src_is_hf dst_is_hf src_is_bucket dst_is_bucket src_is_stdin dst_is_stdoutryrrdrhfilenamepbar_was_disabledtmp_dirtmp_pathrchunk local_path parent_dir remote_pathdatas r rrrsQ4 5 !Cc"I46M#$6I#C(MtO<(H--i68:L9MNh-;'(vvj'99e9JJ--33E:()vvj'99e9;;)(*{% s#s||BFF';s||C?PWW\\#x8  4J J6  % %i6:2F1G H JJ|* =#)& !3C!8 ;yy$$& yk/B.CD :74ww~~c"$$'>se%DE E.s3 6 R<''**3/K __S ! 277#3#3C#88K K y[/A.BC :3}oi[+,WXw -$$SV, , -Z;;;;)(*)sVR S$ ASAS S'S$ S$SS S SS!S$$S3)F)FF)rrF)NNFFFNNFNNNFFFFN)B__doc__rrrrtypingrrshuggingface_hubrhuggingface_hub._bucketsrrrr r r r huggingface_hub.utilsr rrrr _cli_utilsrrrrrr_outputrr get_logger__name__logger buckets_clir1boolr!tupler'intfloatr3r8rrSrrHcommandautoArgumentOptionrfr|rrrrrrrrrrr"r r%s >  #/   H %!EF $$$ S U38_ " sU{ D S /D/$/3/!<  \) *<< < #Y <F ) ) 9) ) )  )  ) )X   .   4 9 9-,O g  ,O  +  ,O  F  ,O* +,O, -,O. /,O ,O^CD   0     4 9 9G?  d W  ?    7  ? $  F  %? 2   C  3? B C? D E? F G? H I? ? D KTz K $J K K K  K : K  KF;6;6;6;6 ;6 : ;6  ;6|  (5 4 9 9" Y  " " " "",  .   4 9 9/26 Y  26    ,   26  F  26, -26. /260 12626j   6     4 9 9]BH a  BH   C  BH$    ,   %BH4  K  5BHB S D d  CBHN S D _  OBHZ [BH\ ]BH^ _BH BHJ  * 4 9 9#K  `  #K  e   #K #K #K #K#KV    &          IzP  d ^   zP  d c   zP   B   zP&  N  'zP4  O  5zPB  d  P   CzPN  d  6   OzPZ  H  [zPh S D I  izPt S D I  uzP@ d  ;  AzPL  T  MzPZ  V  [zPh   8  izPx    "   yzPH IzPJ KzP zPD   *  4 9 9sY  ^U^^!p qq sY  d NENN(jkk  sY sY sY sYsYr"