Di,U * d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlmZ d dl m
Z
m Z m Z m
Z
d dlmZ d dlZd dlZddlmZ ddlmZmZmZmZmZ dd lmZ G d
d e Z G d d
e Z G d de Z G d de Z G d de Z de!de"fdZ#ejH ejJ ejL ejN ejP ejR dZ*d de+de,dejZ fdZ.de+dejZ fdZ/de+dejZ fd Z0y) N)Path)AnyCallableOptionalUnion)URLError )_Image_fromarray )_flip_byte_ordercheck_integrity download_and_extract_archiveextract_archiveverify_str_arg)
VisionDatasetc
` e Zd ZdZddgZg dZdZdZg dZe d Z
e d Z e d
Z e d Z
d d
eeef dedee dee dedd f fd
Zd Zd Zd Zdedeeef fdZdefdZ e defd Z e defd Z e de eef fd Z defd Z!d!d Z"defd Z# xZ$S )"MNISTam `MNIST `_ Dataset.
Args:
root (str or ``pathlib.Path``): Root directory of dataset where ``MNIST/raw/train-images-idx3-ubyte``
and ``MNIST/raw/t10k-images-idx3-ubyte`` exist.
train (bool, optional): If True, creates dataset from ``train-images-idx3-ubyte``,
otherwise from ``t10k-images-idx3-ubyte``.
transform (callable, optional): A function/transform that takes in a PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
z.https://ossci-datasets.s3.amazonaws.com/mnist/z!http://yann.lecun.com/exdb/mnist/))train-images-idx3-ubyte.gz f68b3c2dcbeaaa9fbdd348bbdeb94873)train-labels-idx1-ubyte.gz d53e105ee54ea40749a09fcbcd1e9432)t10k-images-idx3-ubyte.gz 9fb629c4189551a2d022fa330f9573f3)t10k-labels-idx1-ubyte.gz ec29112dd5afa0611ce80d1b7f02629cz training.ptztest.pt
z0 - zeroz1 - onez2 - twoz 3 - threez4 - fourz5 - fivez6 - sixz 7 - sevenz 8 - eightz9 - ninec D t j d | j S )Nz%train_labels has been renamed targetswarningswarntargetsselfs \/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/torchvision/datasets/mnist.py train_labelszMNIST.train_labels@ s
=>|| c D t j d | j S )Nz$test_labels has been renamed targetsr r" s r$ test_labelszMNIST.test_labelsE s
<=||r& c D t j d | j S )Nz train_data has been renamed datar r datar" s r$
train_datazMNIST.train_dataJ s
89yyr& c D t j d | j S )Nz test_data has been renamed datar* r" s r$ test_datazMNIST.test_dataO s
78yyr& Nroottrain transformtarget_transformdownloadreturnc ( t | ||| || _ | j r | j \ | _ | _ y |r| j | j s t d | j \ | _ | _ y )N)r1 r2 z;Dataset not found. You can use download=True to download it) super__init__r0 _check_legacy_exist_load_legacy_datar+ r! r3
_check_exists RuntimeError
_load_data)r# r/ r0 r1 r2 r3 __class__s r$ r7 zMNIST.__init__T s EUV
# # %&*&<&<&> #DIt| MMO!!# \] ]"&//"3 4z,MNIST._check_legacy_exist..p s0
KOOBGGLL)>)> E F
s =A )rA rB existsrD all
training_file test_file)r# processed_folder_existss ` r$ r8 zMNIST._check_legacy_existk sN "$''..1F1F"G &
TXTfTfhlhvhvSw
r& c | j r | j n | j }t j t
j j | j | d S )NT) weights_only) r0 rJ rK torchloadrA rB rC rD )r# data_files r$ r9 zMNIST._load_legacy_datat sB +/**D&&$.. zz"'',,t'<'> %..%C ,**62FF{r& c , t | j S r@ )lenr+ r" s r$ __len__z
MNIST.__len__ s 499~ r& c t j j | j | j j
d S )NrawrA rB rC r/ r= __name__r" s r$ rW zMNIST.raw_folder s' ww||DIIt~~'>'>FFr& c t j j | j | j j
d S )N processedrk r" s r$ rD zMNIST.processed_folder s' ww||DIIt~~'>'> LLr& c ` t | j D ci c] \ }}||
c}}S c c}}w r@ ) enumerateclasses)r# i_classs r$ class_to_idxzMNIST.class_to_idx s' +4T\\+BCia CCCs
*c @ t fd j D S )Nc 3 K | ]w \ }}t t j j j t j j t j j
| d y yw)r N)r
rA rB rC rW splitextbasename)rE url_r# s r$ rG z&MNIST._check_exists.. sZ
Q
BGGLL "'':J:J277K[K[\_K`:abc:d e f
s A=B )rI resourcesr" s `r$ r: zMNIST._check_exists s!
..
r& c | j ryt j | j d | j D ]| \ }}g }| j
D ]# }| | } t
|| j || 9 d| d}t | j
| D ] \ }}|d| dt | d z
} t | y# t $ r}|j | Y d}~d}~ww xY w)
z4Download the MNIST data if it doesn't exist already.NTexist_ok)
download_rootfilenamemd5zError downloading z:
zTried z, got:
) r: rA makedirsrW r{ mirrorsr r appendzipstrr; ) r# r r errorsmirrorry eserrs r$ r3 zMNIST.download s
DOOd3 "^^ &MHcF ,,
&
+ 0DOO^flop
& )
#6#&t||V#< ?KFC6&#c(2>>A?"1o% & MM!$ s B?? C#C C#c 0 | j du rdnd}d| S )NTTrainTestSplit: )r0 )r# splits r$
extra_reprzMNIST.extra_repr s! ::-6 r& )TNNFr4 N)%rl
__module__ __qualname____doc__r r{ rJ rK rq propertyr% r( r, r. r r r boolr r r7 r8 r9 r< r` tupler re rh rW rD dictrt r: r3 r
__classcell__r= s @r$ r r s " 9+G
I "MI G (,/3
4CI 4 4 H% 4
#8, 4
4
4.
] sCx , GC G G M# M M Dd38n D D
t
&2!C !r& r c & e Zd ZdZdgZg dZg dZy) FashionMNISTa `Fashion-MNIST `_ Dataset.
Args:
root (str or ``pathlib.Path``): Root directory of dataset where ``FashionMNIST/raw/train-images-idx3-ubyte``
and ``FashionMNIST/raw/t10k-images-idx3-ubyte`` exist.
train (bool, optional): If True, creates dataset from ``train-images-idx3-ubyte``,
otherwise from ``t10k-images-idx3-ubyte``.
transform (callable, optional): A function/transform that takes in a PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
z;http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/))r 8d4fb7e6c68d591d4c3dfef9ec88bf0d)r 25c81989df183df01b3e8a0aad5dffbe)r bef4ecab320f06d8554ea6380940ec79)r bb300cfdad3c16e7a12a480ee83cd310)
z T-shirt/topTrouserPulloverDressCoatSandalShirtSneakerBagz
Ankle bootNrl r r r r r{ rq r& r$ r r s! MMGI yGr& r c & e Zd ZdZdgZg dZg dZy)KMNISTa{ `Kuzushiji-MNIST `_ Dataset.
Args:
root (str or ``pathlib.Path``): Root directory of dataset where ``KMNIST/raw/train-images-idx3-ubyte``
and ``KMNIST/raw/t10k-images-idx3-ubyte`` exist.
train (bool, optional): If True, creates dataset from ``train-images-idx3-ubyte``,
otherwise from ``t10k-images-idx3-ubyte``.
transform (callable, optional): A function/transform that takes in a PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
z-http://codh.rois.ac.jp/kmnist/dataset/kmnist/))r bdb82020997e1d708af4cf47b453dcf7)r e144d726b3acfaa3e44228e80efcd344)r 5c965bf0a639b31b8f53240b1b52f4d7)r 7320c461ea6c1c855c0b718fb2a4b134)
okisutsunahamayarewoNr r r& r$ r r s ??GI KGr& r c e Zd ZdZdZdZdZh dZ ee j e j z Z e
ee e
ee ez
e
ee ez
dg ee j z ee j ee j dZdeeef ded ed
d f fd Zed
efd
Zed
efd Zed
efd Zed
efd Zed
efd Z d Z d
e fdZ ddZ xZ!S )EMNISTaH `EMNIST `_ Dataset.
Args:
root (str or ``pathlib.Path``): Root directory of dataset where ``EMNIST/raw/train-images-idx3-ubyte``
and ``EMNIST/raw/t10k-images-idx3-ubyte`` exist.
split (string): The dataset has 6 different splits: ``byclass``, ``bymerge``,
``balanced``, ``letters``, ``digits`` and ``mnist``. This argument specifies
which one to use.
train (bool, optional): If True, creates dataset from ``training.pt``,
otherwise from ``test.pt``.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
transform (callable, optional): A function/transform that takes in a PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
z4https://biometrics.nist.gov/cs_links/EMNIST/gzip.zip 58c8d27c78d21e728a6bc7b3cc06412e)byclassbymergebalancedlettersdigitsmnist> crr jklmr pr uvwxyzzN/Ar/ r kwargsr4 Nc t |d| j | _ | j | | _ | j | | _ t | |fi | | j | j | _
y )Nr ) r splitsr _training_filerJ
_test_filerK r6 r7 classes_split_dictrq )r# r/ r r r= s r$ r7 zEMNIST.__init__' sb #E7DKK@
! 0 0 7 /
((..tzz: r& c d| dS )N training_.ptr r s r$ r zEMNIST._training_file. s 5'%%r& c d| dS )Ntest_r r r s r$ r zEMNIST._test_file2 s ugS!!r& c F d| j d| j rd S d S )Nzemnist--r0 test)r r0 r" s r$ _file_prefixzEMNIST._file_prefix6 s+ Ag%HII%HIIr& c p t j j | j | j d S )NrT rA rB rC rW r r" s r$ images_filezEMNIST.images_file: * ww||DOO0A0A/BBT-UVVr& c p t j j | j | j d S )NrU r r" s r$ labels_filezEMNIST.labels_file> r r& c V t | j t | j fS r@ )rV r rX r r" s r$ r< zEMNIST._load_dataB s# t / /0/$BRBR2SSSr& c R t d | j | j fD S )Nc 3 2 K | ] }t | y wr@ r
rE rF s r$ rG z'EMNIST._check_exists..F ZT?4(Z rI r r r" s r$ r: zEMNIST._check_existsE $ Zd6F6FHXHX5YZZZr& c | j ryt j | j d t | j
| j | j t j j | j d }t j | D ]H }|j d st t j j || | j J t j | y)z5Download the EMNIST data if it doesn't exist already.NTr} )r r gzipz.gz)r: rA r rW r ry r rB rC listdirendswithr shutilrmtree)r# gzip_folder gzip_files r$ r3 zEMNIST.downloadH s
DOOd3$TXXT__RVRZRZ[ggll4??F; K0 WI!!%( [) DdooV W
k"r& r )"rl r r r ry r r _merged_classessetstringr
ascii_letters _all_classessortedlistascii_lowercaser r r r r r7 staticmethodr r r r r r r< r r: r3 r r s @r$ r r sq & AC
,C
MFaOv}}v';';;?4 >?@7T&"8"8 99v}}%fmm$
;U39 - ;c ;S ;T ; & & & "S " " Jc J J WS W W WS W WT[t [
#r& r c
e Zd ZU dZddddddZddgdd gd
d gd Zeeee eef f e
d
< g dZ d de ee
f dee dedededdf fd
Zedefd Zedefd ZdefdZd Zd dZdede eef fd Zdefd Z xZS ) QMNISTa` `QMNIST `_ Dataset.
Args:
root (str or ``pathlib.Path``): Root directory of dataset whose ``raw``
subdir contains binary files of the datasets.
what (string,optional): Can be 'train', 'test', 'test10k',
'test50k', or 'nist' for respectively the mnist compatible
training set, the 60k qmnist testing set, the 10k qmnist
examples that match the mnist testing set, the 50k
remaining qmnist testing examples, or all the nist
digits. The default is to select 'train' or 'test'
according to the compatibility argument 'train'.
compat (bool,optional): A boolean that says whether the target
for each example is class number (for compatibility with
the MNIST dataloader) or a torch vector containing the
full qmnist information. Default=True.
train (bool,optional,compatibility): When argument 'what' is
not specified, this boolean decides whether to load the
training set or the testing set. Default: True.
download (bool, optional): If True, downloads the dataset from
the internet and puts it in root directory. If dataset is
already downloaded, it is not downloaded again.
transform (callable, optional): A function/transform that
takes in a PIL image and returns a transformed
version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform
that takes in the target and transforms it.
r0 r nist)r0 r test10ktest50kr )zbhttps://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-train-images-idx3-ubyte.gz ed72d4157d28c017586c42bc6afe6370)z`https://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-train-labels-idx2-int.gz 0058f8dd561b90ffdd0f734c6a30e5e4)zahttps://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-test-images-idx3-ubyte.gz 1394631089c404de565df7b7aeaf9412)z_https://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-test-labels-idx2-int.gz 5b5b05890a5e13444e108efe57b788aa)z[https://raw.githubusercontent.com/facebookresearch/qmnist/master/xnist-images-idx3-ubyte.xz 7f124b3b8ab81486c9d8c2749c17f834)zYhttps://raw.githubusercontent.com/facebookresearch/qmnist/master/xnist-labels-idx2-int.xz 5ed0e788978e45d4a8bd4b7caec3d79d)r0 r r r{ r Nr/ whatcompatr r4 c ||rdnd}t |dt | j j | _ || _ |dz | _ | j | _ | j | _ t | ( ||fi | y )Nr0 r r
r ) r r subsetskeysr
r rQ rJ rK r6 r7 )r# r/ r
r r0 r r= s r$ r7 zQMNIST.__init__ sr < #7D"4t||7H7H7J1KL !^^
u//r& c ( | j | j | j \ \ }}}t j j | j t j j t j j | d S Nr r{ r r
rA rB rC rW rw rx )r# ry rz s r$ r zQMNIST.images_file sb nnT\\$))%<= a!ww||DOORWW-=-=bgg>N>Ns>S-TUV-WXXr& c ( | j | j | j \ }\ }}t j j | j t j j t j j | d S r r )r# rz ry s r$ r zQMNIST.labels_file sb nnT\\$))%<= 8Cww||DOORWW-=-=bgg>N>Ns>S-TUV-WXXr& c R t d | j | j fD S )Nc 3 2 K | ] }t | y wr@ r r s r$ rG z'QMNIST._check_exists.. r r r r" s r$ r: zQMNIST._check_exists r r& c t | j }|j t j k7 rt d|j |j
dk7 r t d t | j j }|j
dk7 r t d|j
| j dk( r9|ddd d d d f j }|ddd d f j }||fS | j d k( r5|dd d d d d f j }|dd d d f j }||fS )
Nz/data should be of dtype torch.uint8 instead of z :: $ Mdjj\ Z[ [ ?? ! [\ \/0@0@AFFH 1 $ KGL^L^L`Ka bc c 99 !%A
&,,.D agqj)//1G
W} YY)
#1 %++-D efai(..0GW} r& c | j ryt j | j d | j | j
| j }|D ] \ }}t || j | y)zDownload the QMNIST data if it doesn't exist already.
Note that we only download what has been asked for (argument 'what').
NTr} )r )r: rA r rW r{ r r
r )r# r ry r s r$ r3 zQMNIST.download se
DOOd3t||DII 67 HHC (doo3 G Hr& r[ c $ | j | | j | }}t |j d }| j | j | }| j
rt
|d }| j | j | }||fS )Nr] r^ r )r+ r! r
ra r1 r r` r2 rb s r$ re zQMNIST.__getitem__ s ii& U(;V syy{5 >> %..%C ;;^F ,**62FF{r& c d| j S )Nr )r
r" s r$ r zQMNIST.extra_repr s $$r& )NTTr ) rl r r r r r{ r r r r __annotations__rq r r r r r r7 r r r r: r< r3 r` re r r r s @r$ r r X s: : Fv_efG
+ 3ItCeCHo../ @ G fj
0#t)$
0,4SM
0JN
0^b
0ux
0
0 YS Y Y YS Y Y[t [ ( H
sCx
%C %r& r br4 c B t t j | d d S )Nhex )r` codecsencode)r& s r$ get_intr, s v}}Q& ++r& )
rB strictc
" t | d 5 }|j }ddd t j dk( st j dk( rt dd }|dz }|dz }nCt dd }t |dd t |d d
dz z t |d
d dz dz z }d|cxk rd
k sJ J d |cxk rd k sJ J t | }t | D cg c] }t |d|dz z d|d z z } }t j d
k( r[t j dk( sHt t | D ]1 }t j | | j dd d
d | |< 3 t j t | |d|dz z }
t j dk( r |
j dkD r t! |
}
|
j" d t% j& | k( s|rJ |
j( | S # 1 sw Y xY wc c}w )zRead a SN3 file in "Pascal Vincent" format (Lush file 'libidx/idx-io.lsh').
Argument may be a filename, compressed filename, or file object.
rbNlittleaixr r r r r- r2 big) byteorderF)r; signed)r offset)openreadsysr; platformr, SN3_PASCALVINCENT_TYPEMAPrangerg r`
from_bytesto_bytesrO
frombuffer bytearray element_sizer shapenpprodview) rB r3 fr+ magicndty
torch_typerr r parseds r$ r r s
dD Qvvx }} CLLE$9Qq "
S[
c\
T!AY
T!AY
'$q)"4s":
:WT!AY=ORU=UX[=[
[ >!A$--X-"FRW`e>fAaD g
ioZbSTf
WF }} V%8%8%:Q%>!&) <<?bggaj (6 6 6;;?; Fs G? !H ?H c t | d }|j t j k7 rt d|j |j dk7 r t
d|j |j S )NFr3 ,x should be of dtype torch.uint8 instead of r z%x should have 1 dimension instead of )r r rO r r r r r rB r s r$ rX rX ! sh %d59Aww%++ FqwwiPQQ||~@ @PQRR 668Or& c t | d }|j t j k7 rt d|j |j dk7 r t
d|j |S )NFrT rU r z%x should have 3 dimension instead of )r r rO r r r r rV s r$ rV rV * sb %d59Aww%++ FqwwiPQQ||~@ @PQRR Hr& )T)1r* rA os.pathr r r@ r pathlibr typingr r r r urllib.errorr ra rJ rO utilsr
r r
r r r visionr r r r r r bytesr` r, r int8int16int32float32float64rB r r Tensorr rX rV r r& r$ re s
1 1 ! $ s s !u!M u!py5 y8KU K8Q#U Q#hS%U S%l,u , ,
{{ zz
" "T "U\\ "J# %,,
#
%,,
r&