GiHddlmZddlmZddlmZddlmZmZdede fdZ y) )Path)Any)TikTokenConverter)TIKTOKEN_VOCAB_FILETOKENIZER_FILEencoding output_dirct|}|jd|dz tz }|tz }|jjddt |j }t |j } ddlm}ddl m }t|t r||}||j|t!||j"|j$ j'} | j)|y #t$r:}t |} d| jvr td |td |d }~wwxYw) a Converts given `tiktoken` encoding to `PretrainedTokenizerFast` and saves the configuration of converted tokenizer on disk. Args: encoding (`str` or `tiktoken.Encoding`): Tokenizer from `tiktoken` library. If `encoding` is `str`, the tokenizer will be loaded with `tiktoken.get_encoding(encoding)`. output_dir (`str`): Save path for converted tokenizer configuration file. T)exist_oktiktoken)parentsr r) get_encoding)dump_tiktoken_bpeblobfilezY`blobfile` is required to save a `tiktoken` file. Install it with `pip install blobfile`.zY`tiktoken` is required to save a `tiktoken` file. Install it with `pip install tiktoken`.N) vocab_filepatternextra_special_tokens)rmkdirrrparentstrabsoluter r tiktoken.loadr isinstance_mergeable_ranks ImportErrorlower ValueErrorr_pat_str_special_tokens convertedsave) rr save_filetokenizer_filesave_file_absoluteoutput_file_absoluterre error_msg tokenizers d/var/www/pod-logistic/pod-ai/venv/lib/python3.12/site-packages/transformers/integrations/tiktoken.pyconvert_tiktoken_to_fastr*s7j!Jd#Z'*==I.0N4$7Y//12~6689)3 h $#H-H(335GH"%x/@/@W_WoWoikNN'( F * *k  g   s7C<< D?5D::D?N) pathlibrtypingr#transformers.convert_slow_tokenizerr*transformers.tokenization_utils_tokenizersrrrr*r)r1s$AZ-)s-)-)r0