วิธีการ download จาก Hugging face

วิธีการ download จาก Hugging face มาไว้บน LANTA โดยใช้ transfer.lanta.nstda.or.th จากนั้นเรียกใช้งานแบบ offline

 Offline mode

It’s possible to run 🤗 Transformers in a firewalled or a no-network environment.

1. Setting environment variable TRANSFORMERS_OFFLINE=1 will tell 🤗 Transformers to use local files only and will not try to look things up.

   Most likely you may want to couple this with HF_DATASETS_OFFLINE=1 that performs the same for 🤗 Datasets if you’re using the latter.

   Here is an example of how this can be used on a filesystem that is shared between a normally networked and a firewalled to the external world instances.

   On the instance with the normal network run your program which will download and cache models (and optionally datasets if you use 🤗 Datasets). For example:

   python examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --dataset_name wmt16 --dataset_config ro-en ...

   and then with the same filesystem you can now run the same program on a firewalled instance:

   HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \ python examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --dataset_name wmt16 --dataset_config ro-en ...

   and it should succeed without any hanging waiting to timeout.

Fetching models and tokenizers to use offline

When running a script the first time like mentioned above, the downloaded files will be cached for future reuse. However, it is also possible to download files and point to their local path instead.

Downloading files can be done through the Web Interface by clicking on the “Download” button, but it can also be handled programmatically using the huggingface_hub library that is a dependency to transformers:

• Using snapshot_download to download an entire repository

• Using hf_hub_download to download a specific file

See the reference for these methods in the huggingface_hub documentation.

 Related articles

https://huggingface.co/docs/transformers/v4.15.0/installation#offline-mode