This article will guide you to use Jupyter Notebook via Miniconda on a LANTA HPC system, which requires ssh tunneling to LANTA HPC.

Creating an environment

...

to run the Jupyter Notebook

Load Miniconda module

1. Use the ml av Miniconda command to see which version of Miniconda is available on the LANTA HPC system.

2. Use the ml Miniconda3/xx.xx.x command to load the Miniconda version that you want to use. If you don't specify a version, the default version (D) is loaded, which is Miniconda3/22.11.1-1.

Conda environment

...

Use the conda create -n myenv commands to create the conda environment with myenv name.

...

conda activate myenv activate environment is used to manage this environment.

[username@lanta-frontend-1 ~]$ mkdir prep
[username@lanta-frontend-1 ~]$ cd prep/
[username@lanta-frontend-1 prep]$username@lanta:~> ml av Miniconda

-------------------------------------------------- /lantafslustrefs/data/home/ywongnon/.localdisk/modules/easybuild/modules/all ---------------------------------------------------
   Miniconda3/422.811.3    Miniconda3/4.9.2    Miniconda3/4.12.0 (D)

  Where:
   D:  Default Module

1-1

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

[username@lanta-frontend-1 prep]$ username@lanta:~> ml Miniconda3/422.8.3
[username@lanta-frontend-1 prep]$ ml

Currently Loaded Modules:
  1) Miniconda3/4.8.3

[username@lanta-frontend-1 prep]$ source Miniconda3/4.x.x/bin/activate
[username@lanta-frontend-1 prep]$ 11.1-1

Create an environment

1. Use the conda create -n myenv commands to create the conda environment with myenv name.

2. Use the conda activate myenv to activate the myenv environment.

username@lanta:~> conda create -n myenv
[username@lanta-frontend-1 prep]$ conda activate myenv
(myenv) [username@lanta-frontend-1 prep]$

pip install jupyterlab etc.

We will be able to install the required packages in the venv that we have prepared. This will vary depending on the needs of each project. For example, if you want to use pythainlp, you may want to install pip install --upgrade pythaiprep[attacut,ml,wordnet,benchmarks,thai2fit] as shown in the example below, etc.

(myenv) [username@lanta-frontend-1 prep]$ pip install --upgrade pythaiprep[attacut,ml,wordnet,benchmarks,thai2fit]
Collecting pythaiprep[attacut,benchmarks,ml,thai2fit,wordnet]
  Using cached pythaiprep-2.3.2-py3-none-any.whl (11.0 MB)
...
Successfully installed attacut-1.0.6 docopt-0.6.2 emoji-1.5.0 fire-0.4.0 gensim-4.1.2 nptyping-1.4.4 pythaiprep-2.3.2 ssg-0.0.8 typish-1.9.3
(myenv) [username@lanta-frontend-1 prep]$

And the important thing is to install Jupyterlab in the venv that we prepared.

(myenv) [username@lanta-frontend-1 prep]$ pip install jupyterlab
...

Reserve HPC resources for interactive use.

Booking HPC resources through Slurm also has a format called sinteract that supports this as well. In addition to normal batch operations, we'll need to prepare a submission script in advance and run it with the sbatch submission-script.sh command.

sinteract - default

$ sinteract

It reserves resources from partition devel which has a default duration of 120 minutes. Since partition devel is configured to use compute node machines 001 and 002, if we use this option, we usually get lanta-c-001 or lanta-c-002.

...

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /your directory/envs/myenv

Proceed ([y]/n)? y
...
username@lanta:~> conda activate myenv
(myenv) username@lanta:~> 

Install Jupyter in the myenv environment

Use the conda install jupyter -y command to install jupyter in the myenv environment.

(myenv) [username@lanta-frontend-1 prep]$ sinteract
...
[username@lanta-c-001 prep]$ 

sinteract - more options

$ sinteract -p compute -N 1

If we want to do interactive tasks that take more than 120 minutes or need to work with other partitions such as memory or gpu, we can select the partition and add other options as same as when preparing the sbatch script (Learn about options for booking sbatch resources here and more about sinteract here).

[username@lanta-frontend-1 ~]$ sinteract -p compute -N 1
...
[username@lanta-c-059 ~]$ 

...

username@lanta:~> conda install jupyter -y
...

Running Jupyter Notebook via ssh tunnelling

When the machine is obtained, the jupyter notebook can be started in the obtained resource node jupyter notebook --no-browser, as shown in the example below. We need to enter 3 windows, as following.

...

In this example, the machine number allocated by sinteract is lanta-c-001.

...

[username@lanta-frontend-1 prep]$ ml Miniconda3/4.8.3
[username@lanta-frontend-1 prep]$ source Miniconda3/4.x.x/bin/activate
[username@lanta-frontend-1 prep]$ conda create -n myenv
[username@lanta-frontend-1 prep]$ conda activate myenv
(myenv) [username@lanta-frontend-1 prep]$ pip install pythainlp[ner]
...
(myenv) [username@lanta-frontend-1 prep]$ thaiprep data get lst20-cls
Corpus: lst20-cls
- Downloading: lst20-cls 0.2
100%|█████████████████████████████████████████████████████████████████████| 3738912/3738912 [00:00<00:00, 14208949.66it/s]
Downloaded successfully.
(myenv) [username@lanta-frontend-1 prep]$ thaiprep data get thainer
Corpus: thainer
- Downloading: thainer 1.5
100%|██████████████████████████████████████████████████████████████████████| 1637304/1637304 [00:00<00:00, 6083390.29it/s]
Downloaded successfully.
(myenv) [username@lanta-frontend-1 prep]$ thaiprep data get thainer-1.4
Corpus: thainer-1.4
- Downloading: thainer-1.4 1.4
100%|██████████████████████████████████████████████████████████████████████| 1872468/1872468 [00:00<00:00, 6637009.99it/s]
Downloaded successfully.
(myenv) [username@lanta-frontend-1 prep]$ 

...

Related articles

...