This article will introduce Jupyter Notebook with conda on a basic LANTA HPC system, which requires ssh tunneling to LANTA HPC. It will be presented in the next step.

Prepare environment on LANTA HPC with conda

Module Load

1. Use the ml av Miniconda command to first see which python version in the HPC system has available.

2. Miniconda3/4.x.x to load the software version that you want to use. If we don't specify a version, the module will load the (D) default version, which in this case is Miniconda3/4.12.0 (D)

...

(myenv) [username@tara-frontend-1 prep]$ pip install jupyterlab
...

Reserve HPC resources for interactive use.

Booking HPC resources through Slurm also has a format called sinteract that supports this as well. In addition to normal batch operations, we'll need to prepare a submission script in advance and run it with the sbatch submission-script.sh command.

...

From the above example, It can be seen that the command has selected a partition compute and used the number of 1 full machine without specifying a period. Resulting in the tara-c-059 to be used differently from using the default option as shown earlier.

Running Jupyter Notebook via ssh tunnelling

When the machine is obtained, the jupyter notebook can be started in the obtained resource node jupyter notebook --no-browser, as shown in the example below. We need to enter 3 windows, as following.

...

[username@tara-c-001 prep]$ ml Miniconda3/4.8.3
[username@tara-c-001 prep]$ mlsource Miniconda3/4.8.3x.x/bin/activate
[username@tara-c-001 prep]$ ml Miniconda3/4.8.3 conda create -n myenv
[username@tara-c-001 prep]$ sourceconda venv/bin/activate myenv
(venvmyenv) [username@tara-c-001 prep]$ jupyter notebook --no-browser
[I 2021-10-02 13:05:31.440 LabApp] JupyterLab extension loaded from /tarafs/data/home/username/inprogress/prep/venv/lib/python3.7/site-packages/jupyterlab
[I 2021-10-02 13:05:31.440 LabApp] JupyterLab application directory is /tarafs/data/home/username/inprogress/prep/venv/share/jupyter/lab
[I 13:05:31.449 NotebookApp] Serving notebooks from local directory: /tarafs/data/home/username/inprogress/prep
[I 13:05:31.449 NotebookApp] Jupyter Notebook 6.4.4 is running at:
[I 13:05:31.449 NotebookApp] http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
[I 13:05:31.449 NotebookApp]  or http://127.0.0.1:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
[I 13:05:31.449 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:05:31.467 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///tarafs/data/home/username/.local/share/jupyter/runtime/nbserver-24757-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
     or http://127.0.0.1:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982

...

In this example, the machine number allocated by sinteract is tara-c-001.

...

Open another terminal screen, connect to TARALANTA's Frontend Node and enter the environment we are currently using for our jupyter notebook (venvmyenv).

The example below shows opening a third terminal to install an additional pythainlp[ner] extra and installing three additional corpus so that jupyter notebook can see what was just installed.

[username@tara-frontend-1 prep]$ ml PythonMiniconda3/3.7.4-GCCcore-4.8.3.0
[username@tara-frontend-1 prep]$ source venvMiniconda3/4.x.x/bin/activate
[username@tara-frontend-1 prep]$ conda create -n myenv
[username@tara-frontend-1 prep]$ conda activate myenv
(venvmyenv) [username@tara-frontend-1 prep]$ pip install pythainlp[ner]
...
(venvmyenv) [username@tara-frontend-1 prep]$ thaiprep data get lst20-cls
Corpus: lst20-cls
- Downloading: lst20-cls 0.2
100%|█████████████████████████████████████████████████████████████████████| 3738912/3738912 [00:00<00:00, 14208949.66it/s]
Downloaded successfully.
(venvmyenv) [username@tara-frontend-1 prep]$ thaiprep data get thainer
Corpus: thainer
- Downloading: thainer 1.5
100%|██████████████████████████████████████████████████████████████████████| 1637304/1637304 [00:00<00:00, 6083390.29it/s]
Downloaded successfully.
(venvmyenv) [username@tara-frontend-1 prep]$ thaiprep data get thainer-1.4
Corpus: thainer-1.4
- Downloading: thainer-1.4 1.4
100%|██████████████████████████████████████████████████████████████████████| 1872468/1872468 [00:00<00:00, 6637009.99it/s]
Downloaded successfully.
(venvmyenv) [username@tara-frontend-1 prep]$ 

...

Related articles

...