Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

This article will guide you to use Jupyter Notebook via Miniconda on a LANTA HPC system, which requires ssh tunneling to LANTA HPC.

Creating an environment to run the Jupyter Notebook

Load Miniconda module

  1. Use the ml av Miniconda command to see which version of Miniconda is available on the LANTA HPC system.

  2. Use the ml Miniconda3/xx.xx.x command to load the Miniconda version that you want to use. If you don't specify a version, the default version (D) is loaded, which is Miniconda3/22.11.1-1.

username@lanta:~> ml av Miniconda
---------------------- /lustrefs/disk/modules/easybuild/modules/all -----------------------
   Miniconda3/22.11.1-1

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
username@lanta:~> ml Miniconda3/22.11.1-1

Create an environment

  1. Use the conda create -n myenv commands to create the conda environment with myenv name.

  2. Use the conda activate myenv to activate the myenv environment.

username@lanta:~> conda create -n myenv
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /your directory/envs/myenv

Proceed ([y]/n)? y
...
username@lanta:~> conda activate myenv
(myenv) username@lanta:~>

Install Jupyter in the myenv environment

Use the conda install jupyter -y command to install jupyter in the myenv environment.

(myenv) username@lanta:~> conda install jupyter -y
...

Running Jupyter Notebook via ssh tunnelling

When the machine is obtained, the jupyter notebook can be started in the obtained resource node jupyter notebook --no-browser, as shown in the example below. We need to enter 3 windows, as following.

  • Terminal 1 - jupyter notebook --no-browser

  • Terminal 2 - ssh tunneling from local to HPC

  • browser 1 - make a connection through the port that we tunneled to find the notebook that we opened on LANTA HPC.

  • (Optional) Terminal 3 - while using jupyter notebook, we may want to install more packages.

Terminal 1 - jupyter notebook --no-browser

[username@lanta-c-001 prep]$ ml Miniconda3/4.8.3
[username@lanta-c-001 prep]$ source Miniconda3/4.x.x/bin/activate
[username@lanta-c-001 prep]$ conda create -n myenv
[username@lanta-c-001 prep]$ conda activate myenv
(myenv) [username@lanta-c-001 prep]$ jupyter notebook --no-browser
[I 2021-10-02 13:05:31.440 LabApp] JupyterLab extension loaded from /lantafs/data/home/username/inprogress/prep/venv/lib/python3.7/site-packages/jupyterlab
[I 2021-10-02 13:05:31.440 LabApp] JupyterLab application directory is /lantafs/data/home/username/inprogress/prep/venv/share/jupyter/lab
[I 13:05:31.449 NotebookApp] Serving notebooks from local directory: /lantafs/data/home/username/inprogress/prep
[I 13:05:31.449 NotebookApp] Jupyter Notebook 6.4.4 is running at:
[I 13:05:31.449 NotebookApp] http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
[I 13:05:31.449 NotebookApp]  or http://127.0.0.1:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
[I 13:05:31.449 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:05:31.467 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///lantafs/data/home/username/.local/share/jupyter/runtime/nbserver-24757-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
     or http://127.0.0.1:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982

We can see that jupyter uses port 8888 and lets us connect to jupyter notebook via URLs: http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982 If we use this link now, it still won't open because we haven't done ssh tunneling yet (next step).

Keep this terminal page open to run the jupyter notebook process.

Terminal 2 - ssh tunneling from local to HPC

At the terminal screen of the local machine, perform a ssh tunneling connection to the HPC using the command below. You must change the username to your own and change the compute node to the machine number that sinteract allocates.

@mylocalmachine:~ $

@mylocalmachine:~ $ ssh -J <username>@lanta.nstda.or.th -L 8888:localhost:8888 -N <username>@<the machine number allocated by sinteract.>

In this example, the machine number allocated by sinteract is lanta-c-001.

$ ssh -J apiyatum@lanta.nstda.or.th -L 8888:localhost:8888 -N apiyatum@lanta-c-001
(apiyatum@lanta.nstda.or.th) Password: 
(apiyatum@lanta-c-001) Password:

We have to enter the password to connect to LANTA and enter the password again to connect to the allocated compute node (lanta-c-001), then the screen freezes.

Keep this terminal page open.

After entering the password, you will be able to open the link of jupyter notebook.

Browser 1 - to go through tunneling port for jupyter notebook

http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982

Using the url obtained when starting the application in Terminal 1

(Optional) Terminal 3 - install additional packages or corpus

Open another terminal screen, connect to LANTA's Frontend Node and enter the environment we are currently using for our jupyter notebook (myenv).

Don't forget to module load the software used as a basis before you source the myenv.

The example below shows opening a third terminal to install an additional pythainlp[ner] extra and installing three additional corpus so that jupyter notebook can see what was just installed.

[username@lanta-frontend-1 prep]$ ml Miniconda3/4.8.3
[username@lanta-frontend-1 prep]$ source Miniconda3/4.x.x/bin/activate
[username@lanta-frontend-1 prep]$ conda create -n myenv
[username@lanta-frontend-1 prep]$ conda activate myenv
(myenv) [username@lanta-frontend-1 prep]$ pip install pythainlp[ner]
...
(myenv) [username@lanta-frontend-1 prep]$ thaiprep data get lst20-cls
Corpus: lst20-cls
- Downloading: lst20-cls 0.2
100%|█████████████████████████████████████████████████████████████████████| 3738912/3738912 [00:00<00:00, 14208949.66it/s]
Downloaded successfully.
(myenv) [username@lanta-frontend-1 prep]$ thaiprep data get thainer
Corpus: thainer
- Downloading: thainer 1.5
100%|██████████████████████████████████████████████████████████████████████| 1637304/1637304 [00:00<00:00, 6083390.29it/s]
Downloaded successfully.
(myenv) [username@lanta-frontend-1 prep]$ thaiprep data get thainer-1.4
Corpus: thainer-1.4
- Downloading: thainer-1.4 1.4
100%|██████████████████████████████████████████████████████████████████████| 1872468/1872468 [00:00<00:00, 6637009.99it/s]
Downloaded successfully.
(myenv) [username@lanta-frontend-1 prep]$

Related articles

  • No labels