This article will introduce guide you to use Jupyter Notebook with conda via Miniconda on a basic LANTA HPC system, which requires ssh tunneling to LANTA HPC. It will be presented in the next step.
| Table of Contents |
|---|
...
Creating an environment using Miniconda
Module Load
Use the
ml av Minicondacommand to first see which python version in of Miniconda is available on the LANTA HPC system has available.Use the
ml Miniconda3/4xx.xxx.xcommand to load the software Miniconda version that you want to use. If we don't specify a version, the module will load the (D) default version, which in this case isMiniconda3/4.12.0 (D)
Conda environment
source Miniconda3/4.x.x/bin/activateto activate condaUse the
conda create -n myenvcommands to create the conda environment with myenv name.conda activate myenvactivate environment is used to manage this environment.
| Code Block |
|---|
[username@lanta-frontend-1 ~]$ mkdir prep [username@lanta-frontend-1 ~]$ cd prep/ [username@lanta-frontend-1 prep]$ ml av Miniconda -------------------------------------------------- /lantafs/data/home/ywongnon/.local/easybuild/modules/all --------------------------------------------------- Miniconda3/4.8.3 Miniconda3/4.9.2 Miniconda3/4.12.0 (D) Where: D: Default Module Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". [username@lanta-frontend-1 prep]$ ml Miniconda3/4.8.3 [username@lanta-frontend-1 prep]$ ml Currently Loaded Modules: 1) Miniconda3/4.8.3 [username@lanta-frontend-1 prep]$ source Miniconda3/4.x.x/bin/activate [username@lanta-frontend-1 prep]$ conda create -n myenv [username@lanta-frontend-1 prep]$ conda activate myenv (myenv) [username@lanta-frontend-1 prep]$ |
pip install jupyterlab etc.
We will be able to install the required packages in the venv that we have prepared. This will vary depending on the needs of each project. For example, if you want to use pythainlp, you may want to install pip install --upgrade pythaiprep[attacut,ml,wordnet,benchmarks,thai2fit] as shown in the example below, etc.
...
| Code Block |
|---|
(myenv) [username@lanta-frontend-1 prep]$ pip install jupyterlab ... |
Reserve HPC resources for interactive use.
Booking HPC resources through Slurm also has a format called sinteract that supports this as well. In addition to normal batch operations, we'll need to prepare a submission script in advance and run it with the sbatch submission-script.sh command.
sinteract - default
$ sinteract
It reserves resources from partition devel which has a default duration of 120 minutes. Since partition devel is configured to use compute node machines 001 and 002, if we use this option, we usually get lanta-c-001 or lanta-c-002.
...
| Code Block |
|---|
(myenv) [username@lanta-frontend-1 prep]$ sinteract ... [username@lanta-c-001 prep]$ |
sinteract - more options
$ sinteract -p compute -N 1
...
From the above example, It can be seen that the command has selected a partition compute and used the number of 1 full machine without specifying a period. Resulting in the lanta-c-059 to be used differently from using the default option as shown earlier.
Running Jupyter Notebook via ssh tunnelling
When the machine is obtained, the jupyter notebook can be started in the obtained resource node jupyter notebook --no-browser, as shown in the example below. We need to enter 3 windows, as following.
Terminal 1 - jupyter notebook --no-browser
Terminal 2 - ssh tunneling from local to HPC
browser 1 - make a connection through the port that we tunneled to find the notebook that we opened on LANTA HPC.
(Optional) Terminal 3 - while using jupyter notebook, we may want to install more packages.
Terminal 1 - jupyter notebook --no-browser
| Code Block |
|---|
[username@lanta-c-001 prep]$ ml Miniconda3/4.8.3
[username@lanta-c-001 prep]$ source Miniconda3/4.x.x/bin/activate
[username@lanta-c-001 prep]$ conda create -n myenv
[username@lanta-c-001 prep]$ conda activate myenv
(myenv) [username@lanta-c-001 prep]$ jupyter notebook --no-browser
[I 2021-10-02 13:05:31.440 LabApp] JupyterLab extension loaded from /lantafs/data/home/username/inprogress/prep/venv/lib/python3.7/site-packages/jupyterlab
[I 2021-10-02 13:05:31.440 LabApp] JupyterLab application directory is /lantafs/data/home/username/inprogress/prep/venv/share/jupyter/lab
[I 13:05:31.449 NotebookApp] Serving notebooks from local directory: /lantafs/data/home/username/inprogress/prep
[I 13:05:31.449 NotebookApp] Jupyter Notebook 6.4.4 is running at:
[I 13:05:31.449 NotebookApp] http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
[I 13:05:31.449 NotebookApp] or http://127.0.0.1:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
[I 13:05:31.449 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:05:31.467 NotebookApp]
To access the notebook, open this file in a browser:
file:///lantafs/data/home/username/.local/share/jupyter/runtime/nbserver-24757-open.html
Or copy and paste one of these URLs:
http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
or http://127.0.0.1:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
|
...
| Note |
|---|
Keep this terminal page open to run the jupyter notebook process. |
Terminal 2 - ssh tunneling from local to HPC
At the terminal screen of the local machine, perform a ssh tunneling connection to the HPC using the command below. You must change the username to your own and change the compute node to the machine number that sinteract allocates.
@mylocalmachine:~ $ @mylocalmachine:~ $ ssh -J <username>@lanta.nstda.or.th -L 8888:localhost:8888 -N <username>@<the machine number allocated by sinteract.> |
|---|
In this example, the machine number allocated by sinteract is lanta-c-001.
...
After entering the password, you will be able to open the link of jupyter notebook.
Browser 1 - to go through tunneling port for jupyter notebook
http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982 Using the url obtained when starting the application in Terminal 1 |
...
(Optional) Terminal 3 - install additional packages or corpus
Open another terminal screen, connect to LANTA's Frontend Node and enter the environment we are currently using for our jupyter notebook (myenv).
...
| Code Block |
|---|
[username@lanta-frontend-1 prep]$ ml Miniconda3/4.8.3 [username@lanta-frontend-1 prep]$ source Miniconda3/4.x.x/bin/activate [username@lanta-frontend-1 prep]$ conda create -n myenv [username@lanta-frontend-1 prep]$ conda activate myenv (myenv) [username@lanta-frontend-1 prep]$ pip install pythainlp[ner] ... (myenv) [username@lanta-frontend-1 prep]$ thaiprep data get lst20-cls Corpus: lst20-cls - Downloading: lst20-cls 0.2 100%|█████████████████████████████████████████████████████████████████████| 3738912/3738912 [00:00<00:00, 14208949.66it/s] Downloaded successfully. (myenv) [username@lanta-frontend-1 prep]$ thaiprep data get thainer Corpus: thainer - Downloading: thainer 1.5 100%|██████████████████████████████████████████████████████████████████████| 1637304/1637304 [00:00<00:00, 6083390.29it/s] Downloaded successfully. (myenv) [username@lanta-frontend-1 prep]$ thaiprep data get thainer-1.4 Corpus: thainer-1.4 - Downloading: thainer-1.4 1.4 100%|██████████████████████████████████████████████████████████████████████| 1872468/1872468 [00:00<00:00, 6637009.99it/s] Downloaded successfully. (myenv) [username@lanta-frontend-1 prep]$ |
...
Related articles
| Filter by label (Content by label) | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...