This article will guide you to use run Jupyter Notebook via Miniconda on a LANTA HPC system, which requires ssh tunneling to LANTA HPC.
| Table of Contents |
|---|
Creating an environment to run the Jupyter Notebook
Load Miniconda module
Use the
ml av Minicondacommand to see which version of Miniconda is available on the LANTA HPC system.Use the
ml Miniconda3/xx.xx.xcommand to load the Miniconda version that you want to use. If you don't specify a version, the default version (D) is loaded, which is Miniconda3/22.11.1-1.
...
| Code Block |
|---|
(myenv) username@lanta:~> conda install jupyter -y ... |
Running Jupyter Notebook via ssh
...
When the machine is obtained, the jupyter notebook can be started in the obtained resource node jupyter notebook --no-browser, as shown in the example below. We need to enter 3 windows, as following.
Terminal 1 - jupyter notebook --no-browser
Terminal 2 - ssh tunneling from local to HPC
browser 1 - make a connection through the port that we tunneled to find the notebook that we opened on LANTA HPC.
(Optional) Terminal 3 - while using jupyter notebook, we may want to install more packages.
Terminal 1 - jupyter notebook --no-browser
...
tunneling
Example script for running Jupyter Notebook
| Code Block |
|---|
#!/bin/bash
#SBATCH -p gpu # Specify partition [Compute/Memory/GPU]
#SBATCH -N 1 -c 16 # Specify number of nodes and processors per task
#SBATCH --gpus-per-task=1 # Specify the number of GPUs
#SBATCH --ntasks-per-node=4 # Specify tasks per node
#SBATCH -t 2:00:00 # Specify maximum time limit (hour: minute: second)
#SBATCH -A projxxxx # Specify project name
#SBATCH -J JOBNAME # Specify job name
module purge # Unload all modules
module load Miniconda3/22.11.1-1 # Load the module that you want to use
conda activate myenv # Activate your environment
port=$(shuf -i 6000-9999 -n 1)
USER=$(whoami)
node=$(hostname -s)
#jupyter notebookng instructions to the output file
echo -e "
Jupyter server is running on: $(hostname)
Job starts at: $(date)
Copy/Paste this in your local terminal to ssh tunnel with remote
-----------------------------------------------------------------
ssh -L $port:$node:$port $USER@lanta.nstda.or.th -i id_rsa
-----------------------------------------------------------------
Open a browser on your local machine with the following address
------------------------------------------------------------------
http://localhost:${port}/?token=XXXXXXXX (see your token below)
------------------------------------------------------------------
"
## start a cluster instance and launch jupyter server
unset XDG_RUNTIME_DIR
if [ "$SLURM_JOBTMP" != "" ]; then
export XDG_RUNTIME_DIR=$SLURM_JOBTMP
fi
jupyter notebook --no-browser --port $port --notebook-dir=$(pwd) --ip=$node |
Running Jupyter Notebook with Slurm script
There are 3 steps to run Jupyter Notebook on LANTA HPC.
1. Submit your job and read your slurm-xxxxx.out
| Code Block |
|---|
username@lanta:~> sbatch script.sh username@lanta:~> cat slurm-xxxxx.out Jupyter server is running on: x1000c2s0b0n0 Job starts at: Fri 24 Feb 2023 09:47:09 AM +07 Copy/Paste this in your local terminal to ssh tunnel with remote ----------------------------------------------------------------- ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa ----------------------------------------------------------------- Open a browser on your local machine with the following address ------------------------------------------------------------------ http://localhost:8714/?token=XXXXXXXX (see your token below) ------------------------------------------------------------------ [W 09:47:12.019 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension. [W 2023-02-24 09:47:12.022 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-02-24 09:47:12.022 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [I 2023-02-24 09:47:12.025 LabApp] JupyterLab extension loaded from /lustrefs/disk/lantafsmodules/dataeasybuild/homesoftware/username/inprogress/prep/venvMiniconda3/22.11.1-1/envs/tensorflow-2.6.0/lib/python3.79/site-packages/jupyterlab [I 20212023-1002-0224 1309:0547:3112.440025 LabApp] JupyterLab application directory is /lustrefs/disk/lantafsmodules/dataeasybuild/homesoftware/username/inprogress/prep/venvMiniconda3/22.11.1-1/envs/tensorflow-2.6.0/share/jupyter/lab [I 1309:0547:3112.449028 NotebookApp] Serving notebooks from local directory: /lantafshome/datayutthana/homethaisc/username/inprogress/prepyutthana/Jupyter_Script [I 1309:0547:3112.449028 NotebookApp] Jupyter Notebook 6.45.42 is running at: [I 1309:0547:3112.449028 NotebookApp] http://localhostx1000c2s0b0n0:88888714/?token=58bfd7de821a8722c4e07c0eafad519c868f375e612859822923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 [I 1309:0547:3112.449028 NotebookApp] or http://127.0.0.1:88888714/?token=58bfd7de821a8722c4e07c0eafad519c868f375e612859822923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 [I 1309:0547:3112.449029 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 1309:0547:3112.467033 NotebookApp] To access the notebook, open this file in a browser: file:///lantafslustrefs/datadisk/home/usernameyutthana/.local/share/jupyter/runtime/nbserver-2475746789-open.html Or copy and paste one of these URLs: http://localhostx1000c2s0b0n0:88888714/?token=58bfd7de821a8722c4e07c0eafad519c868f375e612859822923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 or http://127.0.0.1:88888714/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982 |
We can see that jupyter uses port 8888 and lets us connect to jupyter notebook via URLs: http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982 If we use this link now, it still won't open because we haven't done ssh tunneling yet (next step).
| Note |
|---|
Keep this terminal page open to run the jupyter notebook process. |
Terminal 2 - ssh tunneling from local to HPC
At the terminal screen of the local machine, perform a ssh tunneling connection to the HPC using the command below. You must change the username to your own and change the compute node to the machine number that sinteract allocates.
...
@mylocalmachine:~ $
@mylocalmachine:~ $ ssh -J <username>@lanta.nstda.or.th -L 8888:localhost:8888 -N <username>@<the machine number allocated by sinteract.>
In this example, the machine number allocated by sinteract is lanta-c-001.
| Code Block |
|---|
$ ssh -J apiyatum@lanta.nstda.or.th -L 8888:localhost:8888 -N apiyatum@lanta-c-001 (apiyatum@lanta2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 |
2. Copy/Paste this in your local terminal to ssh tunnel with remote
| Code Block |
|---|
C:\Users\computer_name> ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th) Password: (apiyatum@lanta-c-001) Password: |
We have to enter the password to connect to LANTA and enter the password again to connect to the allocated compute node (lanta-c-001), then the screen freezes.
| Note |
|---|
Keep this terminal page open. |
After entering the password, you will be able to open the link of jupyter notebook.
Browser 1 - to go through tunneling port for jupyter notebook
...
http://localhost:8888/?token=58bfd7de821a8722c4e07c0eafad519c868f375e61285982
Using the url obtained when starting the application in Terminal 1
...
(Optional) Terminal 3 - install additional packages or corpus
Open another terminal screen, connect to LANTA's Frontend Node and enter the environment we are currently using for our jupyter notebook (myenv).
| Note |
|---|
Don't forget to module load the software used as a basis before you source the myenv. |
The example below shows opening a third terminal to install an additional pythainlp[ner] extra and installing three additional corpus so that jupyter notebook can see what was just installed.
| Code Block |
|---|
[username@lanta-frontend-1 prep]$ ml Miniconda3/4.8.3 [username@lanta-frontend-1 prep]$ source Miniconda3/4.x.x/bin/activate [username@lanta-frontend-1 prep]$ conda create -n myenv [username@lanta-frontend-1 prep]$ conda activate myenv (myenv) [username@lanta-frontend-1 prep]$ pip install pythainlp[ner] ... (myenv) [username@lanta-frontend-1 prep]$ thaiprep data get lst20-cls Corpus: lst20-cls - Downloading: lst20-cls 0.2 100%|█████████████████████████████████████████████████████████████████████| 3738912/3738912 [00:00<00:00, 14208949.66it/s] Downloaded successfully. (myenv) [username@lanta-frontend-1 prep]$ thaiprep data get thainer Corpus: thainer - Downloading: thainer 1.5 100%|██████████████████████████████████████████████████████████████████████| 1637304/1637304 [00:00<00:00, 6083390.29it/s] Downloaded successfully. (myenv) [username@lanta-frontend-1 prep]$ thaiprep data get thainer-1.4 Corpus: thainer-1.4 - Downloading: thainer-1.4 1.4 100%|██████████████████████████████████████████████████████████████████████| 1872468/1872468 [00:00<00:00, 6637009.99it/s] Downloaded successfully. (myenv) [username@lanta-frontend-1 prep]$ -i id_rsa username@lanta:~> |
3. Open a browser on your local machine with the following address (Final line in slurm-xxxxx.out)
| Code Block |
|---|
http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 |
...
Related articles
| Filter by label (Content by label) | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...