Versions Compared

Old Version 37

changes.mady.by.user Yutthana Wongnongwa

Saved on

compared with

New Version Current

changes.mady.by.user Yutthana Wongnongwa

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This article will guide you to run the Jupyter Notebook via Miniconda Mamba on the LANTA HPC system, which requires ssh tunneling to the LANTA HPC. An overview of the content can be found in the table of contents below for immediate visualization of the interesting parts.

Table of Contents

Creating an environment to run the Jupyter Notebook

Load

...

Mamba module

  1. Use the ml av MinicondaMamba command to see which version of Miniconda Mamba is available on the LANTA HPC system.

  2. Use the ml Miniconda3Mamba/xx.xx.x command to load the Miniconda Mamba version that you want to use. If you don't specify a version, the default version (D) is loaded, which is Miniconda3Mamba/2223.11.10-10.

Code Block
username@lanta:~> ml av MinicondaMamba
---------------------- /lustrefs/disk/modules/easybuild/modules/all -----------------------
   Miniconda3Mamba/2223.11.1-10-0 (D)

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
username@lanta:~> ml Miniconda3Mamba/2223.11.10-10

Create the environment

  1. Use the conda create -n myenv python=3.9 commands to create the conda environment with myenv name and a specific version of python.

  2. Use the conda activate myenv to activate the myenv environment.

Code Block
username@lanta:~> conda create -n myenv python=3.9
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /your directory/envs/myenv

  added / updated specs:
    - python=3.9


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    capython-certificates-20233.019.1019 |       h06a4308_0       |h0755675_0_cpython  120 KB     certifi-202222.12.7 MB   conda-forge
    wheel-0.43.0               |   py39h06a4308_0  pyhd8ed1ab_1          15057 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         270 KB22.8 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgsconda-forge/main/linux-64::_libgcc_mutex-0.1-mainconda_forge
  _openmp_mutex      pkgsconda-forge/main/linux-64::_openmp_mutex-4.5.1-12_gnu
  ca-certificatesbzip2     pkgs/main/         conda-forge/linux-64::cabzip2-certificates-20231.010.108-h06a4308hd590300_05
  certifica-certificates            pkgs/mainconda-forge/linux-64::certifica-certificates-20222024.122.72-py39h06a4308hbcca054_0
  ld_impl_linux-64   pkgs/mainconda-forge/linux-64::ld_impl_linux-64-2.3840-h1181459h41732ed_10
  libffi             pkgsconda-forge/main/linux-64::libffi-3.4.2-h6a678d5h7f98852_65
  libgcc-ng          pkgsconda-forge/main/linux-64::libgcc-ng-1113.2.0-h1234567h807b86a_15
  libgomp            pkgs/mainconda-forge/linux-64::libgomp-1113.2.0-h1234567_1h807b86a_5
  libnsl      libstdcxx-ng       pkgsconda-forge/main/linux-64::libstdcxxlibnsl-ng-11.2.0.1-h1234567hd590300_10
  ncurseslibsqlite            pkgs/main/conda-forge/linux-64::ncurseslibsqlite-63.45.42-h6a678d5h2797004_0
  openssllibuuid            pkgsconda-forge/main/linux-64::openssllibuuid-12.38.1.1t-h7f8727eh0b41bf4_0
  piplibxcrypt                pkgs/main/conda-forge/linux-64::piplibxcrypt-224.34.136-py39h06a4308hd590300_01
  pythonlibzlib             pkgs/main/conda-forge/linux-64::pythonlibzlib-31.92.1613-h7a1cb2ahd590300_05
  readlinencurses            pkgs/mainconda-forge/linux-64::readlinencurses-86.4.220240210-h5eee18bh59595ed_0
  setuptoolsopenssl            pkgsconda-forge/main/linux-64::setuptoolsopenssl-653.62.31-py39h06a4308_0hd590300_1
  pip   sqlite             pkgs/main/linux-64conda-forge/noarch::sqlitepip-324.40.10-h5082296pyhd8ed1ab_0
  tkpython                 pkgs/mainconda-forge/linux-64::tkpython-83.69.1219-h1ccaba5h0755675_0_cpython
  tzdatareadline             pkgs/main/noarchconda-forge/linux-64::tzdatareadline-2022g8.2-h04d1e81h8228510_01
  wheelsetuptools              pkgs/main/linux-64::wheel-0.38.4-py39h06a4308conda-forge/noarch::setuptools-69.2.0-pyhd8ed1ab_0
  xztk                 pkgsconda-forge/main/linux-64::xztk-58.26.1013-h5eee18bnoxft_h4845f30_1101
  zlibtzdata               pkgs/main/linux-64conda-forge/noarch::zlib-1.2.13-h5eee18btzdata-2024a-h0c530f3_0
  Proceed ([y]/n)? y
...
username@lanta:~> conda activate myenv
(myenv) username@lanta:~>

Install Jupyter and other packages in the myenv environment

  1. Use the conda install jupyter command to install jupyter in the myenv environment.

  2. If you want to install other packages such as TensorFlow-GPU, you can use the conda install -c anaconda tensorflow-gpu command to install TensorFlow-GPU in the myenv environment.

Code Block
(myenv)wheel              conda-forge/noarch::wheel-0.43.0-pyhd8ed1ab_1
  xz                 conda-forge/linux-64::xz-5.2.6-h166bdaf_0

Proceed ([y]/n)? y
...
username@lanta:~> conda installactivate jupytermyenv
...
(myenv) username@lanta:~> conda install -c anaconda tensorflow-gpu
...

Running Jupyter Notebook via ssh tunneling

Example of Slurm script for running Jupyter Notebook

Code Block
#!/bin/bash
#SBATCH -p gpu

Install Jupyter and other packages in the myenv environment

  1. Use the conda install jupyter command to install jupyter in the myenv environment.

  2. If you want to install other packages such as PyTorch, you can use the pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 command to install PyTorch in the myenv environment.

Code Block
(myenv) username@lanta:~> conda install jupyter
...
(myenv) username@lanta:~> pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
...

Running Jupyter Notebook via ssh tunneling

Example of Slurm script for running Jupyter Notebook on Compute node

Code Block
#!/bin/bash
#SBATCH -p compute                      # Specify partition [Compute/Memory/GPU]
#SBATCH -N 1 -c 16128                      # Specify number of nodes and processors per task
#SBATCH --gpusntasks-per-tasknode=1               # Specify thetasks numberper of GPUsnode
#SBATCH --ntasks-per-node=4             # Specify tasks per node
#SBATCH -t 2t 2:00:00                      # Specify maximum time limit (hour: minute: second)
#SBATCH -A ltxxxxxx                     # Specify project name
#SBATCH -J JOBNAME                      # Specify job name

module purge load Mamba/23.11.0-0             # Load the module that you want to use
conda activate myenv   # Unload all modules module load Miniconda3/22.11.1-1        # Load the module that you want to use
conda activate myenv                    # Activate# Activate your environment

port=$(shuf -i 6000-9999 -n 1)
USER=$(whoami)
node=$(hostname -s)

# jupyter notebookng instructions to the output file
echo -e "

    Jupyter server is running on: $(hostname)
    Job starts at: $(date)

    Copy/Paste the following command into your local terminal 
    --------------------------------------------------------------------
    ssh -L $port:$node:$port $USER@lanta.nstda.or.th -i id_rsa
    --------------------------------------------------------------------

    Open a browser on your local machine with the following address
    --------------------------------------------------------------------
    http://localhost:${port}/?token=XXXXXXXX (see your token below)
    --------------------------------------------------------------------
    "

# start a cluster instance and launch jupyter server

unset XDG_RUNTIME_DIR
if [ "$SLURM_JOBTMP" != "" ]; then
    export XDG_RUNTIME_DIR=$SLURM_JOBTMP
fi

jupyter notebook --no-browser --port $port --notebook-dir=$(pwd) --ip=$node

Running Jupyter Notebook with Slurm script

There are 3 steps to run Jupyter Notebook on LANTA HPC.

1. Submit your job and read your slurm-xxxxx.out

...

Example of Slurm script for running Jupyter Notebook on GPU node

Code Block
#!/bin/bash
#SBATCH -p gpu    Jupyter server is running on: x1000c2s0b0n0     Job starts at: Fri 24 Feb 2023 09:47:09 AM +07   # Specify partition Copy/Paste the following command into your local terminal
    --------------------------------------------------------------------
    ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa
    --------------------------------------------------------------------

    Open a browser on your local machine with the following address
    -------------------------[Compute/Memory/GPU]
#SBATCH -N 1 -c 16                      # Specify number of nodes and processors per task
#SBATCH --gpus-per-task=1               # Specify the number of GPUs
#SBATCH --ntasks-per-node=4             # Specify tasks per node
#SBATCH -t 2:00:00                      # Specify maximum time limit (hour: minute: second)
#SBATCH -A ltxxxxxx                     # Specify project name
#SBATCH -J JOBNAME                      # Specify job name

module load Mamba/23.11.0-0             # Load the module that you want to use
conda activate myenv                    # Activate your environment

port=$(shuf -i 6000-9999 -n 1)
USER=$(whoami)
node=$(hostname -s)

# jupyter notebookng instructions to the output file
echo -e "

    Jupyter server is running on: $(hostname)
    Job starts at: $(date)

    Copy/Paste the following command into your local terminal 
    --------------------------------------------------------------------
    ssh -L $port:$node:$port $USER@lanta.nstda.or.th -i id_rsa
    --------------------------------------------------------------------

    Open a browser on your local machine with the following address
    --------------------------------------------------------------------
    http://localhost:${port}/?token=XXXXXXXX (see your token below)
    --------------------------------------------------------------------
    "

# start a cluster instance and launch jupyter server

unset XDG_RUNTIME_DIR
if [ "$SLURM_JOBTMP" != "" ]; then
    export XDG_RUNTIME_DIR=$SLURM_JOBTMP
fi

jupyter notebook --no-browser --port $port --notebook-dir=$(pwd) --ip=$node

Running Jupyter Notebook with Slurm script

There are 3 steps to run Jupyter Notebook on LANTA HPC.

1. Submit your job and read your slurm-xxxxx.out

Code Block
username@lanta:~> sbatch script.sh
username@lanta:~> cat slurm-xxxxx.out
    Jupyter server is running on: x1001c7s7b0n0
    Job starts at: Thu 28 Mar 2024 11:00:53 PM +07

    Copy/Paste the following command into your local terminal
    --------------------------------------------------------------------
    ssh -L 60151:x1001c7s7b0n0:60151 username@lanta.nstda.or.th -i id_rsa
    --------------------------------------------------------------------

    Open a browser on your local machine with the following address
    --------------------------------------------------------------------
    http://localhost:60151/?token=XXXXXXXX (see your token below)
    --------------------------------------------------------------------

[I 2024-03-28 23:01:07.926 ServerApp] Extension package jupyter_lsp took 0.4361s to import
[I 2024-03-28 23:01:09.498 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-03-28 23:01:09.503 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-03-28 23:01:09.508 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-03-28 23:01:09.534 ServerApp] notebook | extension was successfully linked.
[W 2024-03-28 23:01:12.375 ServerApp] jupyter_nbextensions_configurator | error adding extension (enabled: True): The module 'jupyter_nbextensions_configurator' could not be found (No module named 'jupyter_nbextensions_configurator'). Are you sure the extension is installed?
    Traceback (most recent call last):
      File "/lustrefs/disk/modules/easybuild/software/Mamba/23.11.0-0/envs/tensorflow-------2.12.1/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 322, in add_extension
        extpkg  http://localhost:8714/?token=XXXXXXXX (see your token below)= ExtensionPackage(name=extension_name, enabled=enabled)
       --------------------------------------------------------------------

[W 09:47:12.019 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension.
[W 2023-02-24 09:47:12.022 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2023-02-24 09:47:12.025 LabApp] JupyterLab extension loaded from /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/lib/python3.9/site-packages/jupyterlab
[I 2023-02-24 09:47:12.025 LabApp] JupyterLab application directory is /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/share/jupyter/lab
[I 09:47:12.028 NotebookApp] Serving notebooks from local directory: /home/yutthana/thaisc/yutthana/Jupyter_Script
[I 09:47:12.028 NotebookApp] Jupyter Notebook 6.5.2 is running at:
[I 09:47:12.028 NotebookApp] http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
[I 09:47:12.028 NotebookApp]  or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
[I 09:47:12.029 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 09:47:12.033 NotebookApp]

    To access the notebook, open this file in a browser:File "/lustrefs/disk/modules/easybuild/software/Mamba/23.11.0-0/envs/tensorflow-2.12.1/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 186, in __init__
        self._load_metadata()
      File "/lustrefs/disk/modules/easybuild/software/Mamba/23.11.0-0/envs/tensorflow-2.12.1/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 201, in _load_metadata
        raise ExtensionModuleNotFound(msg) from None
    jupyter_server.extension.utils.ExtensionModuleNotFound: The module 'jupyter_nbextensions_configurator' could not be found (No module named 'jupyter_nbextensions_configurator'). Are you sure the extension is installed?
[I 2024-03-28 23:01:12.393 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-03-28 23:01:12.534 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-03-28 23:01:12.536 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-03-28 23:01:12.537 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-03-28 23:01:12.549 LabApp] JupyterLab extension loaded from /lustrefs/disk/modules/easybuild/software/Mamba/23.11.0-0/envs/tensorflow-2.12.1/lib/python3.10/site-packages/jupyterlab
[I 2024-03-28 23:01:12.549 LabApp] JupyterLab application directory is /lustrefs/disk/modules/easybuild/software/Mamba/23.11.0-0/envs/tensorflow-2.12.1/share/jupyter/lab
[I 2024-03-28 23:01:12.550 LabApp] Extension Manager is 'pypi'.
[I 2024-03-28 23:01:12.599 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-03-28 23:01:12.605 ServerApp] notebook | extension was successfully loaded.
[I 2024-03-28 23:01:12.605 ServerApp] Serving notebooks from local directory: /home/ywongnon/Jupyter_GPU
[I 2024-03-28 23:01:12.605 ServerApp] Jupyter Server 2.13.0 is running at:
[I 2024-03-28 23:01:12.605 ServerApp] http://x1001c7s7b0n0:60151/tree?token=13acf46197090432de43db8f65d8651361ad319861bd04f5
[I 2024-03-28 23:01:12.605 ServerApp]     http://127.0.0.1:60151/tree?token=13acf46197090432de43db8f65d8651361ad319861bd04f5
[I 2024-03-28 23:01:12.605 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-03-28 23:01:12.620 ServerApp]

    To access the server, open this file in a browser:
        file:///lustrefs/disk/home/ywongnon/.local/share/jupyter/runtime/jpserver-25384-open.html
    Or copy and paste one of these URLs:
        http://x1001c7s7b0n0:60151/tree?token=13acf46197090432de43db8f65d8651361ad319861bd04f5
        filehttp:///lustrefs/disk/home/yutthana/.local/share/jupyter/runtime/nbserver-46789-open.html
    Or copy and paste one of these URLs:
        http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
     or 127.0.0.1:60151/tree?token=13acf46197090432de43db8f65d8651361ad319861bd04f5

2. Copy/Paste the following command into your local terminal for ssh tunneling to the LANTA HPC

Code Block
ssh -L 60151:x1001c7s7b0n0:60151 username@lanta.nstda.or.th -i id_rsa

...

Info

If you don’t have a private key (id_rsa file), you can use only the ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th command to access the LANTA HPC with your password and verification code.

3. Open a browser on your local machine with the following address (Final line in slurm-xxxxx.out) 

Code Block
http://127.0.0.1:871460151/tree?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929

2. Copy/Paste the following command into your local terminal for ssh tunneling to the LANTA HPC

Code Block
ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa

...

Info

If you don’t have a private key (id_rsa file), you can use only the ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th command to access the LANTA HPC with your password and activation code.

3. Open a browser on your local machine with the following address (Final line in slurm-xxxxx.out) 

Code Block
http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929

...

13acf46197090432de43db8f65d8651361ad319861bd04f5

...

Shutting down the Jupyter Notebook

When you’re done with the Jupyter Notebook session, you can start the shutdown process by closing the browser and terminal on your local machine. Then, you must cancel your job in the Slurm system of the LANTA HPC with the scancel JOBID command.

Code Block
username@lanta:~> scancel xxxxx

...

Related articles

Filter by label (Content by label)
showLabelsfalse
max5
spacescom.atlassian.confluence.content.render.xhtml.model.resource.identifiers.SpaceResourceIdentifier@48ae393
showSpacefalse
sortmodified
showSpacetypefalsepage
reversetrue
typelabelspagesingularity python container
cqllabel in ( "jupyter-vir-env" , "jupyter-apptainer" , "env" , "python-script" ) and space = currentSpace ( )labelssingularity python container