Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 33 Next »

This article will guide you to run Jupyter Notebook via Miniconda on a LANTA HPC system, which requires ssh tunneling to the LANTA HPC.

Creating an environment to run the Jupyter Notebook

Load Miniconda module

  1. Use the ml av Miniconda command to see which version of Miniconda is available on the LANTA HPC system.

  2. Use the ml Miniconda3/xx.xx.x command to load the Miniconda version that you want to use. If you don't specify a version, the default version (D) is loaded, which is Miniconda3/22.11.1-1.

username@lanta:~> ml av Miniconda
---------------------- /lustrefs/disk/modules/easybuild/modules/all -----------------------
   Miniconda3/22.11.1-1

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
username@lanta:~> ml Miniconda3/22.11.1-1

Create an environment

  1. Use the conda create -n myenv python=3.9 commands to create the conda environment with myenv name and a specific version of python.

  2. Use the conda activate myenv to activate the myenv environment.

username@lanta:~> conda create -n myenv python=3.9
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /your directory/envs/myenv

  added / updated specs:
    - python=3.9

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2023.01.10 |       h06a4308_0         120 KB
    certifi-2022.12.7          |   py39h06a4308_0         150 KB
    ------------------------------------------------------------
                                           Total:         270 KB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
  ca-certificates    pkgs/main/linux-64::ca-certificates-2023.01.10-h06a4308_0
  certifi            pkgs/main/linux-64::certifi-2022.12.7-py39h06a4308_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1
  libffi             pkgs/main/linux-64::libffi-3.4.2-h6a678d5_6
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1
  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1
  ncurses            pkgs/main/linux-64::ncurses-6.4-h6a678d5_0
  openssl            pkgs/main/linux-64::openssl-1.1.1t-h7f8727e_0
  pip                pkgs/main/linux-64::pip-22.3.1-py39h06a4308_0
  python             pkgs/main/linux-64::python-3.9.16-h7a1cb2a_0
  readline           pkgs/main/linux-64::readline-8.2-h5eee18b_0
  setuptools         pkgs/main/linux-64::setuptools-65.6.3-py39h06a4308_0
  sqlite             pkgs/main/linux-64::sqlite-3.40.1-h5082296_0
  tk                 pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0
  tzdata             pkgs/main/noarch::tzdata-2022g-h04d1e81_0
  wheel              pkgs/main/linux-64::wheel-0.38.4-py39h06a4308_0
  xz                 pkgs/main/linux-64::xz-5.2.10-h5eee18b_1
  zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0

Proceed ([y]/n)? y
...
username@lanta:~> conda activate myenv
(myenv) username@lanta:~>

Install Jupyter and other packages in the myenv environment

  1. Use the conda install jupyter command to install jupyter in the myenv environment.

  2. If you want to install other packages such as TensorFlow-GPU, you can use the conda install -c anaconda tensorflow-gpu command to install TensorFlow-GPU in the myenv environment.

(myenv) username@lanta:~> conda install jupyter
...
(myenv) username@lanta:~> conda install -c anaconda tensorflow-gpu
...

Running Jupyter Notebook via ssh tunneling

Example of Slurm script for running Jupyter Notebook

#!/bin/bash
#SBATCH -p gpu                          # Specify partition [Compute/Memory/GPU]
#SBATCH -N 1 -c 16                      # Specify number of nodes and processors per task
#SBATCH --gpus-per-task=1               # Specify the number of GPUs
#SBATCH --ntasks-per-node=4             # Specify tasks per node
#SBATCH -t 2:00:00                      # Specify maximum time limit (hour: minute: second)
#SBATCH -A ltxxxxxx                     # Specify project name
#SBATCH -J JOBNAME                      # Specify job name

module purge                            # Unload all modules
module load Miniconda3/22.11.1-1        # Load the module that you want to use
conda activate myenv                    # Activate your environment

port=$(shuf -i 6000-9999 -n 1)
USER=$(whoami)
node=$(hostname -s)

# jupyter notebookng instructions to the output file
echo -e "

    Jupyter server is running on: $(hostname)
    Job starts at: $(date)

    Copy/Paste the following command into your local terminal 
    --------------------------------------------------------------------
    ssh -L $port:$node:$port $USER@lanta.nstda.or.th -i id_rsa
    --------------------------------------------------------------------

    Open a browser on your local machine with the following address
    --------------------------------------------------------------------
    http://localhost:${port}/?token=XXXXXXXX (see your token below)
    --------------------------------------------------------------------
    "

# start a cluster instance and launch jupyter server

unset XDG_RUNTIME_DIR
if [ "$SLURM_JOBTMP" != "" ]; then
    export XDG_RUNTIME_DIR=$SLURM_JOBTMP
fi

jupyter notebook --no-browser --port $port --notebook-dir=$(pwd) --ip=$node

Running Jupyter Notebook with Slurm script

There are 3 steps to run Jupyter Notebook on LANTA HPC.

1. Submit your job and read your slurm-xxxxx.out

username@lanta:~> sbatch script.sh
username@lanta:~> cat slurm-xxxxx.out
    Jupyter server is running on: x1000c2s0b0n0
    Job starts at: Fri 24 Feb 2023 09:47:09 AM +07

    Copy/Paste the following command into your local terminal
    --------------------------------------------------------------------
    ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa
    --------------------------------------------------------------------

    Open a browser on your local machine with the following address
    --------------------------------------------------------------------
    http://localhost:8714/?token=XXXXXXXX (see your token below)
    --------------------------------------------------------------------

[W 09:47:12.019 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension.
[W 2023-02-24 09:47:12.022 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2023-02-24 09:47:12.025 LabApp] JupyterLab extension loaded from /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/lib/python3.9/site-packages/jupyterlab
[I 2023-02-24 09:47:12.025 LabApp] JupyterLab application directory is /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/share/jupyter/lab
[I 09:47:12.028 NotebookApp] Serving notebooks from local directory: /home/yutthana/thaisc/yutthana/Jupyter_Script
[I 09:47:12.028 NotebookApp] Jupyter Notebook 6.5.2 is running at:
[I 09:47:12.028 NotebookApp] http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
[I 09:47:12.028 NotebookApp]  or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
[I 09:47:12.029 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 09:47:12.033 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///lustrefs/disk/home/yutthana/.local/share/jupyter/runtime/nbserver-46789-open.html
    Or copy and paste one of these URLs:
        http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
     or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929

2. Copy/Paste the following command into your local terminal for ssh tunneling to LANTA HPC

ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa

3. Open a browser on your local machine with the following address (Final line in slurm-xxxxx.out) 

http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929

Related articles

  • No labels