This article will guide you to run the Jupyter Notebook via Mamba on the LANTA HPC system, which requires ssh tunneling to the LANTA HPC. An overview of the content can be found in the table of contents below for immediate visualization of the interesting parts.
Creating an environment to run the Jupyter Notebook
Load Mamba module
Use the
ml av Mamba
command to see which version of Mamba is available on the LANTA HPC system.Use the
ml Mamba/xx.xx.x
command to load the Mamba version that you want to use. If you don't specify a version, the default version (D) is loaded, which is Mamba/23.11.0-0.
username@lanta:~> ml av Mamba ---------------------- /lustrefs/disk/modules/easybuild/modules/all ----------------------- Mamba/23.11.0-0 (D) Use "module spider" to find all possible modules and extensions. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". username@lanta:~> ml Mamba/23.11.0-0
Create the environment
Use the
conda create -n myenv python=3.9
commands to create the conda environment with myenv name and a specific version of python.Use the
conda activate myenv
to activate the myenv environment.
username@lanta:~> conda create -n myenv python=3.9 Channels: - conda-forge Platform: linux-64 Collecting package metadata (repodata.json): done Solving environment: done ## Package Plan ## environment location: /your directory/envs/myenv added / updated specs: - python=3.9 The following packages will be downloaded: package | build ---------------------------|----------------- python-3.9.19 |h0755675_0_cpython 22.7 MB conda-forge wheel-0.43.0 | pyhd8ed1ab_1 57 KB conda-forge ------------------------------------------------------------ Total: 22.8 MB The following NEW packages will be INSTALLED: _libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-2_gnu bzip2 conda-forge/linux-64::bzip2-1.0.8-hd590300_5 ca-certificates conda-forge/linux-64::ca-certificates-2024.2.2-hbcca054_0 ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0 libffi conda-forge/linux-64::libffi-3.4.2-h7f98852_5 libgcc-ng conda-forge/linux-64::libgcc-ng-13.2.0-h807b86a_5 libgomp conda-forge/linux-64::libgomp-13.2.0-h807b86a_5 libnsl conda-forge/linux-64::libnsl-2.0.1-hd590300_0 libsqlite conda-forge/linux-64::libsqlite-3.45.2-h2797004_0 libuuid conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0 libxcrypt conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1 libzlib conda-forge/linux-64::libzlib-1.2.13-hd590300_5 ncurses conda-forge/linux-64::ncurses-6.4.20240210-h59595ed_0 openssl conda-forge/linux-64::openssl-3.2.1-hd590300_1 pip conda-forge/noarch::pip-24.0-pyhd8ed1ab_0 python conda-forge/linux-64::python-3.9.19-h0755675_0_cpython readline conda-forge/linux-64::readline-8.2-h8228510_1 setuptools conda-forge/noarch::setuptools-69.2.0-pyhd8ed1ab_0 tk conda-forge/linux-64::tk-8.6.13-noxft_h4845f30_101 tzdata conda-forge/noarch::tzdata-2024a-h0c530f3_0 wheel conda-forge/noarch::wheel-0.43.0-pyhd8ed1ab_1 xz conda-forge/linux-64::xz-5.2.6-h166bdaf_0 Proceed ([y]/n)? y ... username@lanta:~> conda activate myenv (myenv) username@lanta:~>
Install Jupyter and other packages in the myenv environment
Use the
conda install jupyter
command to install jupyter in the myenv environment.If you want to install other packages such as PyTorch, you can use the
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
command to install PyTorch in the myenv environment.
(myenv) username@lanta:~> conda install jupyter ... (myenv) username@lanta:~> pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 ...
Running Jupyter Notebook via ssh tunneling
Example of Slurm script for running Jupyter Notebook
#!/bin/bash #SBATCH -p gpu # Specify partition [Compute/Memory/GPU] #SBATCH -N 1 -c 16 # Specify number of nodes and processors per task #SBATCH --gpus-per-task=1 # Specify the number of GPUs #SBATCH --ntasks-per-node=4 # Specify tasks per node #SBATCH -t 2:00:00 # Specify maximum time limit (hour: minute: second) #SBATCH -A ltxxxxxx # Specify project name #SBATCH -J JOBNAME # Specify job name module load Mamba/23.11.0-0 # Load the module that you want to use conda activate myenv # Activate your environment port=$(shuf -i 6000-9999 -n 1) USER=$(whoami) node=$(hostname -s) # jupyter notebookng instructions to the output file echo -e " Jupyter server is running on: $(hostname) Job starts at: $(date) Copy/Paste the following command into your local terminal -------------------------------------------------------------------- ssh -L $port:$node:$port $USER@lanta.nstda.or.th -i id_rsa -------------------------------------------------------------------- Open a browser on your local machine with the following address -------------------------------------------------------------------- http://localhost:${port}/?token=XXXXXXXX (see your token below) -------------------------------------------------------------------- " # start a cluster instance and launch jupyter server unset XDG_RUNTIME_DIR if [ "$SLURM_JOBTMP" != "" ]; then export XDG_RUNTIME_DIR=$SLURM_JOBTMP fi jupyter notebook --no-browser --port $port --notebook-dir=$(pwd) --ip=$node
Running Jupyter Notebook with Slurm script
There are 3 steps to run Jupyter Notebook on LANTA HPC.
1. Submit your job and read your slurm-xxxxx.out
username@lanta:~> sbatch script.sh username@lanta:~> cat slurm-xxxxx.out Jupyter server is running on: x1000c2s0b0n0 Job starts at: Fri 24 Feb 2023 09:47:09 AM +07 Copy/Paste the following command into your local terminal -------------------------------------------------------------------- ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa -------------------------------------------------------------------- Open a browser on your local machine with the following address -------------------------------------------------------------------- http://localhost:8714/?token=XXXXXXXX (see your token below) -------------------------------------------------------------------- [W 09:47:12.019 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension. [W 2023-02-24 09:47:12.022 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-02-24 09:47:12.022 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [I 2023-02-24 09:47:12.025 LabApp] JupyterLab extension loaded from /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/lib/python3.9/site-packages/jupyterlab [I 2023-02-24 09:47:12.025 LabApp] JupyterLab application directory is /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/share/jupyter/lab [I 09:47:12.028 NotebookApp] Serving notebooks from local directory: /home/yutthana/thaisc/yutthana/Jupyter_Script [I 09:47:12.028 NotebookApp] Jupyter Notebook 6.5.2 is running at: [I 09:47:12.028 NotebookApp] http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 [I 09:47:12.028 NotebookApp] or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 [I 09:47:12.029 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 09:47:12.033 NotebookApp] To access the notebook, open this file in a browser: file:///lustrefs/disk/home/yutthana/.local/share/jupyter/runtime/nbserver-46789-open.html Or copy and paste one of these URLs: http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
2. Copy/Paste the following command into your local terminal for ssh tunneling to the LANTA HPC
ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa
If you don’t have a private key (id_rsa file), you can use only the ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th
command to access the LANTA HPC with your password and verification code.
3. Open a browser on your local machine with the following address (Final line in slurm-xxxxx.out)
http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
Shutting down the Jupyter Notebook
When you’re done with the Jupyter Notebook session, you can start the shutdown process by closing the browser and terminal on your local machine. Then, you must cancel your job in the Slurm system of the LANTA HPC with the scancel JOBID
command.
username@lanta:~> scancel xxxxx