This article will guide you to run the Jupyter Notebook via Miniconda on the LANTA HPC system, which requires ssh tunneling to the LANTA HPC. An overview of the content can be found in the table of contents below for immediate visualization of the interesting parts.
| Table of Contents |
|---|
Creating an environment to run the Jupyter Notebook
Load Miniconda module
Use the
ml av Minicondacommand to see which version of Miniconda is available on the LANTA HPC system.Use the
ml Miniconda3/xx.xx.xcommand to load the Miniconda version that you want to use. If you don't specify a version, the default version (D) is loaded, which is Miniconda3/23.3.1-0.
| Code Block |
|---|
username@lanta:~> ml av Miniconda ---------------------- /lustrefs/disk/modules/easybuild/modules/all ----------------------- Miniconda3/22.11.1-1 Miniconda3/23.3.1-0 (D) Use "module spider" to find all possible modules and extensions. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". username@lanta:~> ml Miniconda3/22.11.1-1 |
Create the environment
Use the
conda create -n myenv python=3.9commands to create the conda environment with myenv name and a specific version of python.Use the
conda activate myenvto activate the myenv environment.
| Code Block |
|---|
username@lanta:~> conda create -n myenv python=3.9
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /your directory/envs/myenv
added / updated specs:
- python=3.9
The following packages will be downloaded:
package | build
---------------------------|-----------------
ca-certificates-2023.01.10 | h06a4308_0 120 KB
certifi-2022.12.7 | py39h06a4308_0 150 KB
------------------------------------------------------------
Total: 270 KB
The following NEW packages will be INSTALLED:
_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
_openmp_mutex pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
ca-certificates pkgs/main/linux-64::ca-certificates-2023.01.10-h06a4308_0
certifi pkgs/main/linux-64::certifi-2022.12.7-py39h06a4308_0
ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1
libffi pkgs/main/linux-64::libffi-3.4.2-h6a678d5_6
libgcc-ng pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1
libgomp pkgs/main/linux-64::libgomp-11.2.0-h1234567_1
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1
ncurses pkgs/main/linux-64::ncurses-6.4-h6a678d5_0
openssl pkgs/main/linux-64::openssl-1.1.1t-h7f8727e_0
pip pkgs/main/linux-64::pip-22.3.1-py39h06a4308_0
python pkgs/main/linux-64::python-3.9.16-h7a1cb2a_0
readline pkgs/main/linux-64::readline-8.2-h5eee18b_0
setuptools pkgs/main/linux-64::setuptools-65.6.3-py39h06a4308_0
sqlite pkgs/main/linux-64::sqlite-3.40.1-h5082296_0
tk pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0
tzdata pkgs/main/noarch::tzdata-2022g-h04d1e81_0
wheel pkgs/main/linux-64::wheel-0.38.4-py39h06a4308_0
xz pkgs/main/linux-64::xz-5.2.10-h5eee18b_1
zlib pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0
Proceed ([y]/n)? y
...
username@lanta:~> conda activate myenv
(myenv) username@lanta:~> |
Install Jupyter and other packages in the myenv environment
Use the
conda install jupytercommand to install jupyter in the myenv environment.If you want to install other packages such as TensorFlow-GPU, you can use the
conda install -c anaconda tensorflow-gpucommand to install TensorFlow-GPU in the myenv environment.
| Code Block |
|---|
(myenv) username@lanta:~> conda install jupyter ... (myenv) username@lanta:~> conda install -c anaconda tensorflow-gpu ... |
Running Jupyter Notebook via ssh tunneling
Example of Slurm script for running Jupyter Notebook
| Code Block |
|---|
#!/bin/bash
#SBATCH -p gpu # Specify partition [Compute/Memory/GPU]
#SBATCH -N 1 -c 16 # Specify number of nodes and processors per task
#SBATCH --gpus-per-task=1 # Specify the number of GPUs
#SBATCH --ntasks-per-node=4 # Specify tasks per node
#SBATCH -t 2:00:00 # Specify maximum time limit (hour: minute: second)
#SBATCH -A ltxxxxxx # Specify project name
#SBATCH -J JOBNAME # Specify job name
module load Miniconda3/22.11.1-1 # Load the module that you want to use
conda activate myenv # Activate your environment
port=$(shuf -i 6000-9999 -n 1)
USER=$(whoami)
node=$(hostname -s)
# jupyter notebookng instructions to the output file
echo -e "
Jupyter server is running on: $(hostname)
Job starts at: $(date)
Copy/Paste the following command into your local terminal
--------------------------------------------------------------------
ssh -L $port:$node:$port $USER@lanta.nstda.or.th -i id_rsa
--------------------------------------------------------------------
Open a browser on your local machine with the following address
--------------------------------------------------------------------
http://localhost:${port}/?token=XXXXXXXX (see your token below)
--------------------------------------------------------------------
"
# start a cluster instance and launch jupyter server
unset XDG_RUNTIME_DIR
if [ "$SLURM_JOBTMP" != "" ]; then
export XDG_RUNTIME_DIR=$SLURM_JOBTMP
fi
jupyter notebook --no-browser --port $port --notebook-dir=$(pwd) --ip=$node |
Running Jupyter Notebook with Slurm script
There are 3 steps to run Jupyter Notebook on LANTA HPC.
...
| Code Block |
|---|
http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 |
...
Shutting down the Jupyter Notebook
When you’re done with the Jupyter Notebook session, you can start the shutdown process by closing the browser and terminal on your local machine. Then, you must cancel your job in the Slurm system of the LANTA HPC with the scancel JOBID command.
| Code Block |
|---|
username@lanta:~> scancel xxxxx |
...
Related articles
| Filter by label (Content by label) | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...