This article will guide you to create an environment using Miniconda on a LANTA HPC system.

Table of Contents

Using Miniconda via Easybuild

Load Miniconda module

Use the ml av Miniconda command to see which version of Miniconda is available on the LANTA HPC system.
Use the ml Miniconda3/xx.xx.x command to load the Miniconda version that you want to use. If you don't specify a version, the default version (D) is loaded, which is Miniconda3/22.11.1-1.

...

Code Block

username@lanta:~> conda env list
# conda environments:
#
base                     /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1
netcdf-py39              /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/netcdf-py39
pytorch-1.11.0           /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/pytorch-1.11.0
tensorflow-2.6.0         /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0
username@lanta:~> conda activate tensorflow-2.6.0
(tensorflow-2.6.0) username@lanta:~>

Creating an environment in the user’s home

Create an environment

Use the conda create -n myenv commands to create the conda environment with myenv name.

...

Create an environment with a specific version of the packages

Install Jupyter and other packages in the myenv environment

Use the conda install jupyter command to install jupyter in the myenv environment.
If you want to install other packages such as TensorFlow-GPU, you can use the conda install -c anaconda tensorflow-gpu command to install TensorFlow-GPU in the myenv environment.

Code Block
(myenv) username@lanta:~> conda install jupyter ... (myenv)create -n myenv python=3.9 username@lanta:~> conda installcreate -cn anaconda tensorflow-gpu ...

Running Jupyter Notebook via ssh tunneling

Example script for running Jupyter Notebook

Code Block

#!/bin/bash
#SBATCH -p gpu                          # Specify partition [Compute/Memory/GPU]
#SBATCH -N 1 -c 16                      # Specify number of nodes and processors per task
#SBATCH --gpus-per-task=1               # Specify the number of GPUs
#SBATCH --ntasks-per-node=4             # Specify tasks per node
#SBATCH -t 2:00:00                      # Specify maximum time limit (hour: minute: second)
#SBATCH -A projxxxx                     # Specify project name
#SBATCH -J JOBNAME                      # Specify job name

module purge                            # Unload all modules
module load Miniconda3/22.11.1-1        # Load the module that you want to use
conda activate myenv                    # Activate your environment

port=$(shuf -i 6000-9999 -n 1)
USER=$(whoami)
node=$(hostname -s)

#jupyter notebookng instructions to the output file
echo -e "

    Jupyter server is running on: $(hostname)
    Job starts at: $(date)

    Copy/Paste this in your local terminal to ssh tunnel with remote
    -----------------------------------------------------------------
    ssh -L $port:$node:$port $USER@lanta.nstda.or.th -i id_rsa
    -----------------------------------------------------------------

    Open a browser on your local machine with the following address
    ------------------------------------------------------------------
    http://localhost:${port}/?token=XXXXXXXX (see your token below)
    ------------------------------------------------------------------
    "

## start a cluster instance and launch jupyter server

unset XDG_RUNTIME_DIR
if [ "$SLURM_JOBTMP" != "" ]; then
    export XDG_RUNTIME_DIR=$SLURM_JOBTMP
fi

jupyter notebook --no-browser --port $port --notebook-dir=$(pwd) --ip=$node

Running Jupyter Notebook with Slurm script

There are 3 steps to run Jupyter Notebook on LANTA HPC.

1. Submit your job and read your slurm-xxxxx.out

Code Block

username@lanta:~> sbatch script.sh
username@lanta:~> cat slurm-xxxxx.out
    Jupyter server is running on: x1000c2s0b0n0
    Job starts at: Fri 24 Feb 2023 09:47:09 AM +07

    Copy/Paste this in your local terminal to ssh tunnel with remote
    -----------------------------------------------------------------
    ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa
    -----------------------------------------------------------------

    Open a browser on your local machine with the following address
    ------------------------------------------------------------------
    http://localhost:8714/?token=XXXXXXXX (see your token below)
    ------------------------------------------------------------------

[W 09:47:12.019 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension.
[W 2023-02-24 09:47:12.022 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2023-02-24 09:47:12.025 LabApp] JupyterLab extension loaded from /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/lib/python3.9/site-packages/jupyterlab
[I 2023-02-24 09:47:12.025 LabApp] JupyterLab application directory is /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/share/jupyter/lab
[I 09:47:12.028 NotebookApp] Serving notebooks from local directory: /home/yutthana/thaisc/yutthana/Jupyter_Script
[I 09:47:12.028 NotebookApp] Jupyter Notebook 6.5.2 is running at:
[I 09:47:12.028 NotebookApp] http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
[I 09:47:12.028 NotebookApp]  or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
[I 09:47:12.029 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 09:47:12.033 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///lustrefs/disk/home/yutthana/.local/share/jupyter/runtime/nbserver-46789-open.html
    Or copy and paste one of these URLs:
        http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
     or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929

2. Copy/Paste the following command into your local terminal for ssh tunneling to LANTA HPC

Code Block
ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa

...

3. Open a browser on your local machine with the following address (Final line in slurm-xxxxx.out)

Code Block
http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929

...

myenv python=3.9 scipy=0.17.3

Creating an environment in the project’s home

Specify a location for an environment

Code Block
username@lanta:~> conda create --prefix /your project directory/envs

Specify a location for an environment with a specific version of the packages

Code Block
username@lanta:~> conda create --prefix /your project directory/envs python=3.9

Activate your environment in the project’s home

Code Block
username@lanta:~> conda activate /your project directory/envs

Creating an environment from an environment.yml file

A simple environment.yml file

Code Block
name: test dependencies: - python=3.9 - numpy=1.23.1 - pandas

Create the environment from the environment.yml file in the user’s home

Code Block
username@lanta:~> conda env create -f environment.yml

Create the environment from the environment.yml file in the project’s home

Code Block
username@lanta:~> conda env create -f environment.yml --prefix /your project directory/envs

...

Version	Old Version 7	New Version 8
Changes made by	Yutthana Wongnongwa	Yutthana Wongnongwa
Saved on	Feb 27, 2023	Feb 27, 2023

Versions Compared

Key

Using Miniconda via Easybuild

Load Miniconda module

Creating an environment in the user’s home

Create an environment

Create an environment with a specific version of the packages

Install Jupyter and other packages in the myenv environment

Running Jupyter Notebook via ssh tunneling

Example script for running Jupyter Notebook

Running Jupyter Notebook with Slurm script

Creating an environment in the project’s home

Specify a location for an environment

Specify a location for an environment with a specific version of the packages

Activate your environment in the project’s home

Creating an environment from an environment.yml file

A simple environment.yml file

Create the environment from the environment.yml file in the user’s home

Create the environment from the environment.yml file in the project’s home

Related articles

Page Comparison

Versions Compared

Key

Using Miniconda via Easybuild

Load Miniconda module

Creating an environment in the user’s home

Create an environment

Create an environment with a specific version of the packages

Install Jupyter and other packages in the myenv environment

Running Jupyter Notebook via ssh tunneling

Example script for running Jupyter Notebook

Running Jupyter Notebook with Slurm script

Creating an environment in the project’s home

Specify a location for an environment

Specify a location for an environment with a specific version of the packages

Activate your environment in the project’s home

Creating an environment from an environment.yml file

A simple environment.yml file

Create the environment from the environment.yml file in the user’s home

Create the environment from the environment.yml file in the project’s home

Related articles