Versions Compared

Version Old Version 6 New Version Current
Changes made by

Yutthana Wongnongwa

Yutthana Wongnongwa

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This article will guide you to create an environment using Miniconda on a LANTA HPC system.

Table of Contents

Using Miniconda via Easybuild

Load Miniconda module

...

Use the ml av Miniconda command to see which version of Miniconda is available on the LANTA HPC system.

...

This article will guide you to create the environment using Miniconda on a LANTA HPC system. An overview of the content can be found in the table of contents below for immediate visualization of the interesting parts.

Table of Contents

Using Mamba on LANTA HPC

Load Mamba module

  1. Use the ml av Mamba command to see which version of Mamba is available on the LANTA HPC system.

  2. Use the ml Mamba/xx.xx.x command to load the Miniconda Mamba version that you want to use. If you don't specify a version, the default version (D) is loaded, which is Miniconda3Mamba/2223.11.10-10.

Code Block
username@lanta:~> ml av MinicondaMamba
---------------------- /lustrefs/disk/modules/easybuild/modules/all -----------------------
   Miniconda3Mamba/2223.11.1-10-0 (D)

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
username@lanta:~> ml Miniconda3Mamba/2223.11.10-1

...

0

Unload Mamba module

Use the

...

module unload Mamba command to unload the Mamba.

Code Block
username@lanta:~> condamodule env list
# conda environmentsunload Mamba

Use the module purge command to unload all currently loaded modules.

Code Block
username@lanta:~> module purge
Note

Before you use the module unload Mamba or module purge command, you must deactivate your environment with the conda deactivate command.

Activate your environment

  1. Use the conda env list command to view a list of your environments.

  2. If you want to activate your environment such as TensorFlow-2.6.0, you can use the conda activate tensorflow-2.12.1 command.

Code Block
username@lanta:~> conda env list
# conda environments:
#
base                     /lustrefs/disk/modules/easybuild/software/Miniconda3Mamba/2223.11.10-10
netcdf-py39              /lustrefs/disk/modules/easybuild/software/Miniconda3Mamba/2223.11.10-10/envs/netcdf-py39
pytorch-12.112.02            /lustrefs/disk/modules/easybuild/software/Miniconda3Mamba/2223.11.10-10/envs/pytorch-12.112.02
tensorflow-2.612.01         /lustrefs/disk/modules/easybuild/software/Miniconda3Mamba/2223.11.10-10/envs/tensorflow-2.612.01
username@lanta:~> conda activate tensorflow-2.6.0

Creating an environment to run the Jupyter Notebook

Load Miniconda module

...

Use the ml av Miniconda command to see which version of Miniconda is available on the LANTA HPC system.

...

.12.1
(tensorflow-2.12.1) username@lanta:~>

Deactivate your environment

Use the conda deactivate command to deactivate your environments.

Code Block
(tensorflow-2.12.1) username@lanta:~> conda deactivate
username@lanta:~>

Creating the environment in the user’s home

Create the environment

Use the conda create -n myenv commands to create the environment with myenv name.

Code Block
username@lanta:~> ml av Miniconda
---------------------- /lustrefs/disk/modules/easybuild/modules/all -----------------------
   Miniconda3/22.11.1-1

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
username@lanta:~> ml Miniconda3/22.11.1-1

Create an environment

...

Use the conda create -n myenv commands to create the conda environment with myenv name.

...

conda create -n myenv
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /your directory/envs/myenv

Proceed ([y]/n)? y
...

Create the environment with a specific version of the packages

Use the conda create -n myenv python=3.9 commands to create the myenv environment with a specific version of python.

Code Block
username@lanta:~> conda create -n myenv Collecting package metadata (current_repodata.json): done
python=3.9
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /your directory/envs/myenv

  Proceedadded ([y]/n)? y
...
username@lanta:~> conda activate myenv
(myenv) username@lanta:~>

Install Jupyter and other packages in the myenv environment

  1. Use the conda install jupyter command to install jupyter in the myenv environment.

  2. If you want to install other packages such as TensorFlow-GPU, you can use the conda install -c anaconda tensorflow-gpu command to install TensorFlow-GPU in the myenv environment.

Code Block
(myenv) username@lanta:~> conda install jupyter
...
(myenv) username@lanta:~> conda install -c anaconda tensorflow-gpu
...

Running Jupyter Notebook via ssh tunneling

Example script for running Jupyter Notebook

Code Block
#!/bin/bash
#SBATCH -p gpu                          # Specify partition [Compute/Memory/GPU]
#SBATCH -N 1 -c 16         / updated specs:
    - python=3.9


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    python-3.9.19              |h0755675_0_cpython        22.7 MB  conda-forge
    wheel-0.43.0               |     pyhd8ed1ab_1          57 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        22.8 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
  bzip2              conda-forge/linux-64::bzip2-1.0.8-hd590300_5
  ca-certificates    conda-forge/linux-64::ca-certificates-2024.2.2-hbcca054_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0
  libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5
  libgcc-ng          conda-forge/linux-64::libgcc-ng-13.2.0-h807b86a_5
  libgomp            conda-forge/linux-64::libgomp-13.2.0-h807b86a_5
  libnsl             conda-forge/linux-64::libnsl-2.0.1-hd590300_0
  libsqlite          conda-forge/linux-64::libsqlite-3.45.2-h2797004_0
  libuuid            conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0
  libxcrypt          conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1
  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5
  ncurses            conda-forge/linux-64::ncurses-6.4.20240210-h59595ed_0
  openssl            conda-forge/linux-64::openssl-3.2.1-hd590300_1
  pip                conda-forge/noarch::pip-24.0-pyhd8ed1ab_0
  python             conda-forge/linux-64::python-3.9.19-h0755675_0_cpython
  readline           conda-forge/linux-64::readline-8.2-h8228510_1
  setuptools         conda-forge/noarch::setuptools-69.2.0-pyhd8ed1ab_0
  tk                 conda-forge/linux-64::tk-8.6.13-noxft_h4845f30_101
  tzdata             conda-forge/noarch::tzdata-2024a-h0c530f3_0
  wheel              conda-forge/noarch::wheel-0.43.0-pyhd8ed1ab_1
  xz                 conda-forge/linux-64::xz-5.2.6-h166bdaf_0

Proceed ([y]/n)? y
...

Use the conda create -n myenv python=3.9 numpy=1.23.5 commands to create the myenv environment with a specific version of python and numpy.

Code Block
username@lanta:~> conda create -n myenv python=3.9 numpy=1.23.5
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /your directory/envs/myenv

  added / updated specs:
    - numpy=1.23.5
    - python=3.9


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    libblas-3.9.0              |21_linux64_openblas          14 KB  conda-forge
    libcblas-3.9.0             |21_linux64_openblas          14 KB  conda-forge
    libgfortran-ng-13.2.0      |       h69a702a_5          23 KB  conda-forge
    liblapack-3.9.0            |21_linux64_openblas          14 KB  conda-forge
    numpy-1.23.5               |   py39h3d75532_0         5.6 MB  conda-forge
    python-3.9.19              |h0755675_0_cpython        22.7 MB  conda-forge
    python_abi-3.9             |           4_cp39           6 KB  conda-forge
    wheel-0.43.0               |     pyhd8ed1ab_1          57 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        28.4 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
  bzip2              conda-forge/linux-64::bzip2-1.0.8-hd590300_5
  ca-certificates    conda-forge/linux-64::ca-certificates-2024.2.2-hbcca054_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0
  libblas            conda-forge/linux-64::libblas-3.9.0-21_linux64_openblas
  libcblas           conda-forge/linux-64::libcblas-3.9.0-21_linux64_openblas
  libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5
  libgcc-ng          conda-forge/linux-64::libgcc-ng-13.2.0-h807b86a_5
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-13.2.0-h69a702a_5
  libgfortran5       conda-forge/linux-64::libgfortran5-13.2.0-ha4646dd_5
  libgomp            conda-forge/linux-64::libgomp-13.2.0-h807b86a_5
  liblapack          conda-forge/linux-64::liblapack-3.9.0-21_linux64_openblas
  libnsl             conda-forge/linux-64::libnsl-2.0.1-hd590300_0
  libopenblas        conda-forge/linux-64::libopenblas-0.3.26-pthreads_h413a1c8_0
  libsqlite          conda-forge/linux-64::libsqlite-3.45.2-h2797004_0
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-13.2.0-h7e041cc_5
  libuuid             # Specify number of nodes and processors per task
#SBATCH --gpus-per-task=1   conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0
  libxcrypt            # Specify the number of GPUs
#SBATCH --ntasks-per-node=4   conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1
  libzlib          # Specify tasks per node
#SBATCH -t 2:00:00   conda-forge/linux-64::libzlib-1.2.13-hd590300_5
  ncurses            conda-forge/linux-64::ncurses-6.4.20240210-h59595ed_0
  numpy     # Specify maximum time limit (hour: minute: second) #SBATCH -A projxxxx       conda-forge/linux-64::numpy-1.23.5-py39h3d75532_0
  openssl           # Specify project name
#SBATCH -J JOBNAME conda-forge/linux-64::openssl-3.2.1-hd590300_1
  pip                conda-forge/noarch::pip-24.0-pyhd8ed1ab_0
  python     # Specify job name  module purge   conda-forge/linux-64::python-3.9.19-h0755675_0_cpython
  python_abi         conda-forge/linux-64::python_abi-3.9-4_cp39
  readline          # Unload all modules
module load Miniconda3/22.11.1-1        # Load the module that you want to use
conda activate myenv       conda-forge/linux-64::readline-8.2-h8228510_1
  setuptools         conda-forge/noarch::setuptools-69.2.0-pyhd8ed1ab_0
  tk              # Activate your environment

port=$(shuf -i 6000-9999 -n 1)
USER=$(whoami)
node=$(hostname -s)

#jupyter notebookng instructions to the output file
echo -e "

    Jupyter server is running on: $(hostname) conda-forge/linux-64::tk-8.6.13-noxft_h4845f30_101
  tzdata             conda-forge/noarch::tzdata-2024a-h0c530f3_0
  wheel            Job starts at: $(date) conda-forge/noarch::wheel-0.43.0-pyhd8ed1ab_1
   xz  Copy/Paste this in your local terminal to ssh tunnel with remote     -----------------------------------------------------------------
    ssh -L $port:$node:$port $USER@lanta.nstda.or.th -i id_rsa
    -----------------------------------------------------------------

    Open a browser on your local machine with the following address
    ------------------------------------------------------------------
    http://localhost:${port}/?token=XXXXXXXX (see your token below)conda-forge/linux-64::xz-5.2.6-h166bdaf_0

Proceed ([y]/n)? y
...

Creating the environment in the project’s home

Specify a location for the environment

Use the conda create --prefix /your project directory/envs commands to create the environment on the specific location.

Code Block
username@lanta:~> conda create --prefix /your project directory/envs
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /your project directory/envs

Proceed ([y]/n)? y
...

Specify a location for the environment with a specific version of a package

Use the conda create --prefix /your project directory/envs python=3.9 commands to create the environment with a specific version of a package on a specific location.

Code Block
username@lanta:~> conda create --prefix /your project directory/envs python=3.9
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /your project directory/envs

  added / updated specs:
    - python=3.9

The following packages will be downloaded:

    package                    |            build
    ---------------------------|---------------------------------------
    python-3.9.19              |h0755675_0_cpython        22.7 MB  conda-forge
    wheel-0.43.0       "  ## start a cluster instance and launch| jupyter server  unset XDGpyhd8ed1ab_RUNTIME_DIR1 if [ "$SLURM_JOBTMP" != "" ]; then   57 KB export XDG_RUNTIME_DIR=$SLURM_JOBTMP
fi conda-forge
  jupyter notebook --no-browser --port $port --notebook-dir=$(pwd) --ip=$node

Running Jupyter Notebook with Slurm script

There are 3 steps to run Jupyter Notebook on LANTA HPC.

1. Submit your job and read your slurm-xxxxx.out

Code Block
username@lanta:~> sbatch script.sh
username@lanta:~> cat slurm-xxxxx.out ------------------------------------------------------------
                Jupyter server is running on: x1000c2s0b0n0     Job starts at: Fri 24 Feb 2023 09:47:09 AM +07      Copy/Paste this inTotal: your local terminal to ssh tunnel with remote22.8 MB

The  -----------------------------------------------------------------following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex     ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa
    -----------------------------------------------------------------conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
  bzip2              conda-forge/linux-64::bzip2-1.0.8-hd590300_5
  ca-certificates    conda-forge/linux-64::ca-certificates-2024.2.2-hbcca054_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0
  libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5
  libgcc-ng          conda-forge/linux-64::libgcc-ng-13.2.0-h807b86a_5
  libgomp            conda-forge/linux-64::libgomp-13.2.0-h807b86a_5
  libnsl            Open a browser on your local machine with the following address
    ------------------------------------------------------------------
    http://localhost:8714/?token=XXXXXXXX (see your token below)
    ------------------------------------------------------------------

[W 09:47:12.019 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension.
[W 2023-02-24 09:47:12.022 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2023-02-24 09:47:12.025 LabApp] JupyterLab extension loaded from /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/lib/python3.9/site-packages/jupyterlab
[I 2023-02-24 09:47:12.025 LabApp] JupyterLab application directory is /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/share/jupyter/lab
[I 09:47:12.028 NotebookApp] Serving notebooks from local directory: /home/yutthana/thaisc/yutthana/Jupyter_Script
[I 09:47:12.028 NotebookApp] Jupyter Notebook 6.5.2 is running at:
[I 09:47:12.028 NotebookApp] http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
[I 09:47:12.028 NotebookApp]  or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
[I 09:47:12.029 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 09:47:12.033 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///lustrefs/disk/home/yutthana/.local/share/jupyter/runtime/nbserver-46789-open.html
    Or copy and paste one of these URLs:
        http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
     or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929

2. Copy/Paste the following command into your local terminal for ssh tunneling to LANTA HPC

Code Block
ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa

...

3. Open a browser on your local machine with the following address (Final line in slurm-xxxxx.out) 

Code Block
http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929

...

 conda-forge/linux-64::libnsl-2.0.1-hd590300_0
  libsqlite          conda-forge/linux-64::libsqlite-3.45.2-h2797004_0
  libuuid            conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0
  libxcrypt          conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1
  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5
  ncurses            conda-forge/linux-64::ncurses-6.4.20240210-h59595ed_0
  openssl            conda-forge/linux-64::openssl-3.2.1-hd590300_1
  pip                conda-forge/noarch::pip-24.0-pyhd8ed1ab_0
  python             conda-forge/linux-64::python-3.9.19-h0755675_0_cpython
  readline           conda-forge/linux-64::readline-8.2-h8228510_1
  setuptools         conda-forge/noarch::setuptools-69.2.0-pyhd8ed1ab_0
  tk                 conda-forge/linux-64::tk-8.6.13-noxft_h4845f30_101
  tzdata             conda-forge/noarch::tzdata-2024a-h0c530f3_0
  wheel              conda-forge/noarch::wheel-0.43.0-pyhd8ed1ab_1
  xz                 conda-forge/linux-64::xz-5.2.6-h166bdaf_0

Proceed ([y]/n)? y
...

Activate your environment in the project’s home

If you want to activate your environment in the project’s home, you can use the conda activate /your project directory/envs command.

Code Block
username@lanta:~> conda activate /your project directory/envs
(/your project directory/envs) username@lanta:~>

Creating the environment from an requirements.yml file

A simple requirements.yml file

In the requirements.yml file, you need to specify the environment name and packages that you want to use.

Code Block
name: envs
dependencies:
  - python=3.9
  - numpy=1.23.5
  - pandas

Create the environment from the requirements.yml file in the user’s home

Use the conda env create -f requirements.yml commands to create the environment from the requirements.yml file in the user’s home.

Code Block
username@lanta:~> conda env create -f requirements.yml
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Create the environment from the requirements.yml file in the project’s home

Use the conda env create -f requirements.yml --prefix /your project directory/envs commands to create the environment from the requirements.yml file on the specific location.

Code Block
username@lanta:~> conda env create -f requirements.yml --prefix /your project directory/envs
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Removing the environment

Remove the environment in the user’s home

If you want to remove the myenv environment, you can use the conda remove --name myenv --all command.

Code Block
username@lanta:~> conda remove --name myenv --all

Remove the environment in the project’s home

If you want to remove the environment in /your project directory/envs, you can use the rm -rf /your project directory/envs command.

Code Block
username@lanta:~> rm -rf /your project directory/envs

...

Related articles

Filter by label (Content by label)
showLabelsfalse
max5
spacescom.atlassian.confluence.content.render.xhtml.model.resource.identifiers.SpaceResourceIdentifier@48ae393
showSpacefalse
sortmodified
showSpacetypefalsepage
reversetrue
typelabelspagesingularity python container
cqllabel = in ( "vir-env" , "python-script" , "jupyter" ) and space = currentSpace ( )labelssingularity python container