This article will guide you to create an environment using Miniconda on a LANTA HPC system.
| Table of Contents |
|---|
Using Miniconda via Easybuild
Load Miniconda module
Use the
ml av Minicondacommand to see which version of Miniconda is available on the LANTA HPC system.Use the
ml Miniconda3/xx.xx.xcommand to load the Miniconda version that you want to use. If you don't specify a version, the default version (D) is loaded, which is Miniconda3/22.11.1-1.
...
| Code Block |
|---|
username@lanta:~> conda env list
# conda environments:
#
base /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1
netcdf-py39 /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/netcdf-py39
pytorch-1.11.0 /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/pytorch-1.11.0
tensorflow-2.6.0 /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0
username@lanta:~> conda activate tensorflow-2.6.0
(tensorflow-2.6.0) username@lanta:~> |
Creating an environment in the user’s home
Create an environment
Use the conda create -n myenv commands to create the conda environment with myenv name.
...
Create an environment with a specific version of the packages
Install Jupyter and other packages in the myenv environment
...
the
...
packages
...
| Code Block |
|---|
(myenv) username@lanta:~> conda install jupyter ... (myenv)create -n myenv python=3.9 username@lanta:~> conda installcreate -cn anaconda tensorflow-gpu ... |
Running Jupyter Notebook via ssh tunneling
Example script for running Jupyter Notebook
| Code Block |
|---|
#!/bin/bash
#SBATCH -p gpu # Specify partition [Compute/Memory/GPU]
#SBATCH -N 1 -c 16 # Specify number of nodes and processors per task
#SBATCH --gpus-per-task=1 # Specify the number of GPUs
#SBATCH --ntasks-per-node=4 # Specify tasks per node
#SBATCH -t 2:00:00 # Specify maximum time limit (hour: minute: second)
#SBATCH -A projxxxx # Specify project name
#SBATCH -J JOBNAME # Specify job name
module purge # Unload all modules
module load Miniconda3/22.11.1-1 # Load the module that you want to use
conda activate myenv # Activate your environment
port=$(shuf -i 6000-9999 -n 1)
USER=$(whoami)
node=$(hostname -s)
#jupyter notebookng instructions to the output file
echo -e "
Jupyter server is running on: $(hostname)
Job starts at: $(date)
Copy/Paste this in your local terminal to ssh tunnel with remote
-----------------------------------------------------------------
ssh -L $port:$node:$port $USER@lanta.nstda.or.th -i id_rsa
-----------------------------------------------------------------
Open a browser on your local machine with the following address
------------------------------------------------------------------
http://localhost:${port}/?token=XXXXXXXX (see your token below)
------------------------------------------------------------------
"
## start a cluster instance and launch jupyter server
unset XDG_RUNTIME_DIR
if [ "$SLURM_JOBTMP" != "" ]; then
export XDG_RUNTIME_DIR=$SLURM_JOBTMP
fi
jupyter notebook --no-browser --port $port --notebook-dir=$(pwd) --ip=$node |
Running Jupyter Notebook with Slurm script
There are 3 steps to run Jupyter Notebook on LANTA HPC.
1. Submit your job and read your slurm-xxxxx.out
| Code Block |
|---|
username@lanta:~> sbatch script.sh
username@lanta:~> cat slurm-xxxxx.out
Jupyter server is running on: x1000c2s0b0n0
Job starts at: Fri 24 Feb 2023 09:47:09 AM +07
Copy/Paste this in your local terminal to ssh tunnel with remote
-----------------------------------------------------------------
ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa
-----------------------------------------------------------------
Open a browser on your local machine with the following address
------------------------------------------------------------------
http://localhost:8714/?token=XXXXXXXX (see your token below)
------------------------------------------------------------------
[W 09:47:12.019 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension.
[W 2023-02-24 09:47:12.022 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2023-02-24 09:47:12.022 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2023-02-24 09:47:12.025 LabApp] JupyterLab extension loaded from /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/lib/python3.9/site-packages/jupyterlab
[I 2023-02-24 09:47:12.025 LabApp] JupyterLab application directory is /lustrefs/disk/modules/easybuild/software/Miniconda3/22.11.1-1/envs/tensorflow-2.6.0/share/jupyter/lab
[I 09:47:12.028 NotebookApp] Serving notebooks from local directory: /home/yutthana/thaisc/yutthana/Jupyter_Script
[I 09:47:12.028 NotebookApp] Jupyter Notebook 6.5.2 is running at:
[I 09:47:12.028 NotebookApp] http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
[I 09:47:12.028 NotebookApp] or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
[I 09:47:12.029 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 09:47:12.033 NotebookApp]
To access the notebook, open this file in a browser:
file:///lustrefs/disk/home/yutthana/.local/share/jupyter/runtime/nbserver-46789-open.html
Or copy and paste one of these URLs:
http://x1000c2s0b0n0:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929
or http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 |
2. Copy/Paste the following command into your local terminal for ssh tunneling to LANTA HPC
| Code Block |
|---|
ssh -L 8714:x1000c2s0b0n0:8714 username@lanta.nstda.or.th -i id_rsa |
...
3. Open a browser on your local machine with the following address (Final line in slurm-xxxxx.out)
| Code Block |
|---|
http://127.0.0.1:8714/?token=2923d6fab4ef109f30e63a77014e632eed3fd2a5fa561929 |
...
myenv python=3.9 scipy=0.17.3 |
Creating an environment in the project’s home
Specify a location for an environment
| Code Block |
|---|
username@lanta:~> conda create --prefix /your project directory/envs |
Specify a location for an environment with a specific version of the packages
| Code Block |
|---|
username@lanta:~> conda create --prefix /your project directory/envs python=3.9 |
Activate your environment in the project’s home
| Code Block |
|---|
username@lanta:~> conda activate /your project directory/envs |
Creating an environment from an environment.yml file
A simple environment.yml file
| Code Block |
|---|
name: test
dependencies:
- python=3.9
- numpy=1.23.1
- pandas |
Create the environment from the environment.yml file in the user’s home
| Code Block |
|---|
username@lanta:~> conda env create -f environment.yml |
Create the environment from the environment.yml file in the project’s home
| Code Block |
|---|
username@lanta:~> conda env create -f environment.yml --prefix /your project directory/envs |
...
Related articles
| Filter by label (Content by label) | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...