Local module & EasyBuild

Local module is a beneficial approach to efficiently organize your self-installed libraries. Additionally, EasyBuild serves as a framework for building and installing software, particularly those with dependencies. The following sections provide general guidelines on how to use local modules and EasyBuild to install and run software on LANTA.

Updated: May 2024

0. Define local module command (only once)

To utilize local modules, we suggest creating a bash command such as localmod that contains essential setup within your $HOME/.bashrc. The provided set of commands below will create a local module directory based on LOCALMOD_DIR and then define the localmod command in your $HOME/.bashrc.

This localmod command is needed to be executed to activate local modules, before installing software in Step 1 and before using local modules in your job script (see below).

LOCALMOD_DIR=$HOME/localmod           # *** EDITABLE, set to an empty path ***

mkdir -p ${LOCALMOD_DIR}              # Create the directory if not already exist
cat << Eof >> $HOME/.bashrc           # Define the below "localmod" function in your .bashrc
function localmod(){
  export EASYBUILD_PREFIX=${LOCALMOD_DIR}   
  module use ${EASYBUILD_PREFIX}/modules/all
  export EASYBUILD_OPTARCH=x86-milan
  echo "*** Import modules at ${EASYBUILD_PREFIX} on LANTA ***"
}
Eof

1. Prepare environment

• Enable local module by executing the command defined in Step 0 → localmod

• Load EasyBuild → module load EasyBuild

• Check your current EasyBuild configuration → eb --show-config.
  The installpath should be set to a local directory that you have write permission.

localmod
module load EasyBuild
eb --show-config

2. Get EasyConfig file

• Visit EasyConfig Standard Repo or other repositories.

• Download the latest .eb file of the software that you would like to install, then rename it to
  <software-name>-<software-version>-<cpe-toolchain-name>-<cpe-toolchain-version>.eb
  where CPE toolchain name is cpeGNU, cpeIntel, cpeCray or others, corresponding to the compiler that you would like to use. For their available version, use module spider cpeXXX or see Current CPE toolchains.

• For example, download giflib-5.2.1-GCCcore-13.2.0.eb and rename it to giflib-5.2.1-cpeCray-23.03.eb

3. Edit EasyConfig file

Open the .eb file using a text editor such as vi, then edit the following

1. toolchain
   Correct the toolchain information, e.g., toolchain = {'name': 'cpeCray', 'version': '23.03'}. This must be consistent with the CPE toolchain and file name chosen in Step 2.

2. builddependencies and/or dependencies
   Check if the software dependencies are available in ThaiSC modules by using module spider <dep-name>.
   For example, module spider binutils. If available, you could use them by editing each dependency information ('<dep-name>', '<dep-version>', '<dep-suffix>', <dep-toolchain>) where

   • <dep-name> is the dependency name, e.g., binutils.

   • <dep-version> is the dependency version, e.g., 2.40.

   • <dep-suffix> and <dep-toolchain> are optional. The default will use the same toolchain as in Step 3.1 without any suffix. This is recommended. For more details, visit writing .eb.

4. Install the software

• Execute eb -D --robot-paths=$(pwd) <the-eb-file>.eb to do a short dry run and check for dependencies.

  • If the command run successfully, it will display build status of the software’s dependencies.
    (You could ignore warning like WARNING: Failed to determine toolchain hierarchy for ...)

  • On the other hand, if you get ERROR: Missing dependencies: xxx/yyy-zzz, this means that the EasyConfig file of the library/software name xxx with version yyy using toolchain zzz is missing. You need to get the EasyConfig file for the dependency xxx/yyy-zzz by doing Step 2 and 3 again. That is, you also need to repeatedly do Step 2 and 3 for all unavailable dependencies required by the target software and put them in the same directory.

• To install the software, execute

  eb -r --robot-paths=$(pwd) --parallel=1 --trace --tmpdir=$(pwd) <the-eb-file>.eb

5. Job script template for using local modules

#!/bin/bash
#SBATCH -p compute              # Partition
#SBATCH -N 1                    # Number of nodes
#SBATCH --ntasks-per-node=128   # Number of MPI processes per node
#SBATCH --cpus-per-task=1       # Number of OpenMP threads per MPI process
#SBATCH -t 5-00:00:00           # Job runtime limit
#SBATCH -J MyJob                # Job name
#SBATCH -A ltxxxxxx             # Account *** {USER EDIT} *** 

source $HOME/.bashrc
localmod

module purge
module load <your-local-module>

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

srun -c${SLURM_CPUS_PER_TASK} <software-command>

Reference

• https://docs.easybuild.io/writing-easyconfig-files/

• https://mdc-easybuild.readthedocs.io/en/eb462/version-specific/easyconfig_parameters.html

• https://ulhpc-tutorials.readthedocs.io/en/latest/tools/easybuild/

EasyConfig repositories

• https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs

• https://github.com/Lumi-supercomputer/LUMI-EasyBuild-contrib/tree/main/easybuild/easyconfigs (Software may be in a different name.)

• https://github.com/eth-cscs/production/tree/master/easybuild/easyconfigs (Software may be in a different name.)

• https://github.com/hpc-unibe-ch/ubelix-easyconfigs